Hub management, via cscli allows you to install, upgrade, remove and view installed collections, parsers, scenarios etc.
# # A collection contains parsers and scenarios to form a coherent ensemble. Most of the time, this is the only you will need to install.
Have nginx running ? cscli collections install crowdsecurity/nginx should do the trick !
Browse the hub for more collections .
sudo cscli collections install < collection_name > Copy Install crowdsecurity/whitelist-good-actors collection sudo cscli collections install crowdsecurity/whitelist-good-actors INFO [ 0000 ] crowdsecurity/seo-bots-whitelist : OK INFO [ 0000 ] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rdns_seo_bots.txt' in '/var/lib/crowdsec/data/rdns_seo_bots.txt' INFO [ 0001 ] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rnds_seo_bots.regex' in '/var/lib/crowdsec/data/rdns_seo_bots.regex' INFO [ 0002 ] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/ip_seo_bots.txt' in '/var/lib/crowdsec/data/ip_seo_bots.txt' INFO [ 0002 ] crowdsecurity/cdn-whitelist : OK INFO [ 0002 ] downloading data 'https://www.cloudflare.com/ips-v4' in '/var/lib/crowdsec/data/cloudflare_ips.txt' INFO [ 0003 ] crowdsecurity/rdns : OK INFO [ 0003 ] crowdsecurity/whitelist-good-actors : OK INFO [ 0003 ] /etc/crowdsec/postoverflows/s01-whitelist doesn 't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/seo-bots-whitelist INFO[0003] Enabled postoverflows : crowdsecurity/cdn-whitelist INFO[0003] /etc/crowdsec/postoverflows/s00-enrich doesn' t exist, create INFO [ 0003 ] Enabled postoverflows : crowdsecurity/rdns INFO [ 0003 ] Enabled collections : crowdsecurity/whitelist-good-actors INFO [ 0003 ] Enabled crowdsecurity/whitelist-good-actors INFO [ 0003 ] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec Copy # sudo cscli collections list Copy cscli collections list example sudo cscli collections list ------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH ------------------------------------------------------------------------------------------------------------- crowdsecurity/nginx ✔️ enabled 0.1 /etc/crowdsec/collections/nginx.yaml crowdsecurity/base-http-scenarios ✔️ enabled 0.1 /etc/crowdsec/collections/base-http-scenarios.yaml crowdsecurity/sshd ✔️ enabled 0.1 /etc/crowdsec/collections/sshd.yaml crowdsecurity/linux ✔️ enabled 0.2 /etc/crowdsec/collections/linux.yaml ------------------------------------------------------------------------------------------------------------- Copy This will list only installed parsers.
Use --all to list available parsers.
# sudo cscli hub update sudo cscli collections upgrade < collection_name > Copy Collection upgrade allows you to upgrade an existing collection (and its items) to the latest version.
Upgrade crowdsecurity/sshd collection sudo cscli hub update INFO [ 06-08-2021 04:18:33 PM ] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli collections upgrade crowdsecurity/sshd INFO [ 0000 ] crowdsecurity/sshd : up-to-date WARN [ 0000 ] crowdsecurity/sshd-logs : overwrite WARN [ 0000 ] crowdsecurity/ssh-bf : overwrite WARN [ 0000 ] crowdsecurity/sshd : overwrite INFO [ 0000 ] 📦 crowdsecurity/sshd : updated INFO [ 0000 ] Upgraded 1 items INFO [ 0000 ] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec
Copy # sudo cscli collections inspect < collection_name > Copy Collections inspect will give you detailed information about a given collection, including versioning data and runtime metrics (fetched from prometheus).
cscli collections inspect example sudo cscli collections inspect crowdsecurity/sshd type: collections name: crowdsecurity/sshd filename: sshd.yaml description: 'sshd support : parser and brute-force detection' author: crowdsecurity belongs_to_collections: - crowdsecurity/linux - crowdsecurity/linux remote_path: collections/crowdsecurity/sshd.yaml version: "0.1" local_path: /etc/crowdsec/collections/sshd.yaml localversion: "0.1" localhash: 21159aeb87529efcf1a5033f720413d5321a6451bab679a999f7f01a7aa972b3 installed: true downloaded: true uptodate: true tainted: false local: false parsers: - crowdsecurity/sshd-logs scenarios: - crowdsecurity/ssh-bf
Current metrics :
- ( Scenario ) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+ | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | +---------------+-----------+--------------+--------+---------+ | 0 | 1 | 2 | 10 | 1 | +---------------+-----------+--------------+--------+---------+
Copy # See more about collection here .
# # sudo cscli parsers install < parser_name > Copy Install crowdsecurity/iptables-logs parser sudo cscli parsers install crowdsecurity/iptables-logs INFO [ 0000 ] crowdsecurity/iptables-logs : OK INFO [ 0000 ] Enabled parsers : crowdsecurity/iptables-logs INFO [ 0000 ] Enabled crowdsecurity/iptables-logs INFO [ 0000 ] Run 'systemctl reload crowdsec' for the new configuration to be effective. Copy # sudo cscli parsers list Copy Parsers are yaml files in /etc/crowdsec/parsers/<STAGE>/parser.yaml.
List installed parsers sudo cscli parsers list -------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH -------------------------------------------------------------------------------------------------------------- crowdsecurity/whitelists ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml crowdsecurity/dateparse-enrich ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml crowdsecurity/iptables-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml crowdsecurity/syslog-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml crowdsecurity/sshd-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml crowdsecurity/geoip-enrich ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml crowdsecurity/http-logs ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml crowdsecurity/nginx-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml --------------------------------------------------------------------------------------------------------------
Copy # sudo cscli hub update sudo cscli parsers upgrade < parser_name > Copy Parsers upgrade allows you to upgrade an existing parser to the latest version.
Upgrade crowdsecurity/sshd-logs parser sudo cscli hub update INFO [ 06-08-2021 04:18:33 PM ] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli parsers upgrade crowdsecurity/sshd-logs INFO [ 0000 ] crowdsecurity/sshd : up-to-date WARN [ 0000 ] crowdsecurity/sshd-logs : overwrite WARN [ 0000 ] crowdsecurity/ssh-bf : overwrite WARN [ 0000 ] crowdsecurity/sshd : overwrite INFO [ 0000 ] 📦 crowdsecurity/sshd : updated INFO [ 0000 ] Upgraded 1 items INFO [ 0000 ] Run 'systemctl reload crowdsec' for the new configuration to be effective.
Copy # sudo cscli parsers inspect < parser_name > Copy Parsers inspect will give you detailed information about a given parser, including versioning data and runtime metrics (fetched from prometheus).
Inspect crowdsecurity/sshd-logs parser sudo cscli parsers inspect crowdsecurity/sshd-logs type: parsers stage: s01-parse name: crowdsecurity/sshd-logs filename: sshd-logs.yaml description: Parse openSSH logs author: crowdsecurity belongs_to_collections: - crowdsecurity/sshd remote_path: parsers/s01-parse/crowdsecurity/sshd-logs.yaml version: "0.1" local_path: /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml localversion: "0.1" localhash: ecd40cb8cd95e2bad398824ab67b479362cdbf0e1598b8833e2f537ae3ce2f93 installed: true downloaded: true uptodate: true tainted: false local: false
Current metrics :
- ( Parser ) crowdsecurity/sshd-logs: +-------------------+-------+--------+----------+ | PARSERS | HITS | PARSED | UNPARSED | +-------------------+-------+--------+----------+ | /var/log/auth.log | 94138 | 42404 | 51734 | +-------------------+-------+--------+----------+
Copy # See more details about parsers here .
# Enrichers are basically parsers that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the s02-enrich stage (after most of the parsing happened).
Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the Enriched map of the event .
At the time of writing, the enrichers plugin mechanism implementation is still ongoing (read: the list of available enrichment methods is currently hardcoded).
As an example let's look into the geoip-enrich parser/enricher :
It relies on the geolite2 data created by maxmind and the geoip2 golang module to provide the actual data.
It exposes three methods: GeoIpCity GeoIpASN and IpToRange that are used by the crowdsecurity/geoip-enrich.
Enrichers can be installed as any other parsers with the following command:
sudo cscli parsers install crowdsecurity/geoip-enrich Copy Take a tour at the Hub to find them !
# See more about enrichers here .
# # sudo cscli scenarios install < scenario_name > Copy Install crowdsecurity/http-bf-wordpress_bf scenario sudo cscli scenarios install crowdsecurity/http-bf-wordpress_bf INFO [ 0000 ] crowdsecurity/http-bf-wordpress_bf : OK INFO [ 0000 ] Enabled scenarios : crowdsecurity/http-bf-wordpress_bf INFO [ 0000 ] Enabled crowdsecurity/http-bf-wordpress_bf INFO [ 0000 ] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsec Copy # sudo cscli scenarios list Copy This will list only installed parsers.
Use --all to list available parsers.
Scenario are yaml files in /etc/crowdsec/scenarios/.
List installed scenarios sudo cscli scenarios list --------------------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH --------------------------------------------------------------------------------------------------------------------------- crowdsecurity/ssh-bf ✔️ enabled 0.1 /etc/crowdsec/scenarios/ssh-bf.yaml crowdsecurity/http-bf-wordpress_bf ✔️ enabled 0.1 /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml crowdsecurity/http-crawl-non_statics ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-crawl-non_statics.yaml crowdsecurity/http-probing ✔️ enabled 0.1 /etc/crowdsec/scenarios/http-probing.yaml crowdsecurity/http-sensitive-files ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sensitive-files.yaml crowdsecurity/http-bad-user-agent ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-bad-user-agent.yaml crowdsecurity/http-path-traversal-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-path-traversal-probing.yaml crowdsecurity/http-sqli-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sqli-probing.yaml crowdsecurity/http-backdoors-attempts ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-backdoors-attempts.yaml crowdsecurity/http-xss-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-xss-probing.yaml ---------------------------------------------------------------------------------------------------------------------------
Copy # sudo cscli hub update sudo cscli scenarios upgrade < scenario_name > Copy Scenarios upgrade allows you to upgrade an existing scenario to the latest version.
Upgrade crowdsecurity/http-bf-wordpress_bf scenario sudo cscli hub update INFO [ 06-08-2021 04:18:33 PM ] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli scenarios upgrade crowdsecurity/ssh-bf INFO [ 0000 ] crowdsecurity/ssh-bf : up-to-date WARN [ 0000 ] crowdsecurity/ssh-bf : overwrite INFO [ 0000 ] 📦 crowdsecurity/ssh-bf : updated INFO [ 0000 ] Upgraded 1 items INFO [ 0000 ] Run 'systemctl reload crowdsec' for the new configuration to be effective. Copy # sudo cscli scenarios inspect < scenario_name > Copy Scenarios inspect will give you detailed information about a given scenario, including versioning data and runtime metrics (fetched from prometheus).
Inspect crowdsecurity/http-bf-wordpress_bf scenario sudo cscli scenarios inspect crowdsecurity/ssh-bf type: scenarios name: crowdsecurity/ssh-bf filename: ssh-bf.yaml description: Detect ssh bruteforce author: crowdsecurity references: - http://wikipedia.com/ssh-bf-is-bad belongs_to_collections: - crowdsecurity/sshd remote_path: scenarios/crowdsecurity/ssh-bf.yaml version: "0.1" local_path: /etc/crowdsec/scenarios/ssh-bf.yaml localversion: "0.1" localhash: 4441dcff07020f6690d998b7101e642359ba405c2abb83565bbbdcee36de280f installed: true downloaded: true uptodate: true tainted: false local: false
Current metrics :
- ( Scenario ) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+ | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | +---------------+-----------+--------------+--------+---------+ | 14 | 5700 | 7987 | 42572 | 2273 | +---------------+-----------+--------------+--------+---------+ Copy # See more about scenarios here .