Hub management
Hub management, via cscli allows you to install, upgrade, remove and view installed collections, parsers, scenarios etc.
Collections#
Installation#
A collection contains parsers and scenarios to form a coherent ensemble. Most of the time, this is the only you will need to install.
Have nginx running ? cscli collections install crowdsecurity/nginx should do the trick !
Browse the hub for more collections.
sudo cscli collections install <collection_name>Install crowdsecurity/whitelist-good-actors collection
sudo cscli collections install crowdsecurity/whitelist-good-actorsINFO[0000] crowdsecurity/seo-bots-whitelist : OK INFO[0000] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rdns_seo_bots.txt' in '/var/lib/crowdsec/data/rdns_seo_bots.txt' INFO[0001] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/rnds_seo_bots.regex' in '/var/lib/crowdsec/data/rdns_seo_bots.regex' INFO[0002] downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/whitelists/benign_bots/search_engine_crawlers/ip_seo_bots.txt' in '/var/lib/crowdsec/data/ip_seo_bots.txt' INFO[0002] crowdsecurity/cdn-whitelist : OK INFO[0002] downloading data 'https://www.cloudflare.com/ips-v4' in '/var/lib/crowdsec/data/cloudflare_ips.txt' INFO[0003] crowdsecurity/rdns : OK INFO[0003] crowdsecurity/whitelist-good-actors : OK INFO[0003] /etc/crowdsec/postoverflows/s01-whitelist doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/seo-bots-whitelist INFO[0003] Enabled postoverflows : crowdsecurity/cdn-whitelist INFO[0003] /etc/crowdsec/postoverflows/s00-enrich doesn't exist, create INFO[0003] Enabled postoverflows : crowdsecurity/rdns INFO[0003] Enabled collections : crowdsecurity/whitelist-good-actors INFO[0003] Enabled crowdsecurity/whitelist-good-actors INFO[0003] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsecList#
sudo cscli collections listcscli collections list example
sudo cscli collections list ------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH ------------------------------------------------------------------------------------------------------------- crowdsecurity/nginx ✔️ enabled 0.1 /etc/crowdsec/collections/nginx.yaml crowdsecurity/base-http-scenarios ✔️ enabled 0.1 /etc/crowdsec/collections/base-http-scenarios.yaml crowdsecurity/sshd ✔️ enabled 0.1 /etc/crowdsec/collections/sshd.yaml crowdsecurity/linux ✔️ enabled 0.2 /etc/crowdsec/collections/linux.yaml -------------------------------------------------------------------------------------------------------------tip
This will list only installed parsers.
Use --all to list available parsers.
Upgrade#
sudo cscli hub updatesudo cscli collections upgrade <collection_name>Collection upgrade allows you to upgrade an existing collection (and its items) to the latest version.
Upgrade crowdsecurity/sshd collection
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli collections upgrade crowdsecurity/sshd INFO[0000] crowdsecurity/sshd : up-to-date WARN[0000] crowdsecurity/sshd-logs : overwrite WARN[0000] crowdsecurity/ssh-bf : overwrite WARN[0000] crowdsecurity/sshd : overwrite INFO[0000] 📦 crowdsecurity/sshd : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective.$ systemctl reload crowdsec
Monitor#
sudo cscli collections inspect <collection_name>Collections inspect will give you detailed information about a given collection, including versioning data and runtime metrics (fetched from prometheus).
cscli collections inspect example
sudo cscli collections inspect crowdsecurity/sshd type: collectionsname: crowdsecurity/sshdfilename: sshd.yamldescription: 'sshd support : parser and brute-force detection'author: crowdsecuritybelongs_to_collections:- crowdsecurity/linux- crowdsecurity/linuxremote_path: collections/crowdsecurity/sshd.yamlversion: "0.1"local_path: /etc/crowdsec/collections/sshd.yamllocalversion: "0.1"localhash: 21159aeb87529efcf1a5033f720413d5321a6451bab679a999f7f01a7aa972b3installed: truedownloaded: trueuptodate: truetainted: falselocal: falseparsers:- crowdsecurity/sshd-logsscenarios:- crowdsecurity/ssh-bf
Current metrics :
- (Scenario) crowdsecurity/ssh-bf: +---------------+-----------+--------------+--------+---------+| CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |+---------------+-----------+--------------+--------+---------+| 0 | 1 | 2 | 10 | 1 |+---------------+-----------+--------------+--------+---------+
Reference#
See more about collection here.
Parsers#
Installation#
sudo cscli parsers install <parser_name>Install crowdsecurity/iptables-logs parser
sudo cscli parsers install crowdsecurity/iptables-logs INFO[0000] crowdsecurity/iptables-logs : OK INFO[0000] Enabled parsers : crowdsecurity/iptables-logs INFO[0000] Enabled crowdsecurity/iptables-logs INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. List#
sudo cscli parsers listParsers are yaml files in /etc/crowdsec/parsers/<STAGE>/parser.yaml.
List installed parsers
sudo cscli parsers list-------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH -------------------------------------------------------------------------------------------------------------- crowdsecurity/whitelists ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml crowdsecurity/dateparse-enrich ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml crowdsecurity/iptables-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml crowdsecurity/syslog-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml crowdsecurity/sshd-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml crowdsecurity/geoip-enrich ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml crowdsecurity/http-logs ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml crowdsecurity/nginx-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml --------------------------------------------------------------------------------------------------------------
Upgrade#
sudo cscli hub updatesudo cscli parsers upgrade <parser_name>Parsers upgrade allows you to upgrade an existing parser to the latest version.
Upgrade crowdsecurity/sshd-logs parser
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli parsers upgrade crowdsecurity/sshd-logs INFO[0000] crowdsecurity/sshd : up-to-date WARN[0000] crowdsecurity/sshd-logs : overwrite WARN[0000] crowdsecurity/ssh-bf : overwrite WARN[0000] crowdsecurity/sshd : overwrite INFO[0000] 📦 crowdsecurity/sshd : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective.
Monitor#
sudo cscli parsers inspect <parser_name>Parsers inspect will give you detailed information about a given parser, including versioning data and runtime metrics (fetched from prometheus).
Inspect crowdsecurity/sshd-logs parser
sudo cscli parsers inspect crowdsecurity/sshd-logs type: parsersstage: s01-parsename: crowdsecurity/sshd-logsfilename: sshd-logs.yamldescription: Parse openSSH logsauthor: crowdsecuritybelongs_to_collections:- crowdsecurity/sshdremote_path: parsers/s01-parse/crowdsecurity/sshd-logs.yamlversion: "0.1"local_path: /etc/crowdsec/parsers/s01-parse/sshd-logs.yamllocalversion: "0.1"localhash: ecd40cb8cd95e2bad398824ab67b479362cdbf0e1598b8833e2f537ae3ce2f93installed: truedownloaded: trueuptodate: truetainted: falselocal: false
Current metrics :
- (Parser) crowdsecurity/sshd-logs:+-------------------+-------+--------+----------+| PARSERS | HITS | PARSED | UNPARSED |+-------------------+-------+--------+----------+| /var/log/auth.log | 94138 | 42404 | 51734 |+-------------------+-------+--------+----------+
Reference#
See more details about parsers here.
Enrichers#
Enrichers are basically parsers that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the s02-enrich stage (after most of the parsing happened).
Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the Enriched map of the event.
caution
At the time of writing, the enrichers plugin mechanism implementation is still ongoing (read: the list of available enrichment methods is currently hardcoded).
As an example let's look into the geoip-enrich parser/enricher :
It relies on the geolite2 data created by maxmind and the geoip2 golang module to provide the actual data.
It exposes three methods: GeoIpCity GeoIpASN and IpToRange that are used by the crowdsecurity/geoip-enrich.
Enrichers can be installed as any other parsers with the following command:
sudo cscli parsers install crowdsecurity/geoip-enrichTake a tour at the Hub to find them !
Reference#
See more about enrichers here.
Scenarios#
Installation#
sudo cscli scenarios install <scenario_name>Install crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli scenarios install crowdsecurity/http-bf-wordpress_bfINFO[0000] crowdsecurity/http-bf-wordpress_bf : OK INFO[0000] Enabled scenarios : crowdsecurity/http-bf-wordpress_bf INFO[0000] Enabled crowdsecurity/http-bf-wordpress_bf INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. $ systemctl reload crowdsecList#
sudo cscli scenarios listtip
This will list only installed parsers.
Use --all to list available parsers.
Scenario are yaml files in /etc/crowdsec/scenarios/.
List installed scenarios
sudo cscli scenarios list--------------------------------------------------------------------------------------------------------------------------- NAME 📦 STATUS VERSION LOCAL PATH --------------------------------------------------------------------------------------------------------------------------- crowdsecurity/ssh-bf ✔️ enabled 0.1 /etc/crowdsec/scenarios/ssh-bf.yaml crowdsecurity/http-bf-wordpress_bf ✔️ enabled 0.1 /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml crowdsecurity/http-crawl-non_statics ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-crawl-non_statics.yaml crowdsecurity/http-probing ✔️ enabled 0.1 /etc/crowdsec/scenarios/http-probing.yaml crowdsecurity/http-sensitive-files ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sensitive-files.yaml crowdsecurity/http-bad-user-agent ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-bad-user-agent.yaml crowdsecurity/http-path-traversal-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-path-traversal-probing.yaml crowdsecurity/http-sqli-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sqli-probing.yaml crowdsecurity/http-backdoors-attempts ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-backdoors-attempts.yaml crowdsecurity/http-xss-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-xss-probing.yaml ---------------------------------------------------------------------------------------------------------------------------
Upgrade#
sudo cscli hub updatesudo cscli scenarios upgrade <scenario_name>Scenarios upgrade allows you to upgrade an existing scenario to the latest version.
Upgrade crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli hub updateINFO[06-08-2021 04:18:33 PM] Wrote new 126099 bytes index to /etc/crowdsec/hub/.index.json sudo cscli scenarios upgrade crowdsecurity/ssh-bfINFO[0000] crowdsecurity/ssh-bf : up-to-date WARN[0000] crowdsecurity/ssh-bf : overwrite INFO[0000] 📦 crowdsecurity/ssh-bf : updated INFO[0000] Upgraded 1 items INFO[0000] Run 'systemctl reload crowdsec' for the new configuration to be effective. Monitor#
sudo cscli scenarios inspect <scenario_name>Scenarios inspect will give you detailed information about a given scenario, including versioning data and runtime metrics (fetched from prometheus).
Inspect crowdsecurity/http-bf-wordpress_bf scenario
sudo cscli scenarios inspect crowdsecurity/ssh-bf type: scenariosname: crowdsecurity/ssh-bffilename: ssh-bf.yamldescription: Detect ssh bruteforceauthor: crowdsecurityreferences:- http://wikipedia.com/ssh-bf-is-badbelongs_to_collections:- crowdsecurity/sshdremote_path: scenarios/crowdsecurity/ssh-bf.yamlversion: "0.1"local_path: /etc/crowdsec/scenarios/ssh-bf.yamllocalversion: "0.1"localhash: 4441dcff07020f6690d998b7101e642359ba405c2abb83565bbbdcee36de280finstalled: truedownloaded: trueuptodate: truetainted: falselocal: false
Current metrics :
- (Scenario) crowdsecurity/ssh-bf:+---------------+-----------+--------------+--------+---------+| CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |+---------------+-----------+--------------+--------+---------+| 14 | 5700 | 7987 | 42572 | 2273 |+---------------+-----------+--------------+--------+---------+Reference#
See more about scenarios here.