Filtering with Traefik and Crowdsec

This missive is the culmination of the last couple of days pouring over docs, reading other folks pleas for help in forums, github issues, blogs and YouTube-ers how-tos. It summarizes what turned out to be relatively simple but it was not so easy to find out how. Specifically, the new Traefik Crowdsec Plugin is not yet widely reviewed and getting it to work with Crowdsec is somewhat new.

What Crowdsec & Traefik Can Do For us

Crowdsec can be configured to read your system and access logs and whilst it does so, it checks for common exploits, that is the activities of bad actors on line.

When detected, the IP address from which said bad behavior is detected is added to a list it calls ‘decisions’ where by another app, often called a bouncer can be configured to take action.

A typical response is to block further access.

After a while, several hours typically but that can be configured in Crowdsec, the ban is lifted such that further attempts to access your system from the previously banned IP address is again possible.

In a container orchestrated environment folks have commonly used a 3rd party container that acts as the ‘bouncer’ and as a go-between Crowdsec and Traefik however recently there is another option.

Traefik now has plugins that can be added to extend its functionality and there is a ‘bouncer plugin’ that can be used to interface between it and Crowdsec, potentially reducing the amount of containers needed to be run and reducing the amount of maintenance and software to be updated over time

A Demo of How Crowdsec Can Work With Traefik

To get a log of current blocks that are in place for bad actors as mentioned above, we can shell into our Crowdsec container and ask it to list its decisions :

    ssh -t my-docker-swarm docker exec -it crowdsec cscli decisions list

This shows that Crowdsec is currently configured to block access from an address in France. As this is a test system configured only a day or so ago I am mildly surprised but not shocked to see this. Production systems will generate many more alerts than this !

In Traefik, specific sites are configured using labels to enforce this policy.

Seeing blocks is one thing but testing or proving that this works from my own current IP address can be done with :

    ./nikto.pl -h https://whoami.my-docker-swarm.uk/

Nikto is a perl program you can get from its github repo and run it as the above so long as you are running a competent Linux / mac or windows WSL environment that has perl installed and have permission to do so.

It is not at all recommended that this tool be used upon any sites that you do not own and manage yourself or that you have permissions from the owner to do so.

With that said, this is one of many tools we can use to analyze and protect our own services on line.

What Nicto does basically is that it scans the web address it is given for known issues. It does this aggressively enough for Crowdsec to pick it up and ban the sender.

Hey ho, I am banned and access to this site is ( for me at least ) now blocked :

Result !

This is what a bad actor as we described earlier would now expect to see.

So now, to clear this decision and for me to access this site once again, I can delete this for just my IP address with :

    ssh -t my-docker-swarm docker exec -it crowdsec cscli decisions delete -i x.x.x.x

In a couple of seconds, the ban is no longer in effect, I can access the site again, so long as I don’t run Nikto on it again that is, then I get banned for several hours as is the above IP in France, apparently.

How To Configure Traefik and Crowdsec

As \i mentioned earlier, I chose this time to use a plugin that Traefik has offered for us to use to act as a bouncer and its to be found in the plugins pane of the Traefik dashboard

and its code and further documentation can be found on its github repo

A starting point for Crowdsec can be found at their Docker / Podman Deployment page but I will run through how I have been able to implement both of these as containers below.

Crowdsec Container

The following is my test container configuration for Crowdsec

    services:
    
    crowdsec:
        image: crowdsecurity/crowdsec:v1.6.4
        container_name: crowdsec
        restart: unless-stopped
        expose:
        - 8080 # http api for bouncers
        - 6060 # metrics endpoint for prometheus
        - 7422 # appsec waf endpoint
        volumes:
        # crowdsec container data
        - ./data:/var/lib/crowdsec/data
        - ./etc:/etc/crowdsec
        # log bind mounts into crowdsec
        - /var/log/auth.log:/var/log/auth.log:ro
        - /var/log/syslog:/var/log/syslog:ro
        - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/traefik/logs
        - /home/user/docker/traefik-cr/logs:/var/log/traefik:ro
        environment:
        - GID=1000
        - COLLECTIONS=crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios crowdsecurity/sshd crowdsecurity/linux crowdsecurity/appsec-generic-rules crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-crs
        - BOUNCER_KEY_TRAEFIK=API_KEY_FOR_TRAEFIK_BOUNCER_PLUGIN
    
        networks:
        - traefik-net
    
    networks:
    traefik-net:
        external: true

This I will undoubtedly change over time as it is a test for now but I have to give credit to blog.lrvt.de for a very helpful and insightful article from which I have lifted much of this.

Once the container is running there are a few pieces of housekeeping you need to do namely, register your crowdsec container with an an active account app.crowdsec.net

At the crowdsec console page above you will see and need to take secure and safe note of an enrol command that you need to run in your crowdsec container :

    cscli console enroll -e context YOUR_ENROL_KEY

Next, confirm this action as will be requested back on the crowdsec web console and then restart the Crowdsec container to complete registration.

For the Crowdsec container to be used by our Traefik plugins we need to generate a key on our Crowdsec instance and again, take note of this and keep it secure and safe.

    cscli bouncers add traefik-bouncer
    API key for 'traefik-bouncer':
    
        API_KEY_FOR_TRAEFIK_BOUNCER_PLUGIN
    
    Please keep this key since you will not be able to retrieve it!

next is my traefik composer example file that works with the above

    ---
    services:
    traefik:
        image: traefik:3.2.3
        container_name: traefik
        restart: unless-stopped
        command:
        - "--api.dashboard=true"
        - "--api.insecure=false"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--certificatesresolvers.le.acme.httpchallenge=true"
        - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
        - "--certificatesresolvers.le.acme.email=MY_EMAIL"
        - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
        - "--configFile=/etc/traefik/traefik.yaml"
        labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik-dashboard.rule=Host(`URL_TO_DASHBOARD`)"
        - "traefik.http.routers.traefik-dashboard.entrypoints=websecure"
        - "traefik.http.routers.traefik-dashboard.tls.certresolver=le"
        - "traefik.http.routers.traefik-dashboard.service=api@internal"
        - "traefik.http.routers.traefik-dashboard.middlewares=auth"
        - "traefik.http.middlewares.auth.basicauth.users=user:{SHA}SHA_PASSWORD_STRING"
        - traefik.http.middlewares.crowdsec-root.plugin.bouncer.enabled=true
        - traefik.http.middlewares.crowdsec-root.plugin.bouncer.logLevel=INFO
        - traefik.http.middlewares.crowdsec-root.plugin.bouncer.crowdseclapikey=API_KEY_FOR_TRAEFIK_BOUNCER_PLUGIN
        - traefik.http.middlewares.crowdsec-root.plugin.bouncer.crowdseclapischeme=http
        - traefik.http.middlewares.crowdsec-root.plugin.bouncer.crowdseclapihost=crowdsec:8080
        ports:
        - "80:80"
        - "443:443"
        volumes:
        - ./letsencrypt:/letsencrypt
        - ./traefik.yaml:/etc/traefik/traefik.yaml
        - /etc/localtime:/etc/localtime:ro
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./logs:/logs
        networks:
        - traefik-net
    
    networks:
    traefik-net:
        external: true

The dashboard password SHA_PASSWORD_STRING can be generated with something like

    echo -n 'your-password' | openssl dgst -binary -sha1 | openssl base64

API_KEY_FOR_TRAEFIK_BOUNCER_PLUGIN is from the step earlier when configuring Crowdsec itself

This traefik configuration uses a static configuration file called traefik.yaml and it does not use a dynamic file with a ‘provider’, rather as can be seen by the above labels for crowdsec, these dynamic options are set here, not in a file such as fileConfig.yaml is convention often. I chose to do this as I prefer to keep this central for now to the compose / service deployment and it is I believe clearer and easier to understand.

here is the static config file traefik.yaml for the above

    api:
    insecure: false
    dashboard: true
        
    entryPoints:
    web:
        address: :80
    websecure:
        address: :443
    
    certificatesResolvers:
    le:
        acme:
        email: "MY_EMAIL"
        storage: /letsencrypt/acme.json
        httpChallenge:
            entryPoint: web
    
    providers:
    # file:
    #   filename: /fileConfig.yaml
    #   watch: true
    docker:
        endpoint: unix:///var/run/docker.sock
        exposedByDefault: false
        
    log:
    level: INFO
    
    accessLog:
    filePath: "/logs/traefik.log"
    format: json
    filters:
        statusCodes:
        - "200-299" # log successful http requests
        - "400-599" # log failed http requests
    # collect logs as in-memory buffer before writing into log file
    bufferingSize: 0
    fields:
        headers:
        defaultMode: drop # drop all headers per default
        names:
            User-Agent: keep # log user agent strings
    
    # crowdsec bouncer
    experimental:
    plugins:
        bouncer:
        moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
        version: "v1.3.5"

now, if all is working, in our dash we should see middlewares all working without error

before finally configuring our web service, in this case, whoami to be protected by this new middleware and via our new traefik plugin

    services:
    
    whoami:
        image: traefik/whoami
        container_name: whoami-crowdsec-test
        command:
            - --name=whoami
        environment:
        - WHOAMI_PORT_NUMBER=9003
        networks:
        - traefik-net
    
        labels:
        - traefik.enable=true
        - traefik.http.routers.whoami.rule=Host(`whoami.YOUR-DOMAIN-HERE`)
        - traefik.http.routers.whoami.service=whoami
        - traefik.http.services.whoami.loadbalancer.server.port=9003
        - traefik.http.routers.whoami.entrypoints=websecure
        - traefik.http.routers.whoami.tls.certresolver=le
        - traefik.http.routers.whoami.middlewares=crowdsec-root
    
    networks:
    traefik-net:
        external: true

the line that adds crowdsec protection to this container is

        - traefik.http.routers.whoami.middlewares=crowdsec-root

so this is easy to add to any subsequent containers behind this traefik instance so we can opt in or out individual services that are known to be stable and free of false positives as we go along, rather than configuring this at the top level and for all sites. But that is also an option if your comfortable to do so.

In Summary

The end result is quite pleasing and uses the latest version 3 of Traefik with its newer plugin architecture which is very nice to see and one which I have already started looking into myself. It uses yaegi a command line interpreter for Go that is good not just for traefik plugins I believe but also for general ‘scripting’ that can now be in Go rather than the more traditional Bash, Python, Perl, Ruby and so on which I believe opens up some quite exciting possibilities.

Getting there was not helped by the nature of the Traefik documentation which I see others have observed to be narrative in form, requiring the complete digestions of pages of information in order often to glean a single nugget of information that could have been served more easily if there was a search and / or a more comprehensive reference section than there is currently.

But we can only hope for this to improve over time and I will likely look to write more about Traefik in future as I believe it is a significant shift from the established Apache and NGINX stables of which we have been familiar with now for decades.

The dynamicity of Traefik makes it an ideal fit for container orchestration and at the very least reduces the complexity of configuration as time goes on as new containers / old ones that are stood down simply get added / removed / updated by traefik as this happens, not requiring what has become over time a system of double entry betwixt running services and their load balancers / reverse proxies.

A fascinating and exciting field of interest indeed !