Prometheus metrics on Caddy

Continuing with my Caddy experimentation, I wanted to get some metrics out of it. Caddy provides metrics out of the box on its admin API:

https://localhost:2019/metrics

This works well if you have prometheus running on the same server though in my case, prometheus is running on a different VM and on a private network.

Caddy will enforce https on every vhost by default. For public domains, it’ll try to get a Let’s Encrypt certificate and for IP addresses and localhost, it’ll use an internal CA to sign certificates that it’ll serve for these.

With that in mind, I needed to scrape the metrics endpoint from the outside and using a domain name though this domain name was only internal and as such, Let’s Encrypt wouldn’t be able to verify that I own the domain.

Caddy provides a way to force itself to use the internal CA rather than use Let’s Encrypt using tls internal. In the end, I used this configuration:

web1.int.xiu.io:2021 {
    tls internal # use the internal CA
    metrics # serve the prometheus metrics endpoint
}

And on the prometheus side, I needed to skip TLS verification as I don’t trust Caddy’s CA (yet):

  - job_name: 'caddy'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets:
        - web1.int.xiu.io:2021

And last, I used this dashboard on my Grafana to have a look at what was possible with the metrics. One thing I noticed right away was that the metrics don’t provide a vhost label to differentiate the vhosts. As such and in my usecase, the metrics are not as useful as I had expected. This is already identified and tracked by Caddy in this issue. In the meantime, I’ll get the structured logs Caddy provides in Loki to get the information I’m after (status code per vhost).

That’s it! I noticed there was a way for Caddy to do dns challenges with Let’s Encrypt so I will probably improve that part in order to get Let’s Encrypt certificates even on my home network. I will probably look into that and write a follow up post when I do.

– This is day 11/100 of #100DaysToOffLoad!