It’s a bit tricky to figure out how to use a recording rule together with a Prometheus installation using the CoreOS Prometheus Operator. Here’s how I got it working.
If you take a look at Documentation/user-guides/Alerting.md, you’ll see that Kubernetes ConfigMaps are used to load the Prometheus configuration for each Prometheus (deployed via a CRD) in your cluster. These ConfigMaps are selected via two labels, role: alert-rules
and prometheus: <example>
.
So if you are using kube-prometheus to deploy the Prometheus Operator to your cluster (it gets a bit complicated with all the layers of abstraction), then if you want to deploy a recording rule directly into the k8s
Prometheus instance, you’ll want to do this:
Create the file with your recording rules and save it as something like custom-rules.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-example-rules
labels:
role: alert-rules
prometheus: k8s
data:
example.rules.yaml: |+
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
run kubectl apply -f customrules.yaml -n monitoring
You can then go see if it worked by connecting to the Prometheus instance and looking at /rules to see if it was applied.