From a5897f456281d23e3eb4ad08db688f2ecaedb30a Mon Sep 17 00:00:00 2001
From: Lauri Nevala <lauri.nevala@gmail.com>
Date: Tue, 5 Nov 2019 11:22:18 +0200
Subject: [PATCH] Document required changes when using custom prometheus

---
 troubleshooting/custom-prometheus.md | 37 ++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100644 troubleshooting/custom-prometheus.md

diff --git a/troubleshooting/custom-prometheus.md b/troubleshooting/custom-prometheus.md
new file mode 100644
index 0000000000..fb81eecf24
--- /dev/null
+++ b/troubleshooting/custom-prometheus.md
@@ -0,0 +1,37 @@
+# Using Custom Prometheus
+
+When using custom prometheus with Lens app, Lens expects certain things for prometheus rules and labels. Below is listed the changes required to see metrics properly.
+
+## kube-prometheus
+
+1. To see node metrics properly, please add
+
+```
+- action: replace
+  regex: (.*)
+  replacement: $1
+  sourceLabels:
+  - __meta_kubernetes_pod_node_name
+  targetLabel: kubernetes_node
+```
+
+relabeling to `node-exporter` servicemonitor crd (for example `kubectl edit -n monitoring servicemonitors node-exporter`).
+
+2. To see cluster's pod usage on cluster overview properly, please add `metricsRelabeling` to `kubelet` service monitor (`kubectl edit -n monitoring servicemonitors node-exporter`)
+
+```
+metricRelabelings:
+- action: replace
+  sourceLabels:
+  - node
+  targetLabel: instance
+```
+
+## Helm chart
+
+1. To see cpu metrics properly, please set value of `server.global.scrape_timeout` less than 1 minute, for example
+
+```
+helm upgrade --set server.global.scrape_timeout=30s prometheus stable/prometheus
+```
+