解决ControllerManager、Scheduler、watchdog监控问题

# 默认安装后访问prometheus，会发现有以下有三个报警：

`Watchdog、KubeControllerManagerDown、KubeSchedulerDown`

`Watchdog`是一个正常的报警，这个告警的作用是：如果alermanger或者prometheus本身挂掉了就发不出告警了，因此一般会采用另一个监控来监控prometheus，或者自定义一个持续不断的告警通知，哪一天这个告警通知不发了，说明监控出现问题了。prometheus operator已经考虑了这一点，本身携带一个watchdog，作为对自身的监控。

如果需要关闭，删除或注释掉Watchdog部分
```
 kubectl get prometheusrules -n monitoring
NAME                              AGE
alertmanager-main-rules           10d
grafana-rules                     10d
kube-prometheus-rules             10d   # watchdog 配置在这个中
kube-state-metrics-rules          10d
kubernetes-monitoring-rules       10d
node-exporter-rules               10d
prometheus-k8s-prometheus-rules   10d
prometheus-operator-rules         10d

kubectl edit  prometheusrules kube-prometheus-rules  -n monitoring #edit
```
# KubeControllerManagerDown 和 kubeSchedulerDown
```
1.编辑/etc/kubernetes/manifests/kube-scheduler.yaml，更改--bind-address=127.0.0.1为--bind-address=0.0.0.0
(kube-controller-manager.yaml  也要改 address)
2.创建一个服务并确保它有一个与ServiceMonitor ( )k8s-app: kube-scheduler匹配的标签（对我来说是）****spec.selector.matchLabels
3.确保服务与正确的标签匹配正确的 pod（对我来说，是的component: kube-scheduler）
4.我花了很长时间才找到最后一个。😤 Service 的端口名（对我来说是）必须与ServiceMonitor ( )https-metrics匹配****spec.endpoints.port
5.对 kube-controller-manager 做同样的检查
```

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kube-scheduler #注意这里label 要和svc对的上
---
apiVersion: v1
kind: Service
metadata:
  namespace: kube-system
  name: kube-scheduler
  labels:
    k8s-app: kube-scheduler
spec:
  ports:
  - name: https-metrics
    port: 10259
  selector:
    component: kube-scheduler
```
---
controller-manager
```
cat << "EOF" > kube-controller-manager-svc.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: kube-system
  name: kube-controller-manager
  labels:
    k8s-app: kube-controller-manager
spec:
  ports:
  - name: https-metrics
    port: 10257
  selector:
    component: kube-controller-manager
EOF

kubectl apply -f kube-controller-manager-svc.yaml

# 然后 要去修改prometheus servicemonitor
kubectl edit  servicemonitor kube-controller-manager  -n monitoring 
```

![](/media/202311/2023-11-02_224737_9451930.3412096030091156.png)