自定义指标HPA

![](/media/202310/2023-10-01_112124_1700690.8678801889448551.png)
(1) 资源指标工作流程: hpa -> apiserver -> kube aggregation -> metrics-server -> kubelet(cadvisor)
       (2) 自定义资源指标工作流程: hpa -> apiserver -> kube aggregation -> prometheus-adapter -> prometheus -> pods

kubernetes apiserver 提供了三种 API 用于监控指标相关的操作：

resource metrics API：被设计用来给 k8s 核心组件提供监控指标，例如 kubectl top；
custom metrics API：被设计用来给 HPA 控制器提供指标。
external metrics API：被设计用来通过外部指标扩容

![](/media/202310/2023-10-01_112144_4206510.43865474628195134.png)
Adapter作用是用于k8s与Prometheus进行通讯，充当两者之间的翻译器。

不管是metrics server还是metrics aggreator都是一个注册在k8s当中的一个接口，接口代理到prometheus adapter，然后它向prometheus去查询数据。查询完数据返回给接口，最后给到hpa。

所以prometheus adapter是k8s和prometheus之间的桥梁，metrics aggreator接口是不支持直接从prometheus当中拿数据的。因为hpa metrics的数据接口它是不支持从prometheus当中获取数据。
# 部署custom-metrics-api
```
gti clone https://github.com/qist/k8s.git
cd ./k8s/k8s-yaml/kube-prometheus/custom-metrics-api
kubectl apply -f ./
```
验证是否正常注册到api server
```
kubectl get apiservices |grep custom
```
调用其api，看看是否能够返回监控的数据
```
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | python -mjson.tool
```
# 修改 adapter-config 
# HPA yaml
```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: container_sockets
      target:
        type: AverageValue
        averageValue: 20
```
## configmap
```
apiVersion: v1
data:
  config.yaml: |-
    rules:
    - seriesQuery: 'container_sockets{namespace=~"default", pod=~"backend.*"}'
      resources:
        template: <<.Resource>>
      name:
        as: "container_sockets"
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
    - seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ^container_(.*)_seconds_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[1m])) by (<<.GroupBy>>)
    "resourceRules":
      "cpu":
        "containerLabel": "container"
        "containerQuery": |
          sum by (<<.GroupBy>>) (
            irate (
                container_cpu_usage_seconds_total{<<.LabelMatchers>>,container!="",pod!=""}[120s]
            )
          )
        "nodeQuery": |
          sum by (<<.GroupBy>>) (
            1 - irate(
              node_cpu_seconds_total{mode="idle"}[60s]
            )
            * on(namespace, pod) group_left(node) (
              node_namespace_pod:kube_pod_info:{<<.LabelMatchers>>}
            )
          )
          or sum by (<<.GroupBy>>) (
            1 - irate(
              windows_cpu_time_total{mode="idle", job="windows-exporter",<<.LabelMatchers>>}[4m]
            )
          )
        "resources":
          "overrides":
            "namespace":
              "resource": "namespace"
            "node":
              "resource": "node"
            "pod":
              "resource": "pod"
      "memory":
        "containerLabel": "container"
        "containerQuery": |
          sum by (<<.GroupBy>>) (
            container_memory_working_set_bytes{<<.LabelMatchers>>,container!="",pod!=""}
          )
        "nodeQuery": |
          sum by (<<.GroupBy>>) (
            node_memory_MemTotal_bytes{job="node-exporter",<<.LabelMatchers>>}
            -
            node_memory_MemAvailable_bytes{job="node-exporter",<<.LabelMatchers>>}
          )
          or sum by (<<.GroupBy>>) (
            windows_cs_physical_memory_bytes{job="windows-exporter",<<.LabelMatchers>>}
            -
            windows_memory_available_bytes{job="windows-exporter",<<.LabelMatchers>>}
          )
        "resources":
          "overrides":
            "instance":
              "resource": "node"
            "namespace":
              "resource": "namespace"
            "pod":
              "resource": "pod"
      "window": "5m"
```