prometheus-operator使用外部的端口

kubernetes Prometheus-operator

字数统计: 1.2k阅读时长: 5 min

 2021/04/01   Share

前言

在使用operator的过程中, 我们难免会有一些集群外的监控需求. 一般来说, prometheus operator是通过serviceMonitor发现的,那我们直接自定义endpoints不就行了吗？

使用外部的metrics

创建service

创建endpoints

首先,我们需要创建一个endpoints来指定需要加入的外部metrics

node-exporter-endpoints.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: node-exporter
  namespace: product
subsets:
  - addresses:
      - ip: 10.20.2.4
      - ip: 10.20.2.10
      - ip: 10.20.2.11
      - ip: 10.20.2.5
    ports:
      - port: 9100
        name: http-metrics

创建service

注意:

service 的 metadata.name 必须要跟endpoints的metadata.name一样.

service 的 spec.ports[*].name 必须要跟endpoints的 subsets[*].ports[*].name 一样

apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: product
  labels:
    app: external-node-exporter
spec:
  ports:
    - protocol: TCP
      name: http-metrics
      port: 9100
      targetPort: 9100

创建ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: external-node-exporter
  namespace: product
spec:
  endpoints:
    - interval: 15s
      port: http-metrics
  selector:
    matchLabels:
      app: external-node-exporter

看起来似乎是没什么问题, 但是当我们应用以后， prometheus 却开始报错了.

1
2
3

level=error ts=2021-04-01T07:03:28.234Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"product\""
level=error ts=2021-04-01T07:03:28.234Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"product\""
level=info ts=2021-04-01T07:03:28.271Z caller=main.go:833 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

有点奇怪, 不对劲，为什么没有权限？

定位原因

首先确定了一下prometheus pod所使用的service account
1
serviceAccountName: prometheus-k8s

通过这个serviceAccountName去找对应的binding关系

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-k8s
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring

找到这个clusterrole查看究竟

{
   "apiVersion": "rbac.authorization.k8s.io/v1",
   "kind": "ClusterRole",
   "metadata": {
      "name": "prometheus-k8s"
   },
   "rules": [
      {
         "apiGroups": [
            ""
         ],
         "resources": [
            "nodes/metrics"
         ],
         "verbs": [
            "get"
         ]
      },
      {
         "nonResourceURLs": [
            "/metrics"
         ],
         "verbs": [
            "get"
         ]
      }
   ]
}

我的天, 原来真的没有权限, 但是其他的servicemonitor为何是正常的? 这个问题困惑了我2小时, 一直在想为何仅仅有这点权限其他的监控端口竟然能正常工作。

为了确认,我去翻了翻kube-prometheus的源码.

clusterRole:
     local clusterRole = k.rbac.v1.clusterRole;
     local policyRule = clusterRole.rulesType;

     local nodeMetricsRule = policyRule.new() +
                             policyRule.withApiGroups(['']) +
                             policyRule.withResources(['nodes/metrics']) +
                             policyRule.withVerbs(['get']);

     local metricsRule = policyRule.new() +
                         policyRule.withNonResourceUrls('/metrics') +
                         policyRule.withVerbs(['get']);

作者从一开始就没有创建cluster的等级的权限.

没办法, 为了先把功能完成,我只能把权限给完整一下.

更改为

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

更改以后, prometheus就可以正常的拉取了.

修改标签

不过, 在grafana查看的时候, IP后面总是跟着端口有点不舒服,所以稍微改一下servicemonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: external-node-exporter
  namespace: product
spec:
  endpoints:
    - interval: 15s
      port: http-metrics
      relabelings:
       - action: replace
         regex: (.*):(.*)
         replacement: $1
         sourceLabels:
          - __address__
         targetLabel: instance
  selector:
    matchLabels:
      app: external-node-exporter

后记

但是！还没完.

当我把一切准备就绪的时候. 本着疑问, 又去翻了翻源码,忽然发现，下面还有两段代码我没有注意到.

roleSpecificNamespaces:
  local role = k.rbac.v1.role;
  local policyRule = role.rulesType;
  local coreRule = policyRule.new() +
                   policyRule.withApiGroups(['']) +
                   policyRule.withResources([
                     'services',
                     'endpoints',
                     'pods',
                   ]) +
                   policyRule.withVerbs(['get', 'list', 'watch']);

  local newSpecificRole(namespace) =
    role.new() +
    role.mixin.metadata.withName('prometheus-' + p.name) +
    role.mixin.metadata.withNamespace(namespace) +
    role.withRules(coreRule);

  local roleList = k3.rbac.v1.roleList;
  roleList.new([newSpecificRole(x) for x in p.roleBindingNamespaces]),

roleBindingSpecificNamespaces:
  local roleBinding = k.rbac.v1.roleBinding;

  local newSpecificRoleBinding(namespace) =
    roleBinding.new() +
    roleBinding.mixin.metadata.withName('prometheus-' + p.name) +
    roleBinding.mixin.metadata.withNamespace(namespace) +
    roleBinding.mixin.roleRef.withApiGroup('rbac.authorization.k8s.io') +
    roleBinding.mixin.roleRef.withName('prometheus-' + p.name) +
    roleBinding.mixin.roleRef.mixinInstance({ kind: 'Role' }) +
    roleBinding.withSubjects([{ kind: 'ServiceAccount', name: 'prometheus-' + p.name, namespace: p.namespace }]);

  local roleBindingList = k3.rbac.v1.roleBindingList;
  roleBindingList.new([newSpecificRoleBinding(x) for x in p.roleBindingNamespaces]),

第一个创建了role, 并且赋予了services,endpoints,pods资源的get, list, watch 权限.

这不正是我们prometheus需要的吗? 继续往下看.看到他引用了local roleList = k3.rbac.v1.roleList 准备创建 kind = ruleList 的资源.

p.roleBindingNamespaces 往上找到这个数组

name:: $._config.prometheus.name,
namespace:: $._config.namespace,
roleBindingNamespaces:: $._config.prometheus.namespaces,
replicas:: $._config.prometheus.replicas,
prometheusRules:: $._config.prometheus.rules,
alertmanagerName:: $.alertmanager.service.metadata.name,

这个配置是从config里来的,我们继续往上找

prometheus+:: {
      name: 'k8s',
      replicas: 2,
      rules: {},
      namespaces: ['default', 'kube-system', $._config.namespace],
},

终于真相大白了.

kube-prometheus 的作者, 默认定义了 default,kube-system, 以及用户和他默认写好的$._config.namespace=monitoring这3个名称空间.

换句话来说, Prometheus的权限只允许监控或者说拿到这3个名称空间的servicemonitor的配置.

总结

如果想监控除了 default，kube-system，$._config.namespace=monitoring，需要创建该名称空间对应的role.
创建role有两种方法
- 使用role，分别在需要的名称空间创建
- 使用clusterrole，创建集群等级的权限
最重要的还是分析问题要细致, 因为少看了几行, 结果白折腾1天.

原文作者：Momo

原文链接：https://mo.xmomo521.top/2021/04/01/prometheus-operator使用外部的端口/

发表日期：April 1st 2021, 5:34:05 pm

更新日期：December 21st 2021, 3:32:58 pm

Next Post

重新规划了监控的架构Thanos+Prometheus
Previous Post

备份Grafana仪表盘

CATALOG

1. 前言
2. 使用外部的metrics
1. 2.1. 创建service
  1. 2.1.1. 创建endpoints
  2. 2.1.2. 创建service
2. 2.2. 创建ServiceMonitor
  1. 2.2.1. 定位原因
  2. 2.2.2. 修改标签
3. 后记
4. 总结

