1. Prometheus 整体架构
2. 监控k8s集群
注意使用image已经保存到了本地registry中,如果使用官方的,可以参见:https://hub.docker.com/r/prom/
2.1 使用k8s ConfigMap管理配置文件
prometheus-kubernetes-configmap.yml 下载
apiVersion: v1 kind: ConfigMap metadata: name: prometheus data: prometheus.yml: |- # A scrape configuration for running Prometheus on a Kubernetes cluster. # This uses separate scrape configs for cluster components (i.e. API server, node) # and services to allow each to use different authentication configs. # # Kubernetes labels will be added as Prometheus labels on metrics via the # `labelmap` relabeling action. # Scrape config for cluster components. scrape_configs: - job_name: 'kubernetes-cluster' # Default to scraping over https. If required, just disable this or change to # `http`. scheme: https # This TLS & bearer token file config is used to connect to the actual scrape # endpoints for cluster components. This is separate to discovery auth # configuration (`in_cluster` below) because discovery & scraping are two # separate concerns in Prometheus. tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt # If your node certificates are self-signed or use a different CA to the # master CA, then disable certificate verification below. Note that # certificate verification is an integral part of a secure infrastructure # so this should only be disabled in a controlled environment. You can # disable certificate verification by uncommenting the line below. # insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc' in_cluster: true role: apiserver - job_name: 'kubernetes-nodes' # Default to scraping over https. If required, just disable this or change to # `http`. scheme: https # This TLS & bearer token file config is used to connect to the actual scrape # endpoints for cluster components. This is separate to discovery auth # configuration (`in_cluster` below) because discovery & scraping are two # separate concerns in Prometheus. tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt # If your node certificates are self-signed or use a different CA to the # master CA, then disable certificate verification below. Note that # certificate verification is an integral part of a secure infrastructure # so this should only be disabled in a controlled environment. You can # disable certificate verification by uncommenting the line below. # insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc' in_cluster: true role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) # Scrape config for service endpoints. # # The relabeling allows the actual service scrape endpoint to be configured # via the following annotations: # # * `prometheus.io/scrape`: Only scrape services that have a value of `true` # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need # to set this to `https` & most likely set the `tls_config` of the scrape config. # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. # * `prometheus.io/port`: If the metrics are exposed on a different port to the # service then set this appropriately. - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc' in_cluster: true role: endpoint relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+)(?::\d+);(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_service_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name # Example scrape config for probing services via the Blackbox Exporter. # # The relabeling allows the actual service scrape endpoint to be configured # via the following annotations: # # * `prometheus.io/probe`: Only probe services that have a value of `true` - job_name: 'kubernetes-services' metrics_path: /probe params: module: [http_2xx] kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc' in_cluster: true role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__address__] target_label: __param_target - target_label: __address__ replacement: blackbox - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_service_namespace] target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] target_label: kubernetes_name # Example scrape config for pods # # The relabeling allows the actual pod scrape endpoint to be configured via the # following annotations: # # * `prometheus.io/scrape`: Only scrape pods that have a value of `true` # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`. - job_name: 'kubernetes-pods' kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc' in_cluster: true role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: (.+):(?:\d+);(\d+) replacement: ${1}:${2} target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_pod_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name
如果是自签名证书注意打开: insecure_skip_verify: true
kubectl create -f prometheus-kubernetes-configmap.yml
2.2 启动
prometheus-deployment.yaml 下载
apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: 'true' labels: name: prometheus name: prometheus spec: selector: app: prometheus type: NodePort ports: - name: prometheus protocol: TCP port: 9090 nodePort: 30900 ---- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: prometheus spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: name: prometheus labels: app: prometheus spec: containers: - name: prometheus image: hub.yinnut.com/prometheus:1.0.1 args: - '-storage.local.retention=6h' - '-storage.local.memory-chunks=500000' - '-config.file=/etc/prometheus/prometheus.yml' ports: - name: web containerPort: 9090 volumeMounts: - name: config-volume mountPath: /etc/prometheus volumes: - name: config-volume configMap: name: prometheus
2.3 指标查看
2.4 使用node exporter监控host更多信息
使用k8s新特性DaemonSet,可以保证pod在每个node上运行
node-exporter.yaml 下载
apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: 'true' labels: app: node-exporter name: node-exporter name: node-exporter spec: clusterIP: None ports: - name: scrape port: 9100 protocol: TCP selector: app: node-exporter type: ClusterIP ---- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: node-exporter spec: template: metadata: labels: app: node-exporter name: node-exporter spec: containers: - image: hub.yinnut.com/node-exporter:0.12.0 name: node-exporter ports: - containerPort: 9100 hostPort: 9100 name: scrape hostNetwork: true hostPID: true
kubectl create -f node-exporter.yaml
如果顺利的话,可以在http://xxxxx:30900/targets 中看到 kubernetes-nodes 的选项卡
3. 其他备注
- http://yourservers_ip:9090/consoles/node.html 显示node_exporter相关
- http://yourservers_ip:9090/targets 显示监控的目标
参考文档:
- 六大下一代Docker监测工具
- How to Use Prometheus to Monitor Your CentOS 7 Server1
- How to Use Prometheus to Monitor Your CentOS 7 Server2
- Prometheus and Kubernetes up and running
- Monitoring your Machines with Prometheus
- Monitoring Blog
- images下载地址 https://hub.docker.com/r/prom/ 二进制文件下载: https://prometheus.io/download/
- prometheus-kubernetes.yml 参考样例
- Go的pprof使用