通过 Grafana Agent 上传 Prometheus 指标数据到 Grafana Cloud

Posted on Word count in article: 4.6k Reading time ≈ 8 mins.

介绍

Grafana Cloud 为免费账户提供了一万条指标的存储额度，对于业余项目来说可以考虑将指标上传到由 Grafana Cloud 托管的 Prometheus 中。

安装 Grafana Agent

Prometheus 指标数据的上传需要通过 Grafana Agent 来完成，以下安装步骤以 Ubuntu 为例：

1
2
3
4
5
mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana-agent

安装完成之后通过 sudo systemctl start grafana-agent 将其启动，并可通过 sudo systemctl status grafana-agent 显示 grafana-agent 的当前状态：

1
2
3
4
5
6
7
8
9
10
11
● grafana-agent.service - Monitoring system and forwarder
     Loaded: loaded (/lib/systemd/system/grafana-agent.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-03-12 05:08:43 UTC; 13s ago
       Docs: https://grafana.com/docs/agent/latest/
   Main PID: 1049084 (grafana-agent)
      Tasks: 7 (limit: 1041)
     Memory: 125.6M
     CGroup: /system.slice/grafana-agent.service
             └─1049084 /usr/bin/grafana-agent --config.file /etc/grafana-agent.yaml -server.http.address=127.0.0.1:9090 -server.grpc.address=127.0.0.1:9091

Mar 12 05:08:43 example-name systemd[1]: Started Monitoring system and forwarder.

同时，如果希望系统重启后自动启动 grafana-agent 服务，可以执行如下的命令：

1
sudo systemctl enable grafana-agent.service

另外，可以通过 sudo journalctl -u grafana-agent 查看 grafana-agent 的运行日志：

1
2
3
4
5
6
7
8
-- Logs begin at Sun 2021-12-26 04:48:21 UTC, end at Sun 2023-03-12 06:34:53 UTC. --
Mar 12 05:08:43 example-name systemd[1]: Started Monitoring system and forwarder.
Mar 12 05:38:45 example-name grafana-agent[1049084]: ts=2023-03-12T05:38:45.6366501Z caller=cleaner.go:203 level=warn agent=prometheus component=cleaner msg="unable to fi>
Mar 12 06:08:45 example-name grafana-agent[1049084]: ts=2023-03-12T06:08:45.63549564Z caller=cleaner.go:203 level=warn agent=prometheus component=cleaner msg="unable to f>
Mar 12 06:20:16 example-name systemd[1]: Stopping Monitoring system and forwarder...
Mar 12 06:20:16 example-name systemd[1]: grafana-agent.service: Succeeded.
Mar 12 06:20:16 example-name systemd[1]: Stopped Monitoring system and forwarder.
Mar 12 06:20:16 example-name systemd[1]: Started Monitoring system and forwarder.

上报监控数据

grafana-agent 上报的监控数据分两种，一种是 grafana-agent 自身及其所在主机的监控数据，另一种是自定义服务的监控数据，我们需要修改 grafana-agent 的配置文件来指定如何收集监控数据。

grafana-agent 的默认配置文件为 /etc/grafana-agent.yaml：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Sample config for Grafana Agent
# For a full configuration reference, see: https://grafana.com/docs/agent/latest/configuration/.
server:
  log_level: warn

metrics:
  global:
    scrape_interval: 1m
  wal_directory: '/var/lib/grafana-agent'
  configs:
    # Example Prometheus scrape configuration to scrape the agent itself for metrics.
    # This is not needed if the agent integration is enabled.
    # - name: agent
    #   host_filter: false
    #   scrape_configs:
    #     - job_name: agent
    #       static_configs:
    #         - targets: ['127.0.0.1:9090']

integrations:
  agent:
    enabled: true
  node_exporter:
    enabled: true
    include_exporter_metrics: true
    disable_collectors:
      - "mdadm"

自定义服务的监控数据收集需要定义在 metrics.configs 下，grafana-agent 自身及其所在主机的监控数据收集默认已经是开启的。

假设需要收集由 Spring Boot 的 actuator 模块所暴露的 Prometheus 监控数据，则需要在 metrics.configs 下新增如下类似配置：

1
2
3
4
5
6
7
8
9
- name: 'My Spring Boot App'
  scrape_configs:
    - job_name: 'My Spring Boot App'
      metrics_path: '/actuator/prometheus'
      scrape_interval: 1m
      static_configs:
        - targets: ['127.0.0.1:8080']
          labels:
            application: 'My Spring Boot App'

最后，再通过 remote_write 设置将监控数据推送到 Grafana Cloud 下的 Prometheus：

1
2
3
4
5
remote_write:
  - url: https://prometheus-xxx.grafana.net/api/prom/push
    basic_auth:
      username: username
      password: password

其中 url，username 和 password 这三个信息都可以在所创建的 Grafana Cloud Stack 下的 Prometheus 的详情页中找到。password 对应 Grafana Cloud API Key，如果之前没有创建过的话需要新生成一个，角色选择 MetricsPublisher 即可：

alt

完整的配置文件示例如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
server:
  log_level: warn

metrics:
  global:
    scrape_interval: 1m
  wal_directory: '/var/lib/grafana-agent'
  configs:
    - name: 'My Spring Boot App'
      scrape_configs:
        - job_name: 'My Spring Boot App'
          metrics_path: '/actuator/prometheus'
          scrape_interval: 1m
          static_configs:
            - targets: ['127.0.0.1:8080']
              labels:
                application: 'My Spring Boot App'
      remote_write:
        - url: https://prometheus-xxx.grafana.net/api/prom/push
          basic_auth:
            username: username
            password: password

integrations:
  agent:
    enabled: true
  node_exporter:
    enabled: true
    include_exporter_metrics: true
    disable_collectors:
      - "mdadm"
  prometheus_remote_write:
    - url: https://prometheus-xxx.grafana.net/api/prom/push
      basic_auth:
        username: username
        password: password

配置文件修改完成之后，通过 sudo systemctl restart grafana-agent 来重启 grafana-agent 服务。

Grafana 展示

对于自定义服务的监控展示使用自己熟悉的方式即可，例如 Java 应用可以配合使用 JVM (Micrometer)。

对于 grafana-agent 自身的监控展示可以结合 agent-remote-write.json：

alt

最后，grafana-agent 对于所在主机的监控展示可以借助 Node Exporter Full：

alt

参考