Because I am familiar with the Prometheus model, some server software and small applications are usually monitored through Prometheus. So recently I had a new environment that needed to be rebuilt, so I took the opportunity to document the process of installing and configuring prometheus and grafana.

The build involves installing and deploying prometheus and grafana using Docker, mapping the local disk directory (prometheus data can be quite large, so I am using an additional mounted disk), then also including a reverse proxy for Nginx, and a simple configuration of prometheus. service discovery service, provided by replacing the configuration file, because my crawl target is an internal container platform and the target’s ip changes frequently, so I need a simple dynamic change management.

If you just want to learn how to install and deploy Grafana, you can refer to a previous guide I wrote: Installing and deploying Grafana, which covers deploying a systemd-managed of Grafana, as well as steps for using the Nginx proxy.

compose run prometheus & grafana

Install Docker and compose

  1. [root@liqiang.io]# swapoff -a
  2. [root@liqiang.io]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  3. [root@liqiang.io]# yum install -y docker-ce
  4. [root@liqiang.io]# systemctl start docker
  5. [root@liqiang.io]# systemctl enable docker
  6. [root@liqiang.io]# sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/sbin/docker-compose
  7. [root@liqiang.io]# sudo chmod +x /usr/local/sbin/docker-compose

Environment prepare

  1. [root@liqiang.io]# mkdir -p /data/monitoring/{software,configs,grafana,prometheus}
  2. [root@liqiang.io]# mkdir -p /data/monitoring/configs/{grafana,prometheus}
  3. [root@liqiang.io]# cd /data/monitoring/software && tar -xvf prometheus-2.42.0.linux-amd64.tar.gz && ln -s prometheus-2.42.0.linux-amd64 prometheus
  4. [root@liqiang.io]# cp /data/monitoring/software/prometheus/prometheus.yml /data/monitoring/configs/prometheus/prometheus.yml
  5. [root@liqiang.io]# cat > /data/monitoring/configs/grafana/grafana.ini <<EOF
  6. [server]
  7. protocol = http
  8. http_addr =
  9. http_port = 3000
  10. domain = grafana.liqiang.io
  11. enforce_domain = true
  12. serve_from_sub_path = true
  13. root_url = http://%(domain)s:%(http_port)s/grafana
  14. EOF

Run prometheus & grafana

  1. [root@liqiang.io]# cat <<EOF >/data/monitoring/software/docker-compose.yaml
  2. version: "3.7"
  3. services:
  4. prometheus:
  5. image: prom/prometheus:latest
  6. user: "${UID}:${GID}"
  7. volumes:
  8. - /data/monitoring/configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
  9. - /data/monitoring/prometheus:/prometheus
  10. command:
  11. - '--web.enable-lifecycle'
  12. - '--config.file=/etc/prometheus/prometheus.yml'
  13. - '--storage.tsdb.path=/prometheus'
  14. - '--web.console.libraries=/usr/share/prometheus/console_libraries'
  15. - '--web.console.templates=/usr/share/prometheus/consoles'
  16. ports:
  17. - 9090:9090
  18. grafana:
  19. image: grafana/grafana:latest
  20. user: "${UID}:${GID}"
  21. volumes:
  22. - /data/monitoring/configs/grafana/grafana.ini:/etc/grafana/grafana.ini
  23. - /data/monitoring/grafana:/var/lib/grafana
  24. ports:
  25. - 3000:3000
  26. links:
  27. - prometheus
  28. EOF
  29. [root@liqiang.io]# cd /data/monitoring/software/ && docker-compose up -d

At this point, both Prometheus and Grafana have been installed and deployed, and you can access Grafana via http://localhost:3000. However, I used to use Nginx as a proxy so that I didn’t have to specify an additional port.

Configuring Nginx

Since I already have Nginx installed, I don’t need to install it again. If you don’t have it installed, the easy option is to run another Nginx with docker-compose, or install one directly with Linux’s package management software, for example, I installed it directly with Yum by

  1. [root@liqiang.io]# yum install -y nginx
  2. [root@liqiang.io]# mkdir -p /etc/nginx/grafana
  3. [root@liqiang.io]# cat> /etc/nginx/grafana/grafana.conf << EOF
  4. location /grafana {
  5. proxy_pass;
  6. proxy_set_header X-Real-IP $remote_addr;
  7. proxy_set_header Host $host;
  8. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  9. proxy_http_version 1.1;
  10. proxy_set_header Upgrade $http_upgrade;
  11. proxy_set_header Connection "upgrade";
  12. proxy_redirect off;
  13. }
  14. EOF
  15. [root@liqiang.io]# echo "Please add 'include /etc/nginx/grafana/*.conf;' to your nginx.conf"
  16. [root@liqiang.io]# nginx -t
  17. [root@liqiang.io]# nginx -s reload

Configurare Grafana

Open the browser and use this URL:http://grafana.liqiang.io/grafana,I think you can see this page:

Figure 1:Grafana Login Page

The default username and password for grafana is:

You will then be asked to change your password, it is best to do so.

Service discovery procedures

Prometheus supports a very wide range of service discovery methods, for example:

However, these service discovery features are that the URL Path of the Target you need to collect is fixed and can only be a list of service discovery Endpoints, meaning that you can only use the number of Containers for a specific service discovery and cannot dynamically increase or decrease the type of service.

For example, if you have two services, Order and Account, then Prometheus’ service discovery supports you to dynamically change the number of containers for these two services, for example from 100 to 150. However, if you add a new service, Item, then this service discovery provided by Prometheus is not supported.

So what do we do to dynamically scale the service, here I have chosen to write a program that then dynamically does the service metadata work and then, depending on the actual situation (e.g. the deployment platform of the service), extend this program to do the dynamic discovery of the Endpoint:.

Service Metadata Discovery

There are two key points in the implementation of service metadata discovery.

Since the code is quite long, I won’t post it, but if you are interested you can just open the Github Repo: devops/prometheus/service_discovery to see for yourself~, and in practice, add a service to the end of docker-compose.yaml:

  1. [root@liqiang.io]# cat docker-compose.yaml
  2. ... ...
  3. service-discovery:
  4. image: liqiangliu/prometheus-service-discovery:0.3.0
  5. user: "${UID}:${GID}"
  6. environment:
  7. - PROMETHEUS_ADDR=http://prometheus:9090
  8. volumes:
  9. - /data/monitoring/configs/prometheus:/etc/prometheus
  10. ports:
  11. - 5555:5555

Then try to add a service:

  1. [root@liqiang.io]# curl --location --request POST '' --header 'Content-Type: application/json' --data-raw '{
  2. "name": "prometheus-default",
  3. "endpoints": [
  4. {
  5. "host": "prometheus",
  6. "port": 9090
  7. }
  8. ]
  9. }'


This completes our docker compose based prometheus and grafana monitoring system, and supports a simple service discovery feature, which is simple to apply, so you can extend it yourself.