重新编译 Nginx 支持 Prometheus 监控

0. 概述

最近有个统计 Nginx 的需求，按照以前 ETL 的操作步骤来说，可能处理方式是写一个 Python 项目，然后解析一下 Nginx 的访问日志，然后暴露统计数据接口就可以了。但是，本着不作不舒服的原则，我想让 nginx 自己暴露这个功能，于是就有了这篇文章。

要实现这个目标，第一步还是要先能编译 Nginx，然后是在编译的时候加入 Lua 支持，最后，就是插入 prometheus 监控的 lua 模块了，具体的操作流程如下。

1. 前提安装

Ubuntu

[root@liqiang.io]# apt install -y wget make build-essential libssl-dev

CentOS

[root@liqiang.io]#

2. 安装 LuaJIT

[root@liqiang.io]# wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
[root@liqiang.io]# tar zxf LuaJIT-2.0.5.tar.gz
[root@liqiang.io]# cd LuaJIT-2.0.5
[root@liqiang.io]# make install PREFIX=/usr/local/LuaJIT
[root@liqiang.io]# export LUAJIT_LIB=/usr/local/LuaJIT/lib
[root@liqiang.io]# export LUAJIT_INC=/usr/local/LuaJIT/include/luajit-2.0
[root@liqiang.io]# export LD_LIBRARY_PATH=/usr/local/LuaJIT/lib/:$LD_LIBRARY_PATH

3. 安装各种扩展支持

偷懒做法

[root@liqiang.io]# apt-get install libpcre3-dev \
    libssl-dev perl make build-essential curl

麻烦做法

[root@liqiang.io]# wget https://github.com/openresty/echo-nginx-module/archive/v0.61.tar.gz
[root@liqiang.io]# tar zxf v0.61.tar.gz
[root@liqiang.io]# wget https://onboardcloud.dl.sourceforge.net/project/pcre/pcre/8.43/pcre-8.43.tar.gz
[root@liqiang.io]# tar zxf pcre-8.43.tar.gz
[root@liqiang.io]# cd pcre-8.43 
[root@liqiang.io]# ./configure && make && make install
[root@liqiang.io]# cd ..
[root@liqiang.io]# wget https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1.tar.gz
[root@liqiang.io]# tar zxf v0.3.1.tar.gz
[root@liqiang.io]# wget https://github.com/openresty/lua-nginx-module/archive/v0.10.15.tar.gz
[root@liqiang.io]# tar zxf v0.10.15.tar.gz
[root@liqiang.io]# wget http://zlib.net/zlib-1.3.1.tar.gz
[root@liqiang.io]# tar zxf zlib-1.3.1.tar.gz
[root@liqiang.io]# cd zlib-1.3.1
[root@liqiang.io]# ./configure && make && make install
[root@liqiang.io]# git clone https://github.com/chobits/ngx_http_proxy_connect_module.git
[root@liqiang.io]# cd ..

4. 编译 Nginx

[root@liqiang.io]# wget http://nginx.org/download/nginx-1.21.2.tar.gz
[root@liqiang.io]# tar zxf nginx-1.21.2.tar.gz
[root@liqiang.io]# cd nginx-1.21.2
[root@liqiang.io]# ./configure \
--prefix=/usr/share/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib64/nginx/modules  \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/run/nginx.pid \
--with-file-aio --with-http_ssl_module \
--user=nginx --group=nginx \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-http_realip_module \
--with-pcre=/root/nginx/pcre-8.43 \
--with-stream --with-stream_ssl_module --with-stream_ssl_preread_module \
--with-http_auth_request_module \
--add-module=/root/nginx/echo-nginx-module-0.61 \
--add-module=/root/nginx/ngx_devel_kit-0.3.1 \
--add-module=/root/nginx/lua-nginx-module-0.10.15
... ...
Configuration summary
  + using PCRE library: /root/nginx/pcre-8.43
  + using system OpenSSL library
  + using system zlib library
  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"
[root@liqiang.io]# make -j 4

至此，Nginx 就算是编译完毕了。这里省略了 N 多内容，请注意上面的各个细节，错过一点可能都会有问题。这里的编译参数可以根据个人的进行调整，如果你不知道需要什么选项，有一些方法可以帮助你选择，一个是：

[root@liqiang.io]# ./configure --help

不过这种方法选择太多，而且专业性也比较强，会让你迷失，我更推荐的一种方式是找到官方编译的，你正在使用中的版本，然后安装它，再通过这条命令来查看：

[root@liqiang.io]#  /usr/sbin/nginx -V
nginx version: nginx/1.16.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-stream_ssl_preread_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-http_auth_request_module --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

直接复制这里的 Configure Arguments 就好了，因为你已经是适配了这个编译选项了，所以最终编译出来的 Nginx 用得也会更加顺手一些。

5. 添加 prometheus 监控

接着就是添加 Prometheus 的 Lua 支持了。这里大概有三个要点需要注意，分别是：

4.1 添加 Nginx 设置

在 Nginx 的 http 配置段中加入这么一段：

[root@liqiang.io]# cat nginx.conf
...
http {
...
    lua_shared_dict prometheus_metrics 10M;
    lua_package_path "/usr/local/nginx/lua/prometheus.lua";
    init_by_lua '
      prometheus = require("prometheus").init("prometheus_metrics")
      metric_requests = prometheus:counter(
        "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
      metric_latency = prometheus:histogram(
        "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
      metric_connections = prometheus:gauge(
        "nginx_http_connections", "Number of HTTP connections", {"state"})
    ';
    log_by_lua '
      metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
    ';
...
}

4.2 添加 Lua 模块

[root@liqiang.io]# wget https://github.com/knyar/nginx-lua-prometheus/archive/0.20181120.tar.gz
[root@liqiang.io]# tar zxf 0.20181120.tar.gz
[root@liqiang.io]# cd nginx-lua-prometheus-0.20181120
[root@liqiang.io]# mkdir -p /usr/local/nginx/lua/
[root@liqiang.io]# cp prometheus.lua /usr/local/nginx/lua/

4.3 添加 Metric 路由

[root@liqiang.io]# cat prometheus.conf
server {
  listen 9145;
  allow 192.168.0.0/16;
  deny all;
  location /metrics {
    content_by_lua '
      metric_connections:set(ngx.var.connections_reading, {"reading"})
      metric_connections:set(ngx.var.connections_waiting, {"waiting"})
      metric_connections:set(ngx.var.connections_writing, {"writing"})
      prometheus:collect()
    ';
  }
}

5. 验证效果

最后，就是验收一下效果啦：

[root@liqiang.io]# curl http://127.0.0.1:9145/metrics
# HELP nginx_http_connections Number of HTTP connections
# TYPE nginx_http_connections gauge
nginx_http_connections{state="reading"} 0
nginx_http_connections{state="waiting"} 0
nginx_http_connections{state="writing"} 1
# HELP nginx_http_request_duration_seconds HTTP request latency
# TYPE nginx_http_request_duration_seconds histogram
nginx_http_request_duration_seconds_bucket{host="",le="00.005"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.010"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.020"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.030"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.050"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.075"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.100"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.200"} 2

格物致知

All Posts