0. 概述

最近有个统计 Nginx 的需求,按照以前 ETL 的操作步骤来说,可能处理方式是写一个 Python 项目,然后解析一下 Nginx 的访问日志,然后暴露统计数据接口就可以了。但是,本着不作不舒服的原则,我想让 nginx 自己暴露这个功能,于是就有了这篇文章。

要实现这个目标,第一步还是要先能编译 Nginx,然后是在编译的时候加入 Lua 支持,最后,就是插入 prometheus 监控的 lua 模块了,具体的操作流程如下。

1. 安装 LuaJIT

[[email protected]]# wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
[[email protected]]# tar zxf LuaJIT-2.0.5.tar.gz
[[email protected]]# cd LuaJIT-2.0.5
[[email protected]]# make install PREFIX=/usr/local/LuaJIT
[[email protected]]# export LUAJIT_LIB=/usr/local/LuaJIT/lib
[[email protected]]# export LUAJIT_INC=/usr/local/LuaJIT/include/luajit-2.0
[[email protected]]# export LD_LIBRARY_PATH=/usr/local/LuaJIT/lib/:$LD_LIBRARY_PATH

2. 安装各种扩展支持

[[email protected]]# wget https://github.com/openresty/echo-nginx-module/archive/v0.61.tar.gz
[[email protected]]# tar zxf v0.61.tar.gz
[[email protected]]# wget https://ftp.pcre.org/pub/pcre/pcre-8.43.tar.gz
[[email protected]]# tar zxf pcre-8.43.tar.gz
[[email protected]]# cd pcre-8.43 
[[email protected]]# ./configure && make && make install
[[email protected]]# wget https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1.tar.gz
[[email protected]]# tar zxf v0.3.1.tar.gz
[[email protected]]# wget https://github.com/openresty/lua-nginx-module/archive/v0.10.15.tar.gz
[[email protected]]# tar zxf v0.10.15.tar.gz
[[email protected]]# wget http://zlib.net/zlib-1.2.11.tar.gz
[[email protected]]# tar zxf zlib-1.2.11.tar.gz
[[email protected]]# cd zlib-1.2.11
[[email protected]]# ./configure && make && make install

3. 编译 Nginx

[[email protected]]# wget http://nginx.org/download/nginx-1.17.7.tar.gz
[[email protected]]# tar zxf nginx-1.17.7.tar.gz
[[email protected]]# cd nginx-1.17.7
[[email protected]]# ./configure \
--prefix=/usr/share/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib64/nginx/modules  \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/run/nginx.pid \
--with-file-aio --with-http_ssl_module \
--user=nginx --group=nginx \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-http_realip_module \
--with-pcre=/home/liuliqiang/soft/installed/network/pcre-8.43 \
--with-stream --with-stream_ssl_module \
--with-http_auth_request_module \
--add-module=/home/liuliqiang/soft/installed/network/echo-nginx-module-0.61 \
--add-module=/home/liuliqiang/soft/installed/network/ngx_devel_kit-0.3.1 \
--add-module=/home/liuliqiang/soft/installed/network/lua-nginx-module-0.10.15
... ...
Configuration summary
  + using PCRE library: /home/liuliqiang/soft/installed/network/pcre-8.43
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"
[[email protected]]# make -j 4

至此,Nginx 就算是编译完毕了。这里省略了 N 多内容,请注意上面的各个细节,错过一点可能都会有问题。这里的编译参数可以根据个人的进行调整,如果你不知道需要什么选项,有一些方法可以帮助你选择,一个是:

[[email protected]]# ./configure --help

不过这种方法选择太多,而且专业性也比较强,会让你迷失,我更推荐的一种方式是找到官方编译的,你正在使用中的版本,然后安装它,再通过这条命令来查看:

[[email protected]]#  /usr/sbin/nginx -V
nginx version: nginx/1.16.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-stream_ssl_preread_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-http_auth_request_module --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

直接复制这里的 Configure Arguments 就好了,因为你已经是适配了这个编译选项了,所以最终编译出来的 Nginx 用得也会更加顺手一些。

4. 添加 prometheus 监控

接着就是添加 Prometheus 的 Lua 支持了。这里大概有三个要点需要注意,分别是:

4.1 添加 Nginx 设置

在 Nginx 的 http 配置段中加入这么一段:

[[email protected]]# cat nginx.conf
...
http {
...
    lua_shared_dict prometheus_metrics 10M;
    lua_package_path "/usr/local/nginx/lua/prometheus.lua";
    init_by_lua '
      prometheus = require("prometheus").init("prometheus_metrics")
      metric_requests = prometheus:counter(
        "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
      metric_latency = prometheus:histogram(
        "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
      metric_connections = prometheus:gauge(
        "nginx_http_connections", "Number of HTTP connections", {"state"})
    ';
    log_by_lua '
      metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
    ';
...
}

4.2 添加 Lua 模块

[[email protected]]# wget https://github.com/knyar/nginx-lua-prometheus/archive/0.20181120.tar.gz
[[email protected]]# tar zxf 0.20181120.tar.gz
[[email protected]]# cd nginx-lua-prometheus-0.20181120
[[email protected]]# mkdir -p /usr/local/nginx/lua/
[[email protected]]# cp prometheus.lua /usr/local/nginx/lua/

4.3 添加 Metric 路由

[[email protected]]# cat prometheus.conf
server {
  listen 9145;
  allow 192.168.0.0/16;
  deny all;
  location /metrics {
    content_by_lua '
      metric_connections:set(ngx.var.connections_reading, {"reading"})
      metric_connections:set(ngx.var.connections_waiting, {"waiting"})
      metric_connections:set(ngx.var.connections_writing, {"writing"})
      prometheus:collect()
    ';
  }
}

5. 验证效果

最后,就是验收一下效果啦:

[[email protected]]# curl http://127.0.0.1:9145/metrics
# HELP nginx_http_connections Number of HTTP connections
# TYPE nginx_http_connections gauge
nginx_http_connections{state="reading"} 0
nginx_http_connections{state="waiting"} 0
nginx_http_connections{state="writing"} 1
# HELP nginx_http_request_duration_seconds HTTP request latency
# TYPE nginx_http_request_duration_seconds histogram
nginx_http_request_duration_seconds_bucket{host="",le="00.005"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.010"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.020"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.030"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.050"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.075"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.100"} 2
nginx_http_request_duration_seconds_bucket{host="",le="00.200"} 2

6. Ref