1. 查看 cgroup kill 的记录

[root@liqiang.io 11:39:35 ~]$ sudo dmesg | grep -i kill 
[15131.672861] prometheus invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[15131.672892]  [<ffffffff81188094>] oom_kill_process+0x254/0x3d0
[15131.672911] Task in /system.slice/system-prometheus.slice killed as a result of limit of /system.slice/system-prometheus.slice
[15131.673051] Memory cgroup out of memory: Kill process 27980 (job-center) score 865 or sacrifice child
[15131.673089] Killed process 27696 (job-center) total-vm:2049880kB, anon-rss:923484kB, file-rss:9756kB, shmem-rss:0kB

这里可以查看到因为哪条限制被 kill 了,不过这个前面的时间没法看,因为这个是距离系统运行的时间的秒数,所以,如果需要让这个时间友好化,可以尝试这条命令:

[root@liqiang.io 11:39:35 ~]$ dmesg | grep -i kill | sed -r 's#^\[([0-9]+\.[0-9]+)\](.*)#echo -n "[";echo -n $(date --date="@$(echo "$(grep btime /proc/stat|cut -d " " -f 2)+\1" | bc)" +"%c");echo -n "]";echo -n "\2"#e'
[Tue 23 Apr 2019 01:14:48 PM CST] prometheus invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[Tue 23 Apr 2019 01:14:48 PM CST] [<ffffffff81188094>] oom_kill_process+0x254/0x3d0
[Tue 23 Apr 2019 01:14:48 PM CST] Task in /system.slice/system-prometheus.slice killed as a result of limit of /system.slice/system-prometheus.slice
[Tue 23 Apr 2019 01:14:48 PM CST] Memory cgroup out of memory: Kill process 27980 (job-center) score 865 or sacrifice child
[Tue 23 Apr 2019 01:14:48 PM CST] Killed process 27696 (job-center) total-vm:2049880kB, anon-rss:923484kB, file-rss:9756kB, shmem-rss:0kB

2. 查看一个进程被哪些 cgroup 限制

[root@liqiang.io 11:39:35 ~]$ ps aux | grep prometheus | grep -v grep | awk {'print $2'}
23654
[root@liqiang.io 11:39:35 ~]$ cat /proc/23654/cgroup
11:hugetlb:/
10:cpuset:/kubernetes/app
9:freezer:/
8:perf_event:/
7:cpuacct,cpu:/system.slice/system-prometheus.slice
6:net_prio,net_cls:/
5:memory:/system.slice/system-prometheus.slice
4:blkio:/system.slice/system-prometheus.slice
3:devices:/system.slice/system-prometheus.slice
2:pids:/system.slice/system-prometheus.slice
1:name=systemd:/system.slice/system-prometheus.slice/prometheus.service