服务器上mysqld，java的进程Out of Memory,被kernel kill 掉了
/var/log/messages 里面日志如下
Aug 10 19:47:16 VM-0-7-centos kernel: 8936 total pagecache pages Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages in swap cache Aug 10 19:47:16 VM-0-7-centos kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 10 19:47:16 VM-0-7-centos kernel: Free swap = 0kB Aug 10 19:47:16 VM-0-7-centos kernel: Total swap = 0kB Aug 10 19:47:16 VM-0-7-centos kernel: 2097016 pages RAM Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages HighMem/MovableOnly Aug 10 19:47:16 VM-0-7-centos kernel: 94855 pages reserved Aug 10 19:47:16 VM-0-7-centos kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Aug 10 19:47:16 VM-0-7-centos kernel: [13357] 27 13357 1244419 814184 2246 0 0 mysqld Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13357 (mysqld) score 407 or sacrifice child Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13357 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3256736kB, file-rss:0kB, shmem-rss:0kB Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas cpuset=/ mems_allowed=0 Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 0 PID: 18863 Comm: VM Periodic Tas Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1 Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014 Aug 10 19:47:16 VM-0-7-centos kernel: Call Trace: Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13388 (mysqld) score 407 or sacrifice child Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13388 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3257420kB, file-rss:0kB, shmem-rss:0kB Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 cpuset=/ mems_allowed=0 Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 1 PID: 26485 Comm: logback-3 Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1 Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014 Aug 10 19:47:16 VM-0-7-centos kernel: [18851] 1000 18851 1442473 397376 1068 0 0 java Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18851 (java) score 199 or sacrifice child Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18851 (java), UID 1000, total-vm:5769892kB, anon-rss:1589504kB, file-rss:0kB, shmem-rss:0kB Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0 Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1 Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014 Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18855 (VM Thread) score 201 or sacrifice child Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18855 (VM Thread), UID 1000, total-vm:5769892kB, anon-rss:1612704kB, file-rss:0kB, shmem-rss:0kB Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0 Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdum Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18863 (VM Periodic Tas) score 201 or sacrifice child Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service: main process exited, code=killed, status=9/KILL Aug 10 19:47:16 VM-0-7-centos systemd: Unit mysqld.service entered failed state. Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service failed. Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service holdoff time over, scheduling restart. Aug 10 19:47:16 VM-0-7-centos systemd: Stopped MySQL Server. Aug 10 19:47:16 VM-0-7-centos systemd: Starting MySQL Server... Aug 10 19:47:26 VM-0-7-centos systemd: Started MySQL Server.
可以看到一开始是： VM Periodic Tas invoked oom-killer ，而VM Periodic Tag可能是JVM线程的东东。 Killed process 13357 (mysqld) 是指把分数最不理想的mysqld给kill了
后面则是 logback-3 invoked oom-killer ，killed process 13388 (mysqld)。再后面则是node invoked oom-killer. 自从ssr之后，对后台的压力增加了。
不过看监控在发生oom-killer时系统内存占用大概是6.8G左右，还有空闲。但还是发生了。而且看日志之前已经发生多次。
看网上的解释：The “OOM Killer” or “Out of Memory Killer” is a process that the Linux kernel employs when the system is critically low on memory.
看网上有人说建立swap分区会好一些，那我只好建了一个400M的swap分区文件。
喜欢艺术的码农