自除夕夜家里的NAS噪声就一直很大，还以为是开了电暖气的缘故，就没在意，今天（初六）突然脑子一抽，上服务器查了一下CPU温度，好家伙，直接86度了。这才觉得事情不简单。
果然，top一看，CPU已经拉满，真是人到中年，丧失警觉性了。
Tasks: 417 total, 1 running, 416 sleeping, 0 stopped, 0 zombie
%Cpu(s): 99.5 us, 0.4 sy, 0.0 ni, 0.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32666988 total, 4505068 free, 15479032 used, 12682888 buff/cache
KiB Swap: 16449532 total, 15792892 free, 656640 used. 14929924 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30693 git 20 0 2463216 2.3g 1292 S 1192 7.4 132:52.85 kthreaddk
31117 git 20 0 2470208 2.3g 8 S 1151 7.4 121:58.45 exe
6922 git 20 0 1262764 843808 4828 S 33.4 2.6 29567:33 gitlab-exporter
7075 gitlab-+ 20 0 2072984 43752 3460 S 9.6 0.1 4618:07 postgres_export
7160 git 20 0 2092912 753212 12080 S 4.0 2.3 12287:59 bundle
7122 gitlab-+ 20 0 8435012 37184 18312 S 3.3 0.1 1566:29 postgres
7207 gitlab-+ 20 0 8422596 19408 13560 S 3.3 0.1 7807:16 postgres 好吧，先kill 9 30693试了一发，果然不是这么容易
先到路由器上禁了NAS的外网，这样挖矿跑不起来，CPU没有负荷。
看上边是git帐号运行的程序，盲猜是gitlab漏洞进来的，先把gitlab停掉gitlab-ctl stop
ls -l /proc/30693查看进程路径，发现程序在执行后已被删除，看来是多进程互相保护之类的技术
total 0
dr-xr-xr-x 2 git git 0 Feb 7 00:29 attr
-rw-r--r-- 1 git git 0 Feb 7 00:29 autogroup
-r-------- 1 git git 0 Feb 7 00:29 auxv
-r--r--r-- 1 git git 0 Feb 7 00:29 cgroup
--w------- 1 git git 0 Feb 7 00:29 clear_refs
-r--r--r-- 1 git git 0 Feb 6 23:59 cmdline
-rw-r--r-- 1 git git 0 Feb 7 00:29 comm
-rw-r--r-- 1 git git 0 Feb 7 00:29 coredump_filter
-r--r--r-- 1 git git 0 Feb 7 00:29 cpuset
lrwxrwxrwx 1 git git 0 Feb 7 00:29 cwd -> /
-r-------- 1 git git 0 Feb 7 00:29 environ
lrwxrwxrwx 1 git git 0 Feb 6 23:59 exe -> /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/679134d83572fc29ffc55c5787c2474a/oacqe5/kthreaddk
dr-x------ 2 git git 0 Feb 7 00:29 fd
dr-x------ 2 git git 0 Feb 7 00:29 fdinfo
-rw-r--r-- 1 git git 0 Feb 7 00:29 gid_map
-r-------- 1 git git 0 Feb 7 00:29 io
-r--r--r-- 1 git git 0 Feb 7 00:29 limits
-rw-r--r-- 1 git git 0 Feb 7 00:29 loginuid
dr-x------ 2 git git 0 Feb 7 00:29 map_files
-r--r--r-- 1 git git 0 Feb 7 00:29 maps
-rw------- 1 git git 0 Feb 7 00:29 mem
-r--r--r-- 1 git git 0 Feb 7 00:29 mountinfo
-r--r--r-- 1 git git 0 Feb 7 00:29 mounts
-r-------- 1 git git 0 Feb 7 00:29 mountstats
dr-xr-xr-x 5 git git 0 Feb 7 00:29 net
dr-x--x--x 2 git git 0 Feb 6 23:59 ns
-r--r--r-- 1 git git 0 Feb 7 00:29 numa_maps
-rw-r--r-- 1 git git 0 Feb 7 00:29 oom_adj
-r--r--r-- 1 git git 0 Feb 7 00:29 oom_score
-rw-r--r-- 1 git git 0 Feb 7 00:29 oom_score_adj
-r--r--r-- 1 git git 0 Feb 7 00:29 pagemap
-r-------- 1 git git 0 Feb 7 00:29 patch_state
-r--r--r-- 1 git git 0 Feb 7 00:29 personality
-rw-r--r-- 1 git git 0 Feb 7 00:29 projid_map
lrwxrwxrwx 1 git git 0 Feb 7 00:29 root -> /
-rw-r--r-- 1 git git 0 Feb 7 00:29 sched
-r--r--r-- 1 git git 0 Feb 7 00:29 schedstat
-r--r--r-- 1 git git 0 Feb 7 00:29 sessionid
-rw-r--r-- 1 git git 0 Feb 7 00:29 setgroups
-r--r--r-- 1 git git 0 Feb 7 00:29 smaps
-r--r--r-- 1 git git 0 Feb 7 00:29 stack
-r--r--r-- 1 git git 0 Feb 6 23:59 stat
-r--r--r-- 1 git git 0 Feb 6 23:59 statm
-r--r--r-- 1 git git 0 Feb 6 23:59 status
-r--r--r-- 1 git git 0 Feb 7 00:29 syscall
dr-xr-xr-x 8 git git 0 Feb 6 23:59 task
-r--r--r-- 1 git git 0 Feb 7 00:29 timers
-rw-r--r-- 1 git git 0 Feb 7 00:29 uid_map
-r--r--r-- 1 git git 0 Feb 7 00:29 wchan再查看crontab文件，发现没有异常，但程序是用git用户运行的，应该查看git用户的crontab文件
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/repositories/sorachen/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/233e394eb6e7257b42dce2b40ce924c2/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/844acd5ccbfc11b5c26c2ca535320059/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/043c682ea5c13e0c879815f6e92c2833/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/faeccba836356a34363092ff7b60af80/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/01d76cb94eb0123a12b271aade238dd0/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/11f18c522da6d097e9d36a6fe5b895ee/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/repositories/foshan/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/909132667da78f4b0000c85175234fea/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /dev/shm/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/42dt33逐一把这些目录禁止执行权限 chmod 600 /xxx
由于它只有git用户的权限，所以跑不出gitlab目录，emmm除了/dev/shm/
再看一下，现在固定在这个目录下面了
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33这时候所有权限都被禁用掉之后，应该会触发系统错误，看一下系统邮件
From [email protected] Mon Feb 7 01:23:01 2022
Return-Path: <[email protected]>
X-Original-To: git
Delivered-To: [email protected]
Received: by file.heibing.org (Postfix, from userid 996)
id 26EFB20B3676; Mon, 7 Feb 2022 01:23:01 +0800 (CST)
From: "(Cron Daemon)" <[email protected]>
To: [email protected]
Subject: Cron <git@file> /home/hfdsoft/gitlab/42dt33
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=66088>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/996>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/opt/gitlab>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=git>
X-Cron-Env: <USER=git>
Message-Id: <[email protected]>
Date: Mon, 7 Feb 2022 01:23:01 +0800 (CST)
/bin/sh: /home/blackice/gitlab/42dt33: Permission denied好了，接下来就是
top 查看一下git用户启动的进程
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17824 git 20 0 63872 48052 4 S 16.1 0.1 2:26.79 yj6s3x
17835 git 20 0 89772 3960 2968 S 0.0 0.0 0:00.00 sendmail
17866 git 20 0 89744 3952 2964 S 0.0 0.0 0:00.00 postdrop
17887 git 20 0 39884 1748 944 S 0.0 0.0 0:00.03 kthreaddk 看一下文件被删除的进程 ls -alR /proc/*/exe 2> /dev/null | grep deleted
[root@file /]# ls -alR /proc/*/exe 2> /dev/null | grep deleted
lrwxrwxrwx 1 git git 0 Feb 7 01:06 /proc/17824/exe -> /dev/shm/yj6s3x (deleted)
lrwxrwxrwx 1 git git 0 Feb 7 01:06 /proc/17887/exe -> /home/blackice/gitlab/backups/uploads/blackice/CPDA/42dt33/kthreaddk (deleted)编写脚本处理
kill 9 17824
kill 9 17887
rm /home/blackice/gitlab/42dt33观察了五分钟，问题不大