kthreaddk挖矿木马清除手记

自除夕夜家里的NAS噪声就一直很大,还以为是开了电暖气的缘故,就没在意,今天(初六)突然脑子一抽,上服务器查了一下CPU温度,好家伙,直接86度了。这才觉得事情不简单。

果然,top一看,CPU已经拉满,真是人到中年,丧失警觉性了。

Tasks: 417 total,   1 running, 416 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.5 us,  0.4 sy,  0.0 ni,  0.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32666988 total,  4505068 free, 15479032 used, 12682888 buff/cache
KiB Swap: 16449532 total, 15792892 free,   656640 used. 14929924 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                      
30693 git       20   0 2463216   2.3g   1292 S  1192  7.4 132:52.85 kthreaddk                                                                    
31117 git       20   0 2470208   2.3g      8 S  1151  7.4 121:58.45 exe                                                                          
 6922 git       20   0 1262764 843808   4828 S  33.4  2.6  29567:33 gitlab-exporter                                                              
 7075 gitlab-+  20   0 2072984  43752   3460 S   9.6  0.1   4618:07 postgres_export                                                              
 7160 git       20   0 2092912 753212  12080 S   4.0  2.3  12287:59 bundle                                                                       
 7122 gitlab-+  20   0 8435012  37184  18312 S   3.3  0.1   1566:29 postgres                                                                     
 7207 gitlab-+  20   0 8422596  19408  13560 S   3.3  0.1   7807:16 postgres                         

好吧,先kill 9 30693试了一发,果然不是这么容易

先到路由器上禁了NAS的外网,这样挖矿跑不起来,CPU没有负荷。

看上边是git帐号运行的程序,盲猜是gitlab漏洞进来的,先把gitlab停掉gitlab-ctl stop

ls -l /proc/30693查看进程路径,发现程序在执行后已被删除,看来是多进程互相保护之类的技术

total 0
dr-xr-xr-x 2 git git 0 Feb  7 00:29 attr
-rw-r--r-- 1 git git 0 Feb  7 00:29 autogroup
-r-------- 1 git git 0 Feb  7 00:29 auxv
-r--r--r-- 1 git git 0 Feb  7 00:29 cgroup
--w------- 1 git git 0 Feb  7 00:29 clear_refs
-r--r--r-- 1 git git 0 Feb  6 23:59 cmdline
-rw-r--r-- 1 git git 0 Feb  7 00:29 comm
-rw-r--r-- 1 git git 0 Feb  7 00:29 coredump_filter
-r--r--r-- 1 git git 0 Feb  7 00:29 cpuset
lrwxrwxrwx 1 git git 0 Feb  7 00:29 cwd -> /
-r-------- 1 git git 0 Feb  7 00:29 environ
lrwxrwxrwx 1 git git 0 Feb  6 23:59 exe -> /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/679134d83572fc29ffc55c5787c2474a/oacqe5/kthreaddk
dr-x------ 2 git git 0 Feb  7 00:29 fd
dr-x------ 2 git git 0 Feb  7 00:29 fdinfo
-rw-r--r-- 1 git git 0 Feb  7 00:29 gid_map
-r-------- 1 git git 0 Feb  7 00:29 io
-r--r--r-- 1 git git 0 Feb  7 00:29 limits
-rw-r--r-- 1 git git 0 Feb  7 00:29 loginuid
dr-x------ 2 git git 0 Feb  7 00:29 map_files
-r--r--r-- 1 git git 0 Feb  7 00:29 maps
-rw------- 1 git git 0 Feb  7 00:29 mem
-r--r--r-- 1 git git 0 Feb  7 00:29 mountinfo
-r--r--r-- 1 git git 0 Feb  7 00:29 mounts
-r-------- 1 git git 0 Feb  7 00:29 mountstats
dr-xr-xr-x 5 git git 0 Feb  7 00:29 net
dr-x--x--x 2 git git 0 Feb  6 23:59 ns
-r--r--r-- 1 git git 0 Feb  7 00:29 numa_maps
-rw-r--r-- 1 git git 0 Feb  7 00:29 oom_adj
-r--r--r-- 1 git git 0 Feb  7 00:29 oom_score
-rw-r--r-- 1 git git 0 Feb  7 00:29 oom_score_adj
-r--r--r-- 1 git git 0 Feb  7 00:29 pagemap
-r-------- 1 git git 0 Feb  7 00:29 patch_state
-r--r--r-- 1 git git 0 Feb  7 00:29 personality
-rw-r--r-- 1 git git 0 Feb  7 00:29 projid_map
lrwxrwxrwx 1 git git 0 Feb  7 00:29 root -> /
-rw-r--r-- 1 git git 0 Feb  7 00:29 sched
-r--r--r-- 1 git git 0 Feb  7 00:29 schedstat
-r--r--r-- 1 git git 0 Feb  7 00:29 sessionid
-rw-r--r-- 1 git git 0 Feb  7 00:29 setgroups
-r--r--r-- 1 git git 0 Feb  7 00:29 smaps
-r--r--r-- 1 git git 0 Feb  7 00:29 stack
-r--r--r-- 1 git git 0 Feb  6 23:59 stat
-r--r--r-- 1 git git 0 Feb  6 23:59 statm
-r--r--r-- 1 git git 0 Feb  6 23:59 status
-r--r--r-- 1 git git 0 Feb  7 00:29 syscall
dr-xr-xr-x 8 git git 0 Feb  6 23:59 task
-r--r--r-- 1 git git 0 Feb  7 00:29 timers
-rw-r--r-- 1 git git 0 Feb  7 00:29 uid_map
-r--r--r-- 1 git git 0 Feb  7 00:29 wchan

再查看crontab文件,发现没有异常,但程序是用git用户运行的,应该查看git用户的crontab文件

[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/repositories/sorachen/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/233e394eb6e7257b42dce2b40ce924c2/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/844acd5ccbfc11b5c26c2ca535320059/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/043c682ea5c13e0c879815f6e92c2833/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/faeccba836356a34363092ff7b60af80/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/01d76cb94eb0123a12b271aade238dd0/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/blackice/CPDA/11f18c522da6d097e9d36a6fe5b895ee/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/repositories/foshan/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/uploads/MaskWord/MaskWordOA/909132667da78f4b0000c85175234fea/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /dev/shm/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/backups/42dt33

逐一把这些目录禁止执行权限 chmod 600 /xxx

由于它只有git用户的权限,所以跑不出gitlab目录,emmm除了/dev/shm/

再看一下,现在固定在这个目录下面了

[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33
[root@file 6301]# crontab -u git -l
* * * * * /home/blackice/gitlab/42dt33

这时候所有权限都被禁用掉之后,应该会触发系统错误,看一下系统邮件

From git@file.heibing.org  Mon Feb  7 01:23:01 2022
Return-Path: <git@file.heibing.org>
X-Original-To: git
Delivered-To: git@file.heibing.org
Received: by file.heibing.org (Postfix, from userid 996)
    id 26EFB20B3676; Mon,  7 Feb 2022 01:23:01 +0800 (CST)
From: "(Cron Daemon)" <git@file.heibing.org>
To: git@file.heibing.org
Subject: Cron <git@file> /home/hfdsoft/gitlab/42dt33
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=66088>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/996>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/opt/gitlab>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=git>
X-Cron-Env: <USER=git>
Message-Id: <20220206172301.26EFB20B3676@file.heibing.org>
Date: Mon,  7 Feb 2022 01:23:01 +0800 (CST)

/bin/sh: /home/blackice/gitlab/42dt33: Permission denied

好了,接下来就是

top 查看一下git用户启动的进程

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                    
17824 git       20   0   63872  48052      4 S  16.1  0.1   2:26.79 yj6s3x                                                                     
17835 git       20   0   89772   3960   2968 S   0.0  0.0   0:00.00 sendmail                                                                   
17866 git       20   0   89744   3952   2964 S   0.0  0.0   0:00.00 postdrop                                                                   
17887 git       20   0   39884   1748    944 S   0.0  0.0   0:00.03 kthreaddk                                                                  

看一下文件被删除的进程 ls -alR /proc/*/exe 2> /dev/null | grep deleted

[root@file /]# ls -alR /proc/*/exe 2> /dev/null | grep deleted
lrwxrwxrwx 1 git     git   0 Feb  7 01:06 /proc/17824/exe -> /dev/shm/yj6s3x (deleted)
lrwxrwxrwx 1 git     git   0 Feb  7 01:06 /proc/17887/exe -> /home/blackice/gitlab/backups/uploads/blackice/CPDA/42dt33/kthreaddk (deleted)

编写脚本处理

kill 9 17824
kill 9 17887
rm /home/blackice/gitlab/42dt33

观察了五分钟,问题不大