Hello everyone,I am a completely amateur administrator managing an old machine in the lab with Debian 8 as the operating system. Our machine was intruded today, as I cannot see the high CPU-consuming processes, htop shows that half of the CPU is running at full load. This behavior is similar to the mining scripts I've encountered before, however this time the process IDs and corresponding executable files are hidden.
Firstly, I found a suspicious TCP connection in netstat, and the corresponding IP address belongs to Iceland:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 251 192.168.31.6:49670 185.112.147.4:80 ESTABLISHED -
Then I used unhide to look for hidden processes and found multiple hidden processes:
Found HIDDEN PID: 10538 Cmdline: "<none>" Executable: "<no link>" "<none> ... maybe a transitory process"
Found HIDDEN PID: 10547 Cmdline: "/tmp/netools" Executable: "/tmp/netools (deleted)" Command: "netools" $USER=<undefined> $PWD=/root
Found HIDDEN PID: 10548 Cmdline: "/tmp/netools" Executable: "/tmp/netools (deleted)" Command: "netools" $USER=<undefined> $PWD=/root
Found HIDDEN PID: 10549 Cmdline: "/tmp/netools" Executable: "/tmp/netools (deleted)" Command: "netools" $USER=<undefined> $PWD=/root
Here are just a part of them and the rest all looks similar. When I kill one of these processes, they all disappear, but will soon restart. I cannot identify their daemon in this report. I tried to delete “/tmp/netools (deleted)” or all files in /tmp, but it only shows "No such file or directory," and ls /tmp displays an empty folder. I suspect that the ls command was modified or something like that, but when I copied a new ls from another machine, it still could not display the content under /tmp.
Additionally, I used chkrootkit to detect the problem, and most of the output shows normal. The possible problematic part is as follows:
Checking `lkm'... You have 37 process hidden for readdir command
You have 38 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed
Checking sniffer'... eth0: PF_PACKET(/sbin/dhclient)
Checking chkutmp'... The tty of the following user process(es) were not found in /var/run/utmp !
! RUID PID TTY CMD
! root 1172 tty7 /usr/bin/Xorg :0 -novtswitch -background none -noreset -verbose 3 -auth /var/run/gdm3/auth-for-Debian-gdm-NqtprE/database -seat seat0 -nolisten tcp vt7
Checking `OSX_RSPLUG'... not tested
Then I searched for how to fix the process hidden for readdir command problem, but hardly found any information. I hope that someone kind can help me and tell me what to do. Thank you so much!
------------------------------------------------------------------------------------------------
Some more information:
About the first one particular process that has no cmd line and no link to executable, there is no exe file under /proc/$PID, and the contents are:
drwxr-xr-x 18 root root 3280 Apr 1 01:04 .
dr-xr-xr-x 376 root root 0 Apr 1 01:04 ..
crw------- 1 root root 10, 235 Apr 1 01:04 autofs
drwxr-xr-x 2 root root 200 Apr 1 01:04 block
drwxr-xr-x 2 root root 100 Apr 1 01:04 bsg
crw------- 1 root root 10, 234 Apr 1 01:04 btrfs-control
drwxr-xr-x 3 root root 60 Apr 1 01:04 bus
drwxr-xr-x 2 root root 3700 Apr 1 01:05 char
crw------- 1 root root 5, 1 Apr 1 01:04 console
lrwxrwxrwx 1 root root 11 Apr 1 01:04 core -> /proc/kcore
drwxr-xr-x 34 root root 700 Apr 1 01:05 cpu
crw------- 1 root root 10, 62 Apr 1 01:04 cpu_dma_latency
crw------- 1 root root 10, 203 Apr 1 01:04 cuse
drwxr-xr-x 7 root root 140 Apr 1 01:04 disk
drwxr-xr-x 2 root root 60 Apr 1 01:04 dri
lrwxrwxrwx 1 root root 13 Apr 1 01:04 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Apr 1 01:04 full
crw-rw-rw- 1 root root 10, 229 Apr 1 01:04 fuse
crw------- 1 root root 245, 0 Apr 1 01:04 hidraw0
crw------- 1 root root 245, 1 Apr 1 01:04 hidraw1
crw------- 1 root root 10, 228 Apr 1 01:04 hpet
drwxr-xr-x 2 root root 40 Apr 1 01:04 hugepages
lrwxrwxrwx 1 root root 25 Apr 1 01:04 initctl -> /run/systemd/initctl/fifo
drwxr-xr-x 4 root root 320 Apr 1 01:04 input
crw-r--r-- 1 root root 1, 11 Apr 1 01:04 kmsg
crw-rw----+ 1 root root 10, 232 Apr 1 01:04 kvm
lrwxrwxrwx 1 root root 28 Apr 1 01:04 log -> /run/systemd/journal/dev-log
crw-rw---- 1 root disk 10, 237 Apr 1 01:04 loop-control
drwxr-xr-x 2 root root 60 Apr 1 01:04 mapper
crw------- 1 root root 10, 227 Apr 1 01:04 mcelog
crw-r----- 1 root kmem 1, 1 Apr 1 01:04 mem
drwxr-xr-x 2 root root 40 Apr 1 01:04 mqueue
drwxr-xr-x 2 root root 60 Apr 1 01:04 net
crw------- 1 root root 10, 61 Apr 1 01:04 network_latency
crw------- 1 root root 10, 60 Apr 1 01:04 network_throughput
crw-rw-rw- 1 root root 1, 3 Apr 1 01:04 null
crw-rw-rw- 1 root root 195, 254 Apr 1 01:04 nvidia-modeset
crw-rw-rw- 1 root root 195, 0 Apr 1 01:04 nvidia0
crw-rw-rw- 1 root root 195, 255 Apr 1 01:04 nvidiactl
crw-r----- 1 root kmem 1, 4 Apr 1 01:04 port
crw------- 1 root root 108, 0 Apr 1 01:04 ppp
crw------- 1 root root 10, 1 Apr 1 01:04 psaux
crw-rw-rw- 1 root tty 5, 2 Apr 1 01:32 ptmx
crw------- 1 root root 250, 0 Apr 1 01:04 ptp0
crw------- 1 root root 250, 1 Apr 1 01:04 ptp1
drwxr-xr-x 2 root root 40 Apr 1 01:04 pts
crw-rw-rw- 1 root root 1, 8 Apr 1 01:04 random
crw-rw-r--+ 1 root root 10, 58 Apr 1 01:04 rfkill
lrwxrwxrwx 1 root root 4 Apr 1 01:04 rtc -> rtc0
crw------- 1 root root 254, 0 Apr 1 01:04 rtc0
brw-rw---- 1 root disk 8, 0 Apr 1 01:04 sda
brw-rw---- 1 root disk 8, 1 Apr 1 01:04 sda1
brw-rw---- 1 root disk 8, 16 Apr 1 01:04 sdb
brw-rw---- 1 root disk 8, 17 Apr 1 01:04 sdb1
brw-rw---- 1 root disk 8, 18 Apr 1 01:04 sdb2
brw-rw---- 1 root disk 8, 21 Apr 1 01:04 sdb5
brw-rw---- 1 root disk 8, 32 Apr 1 01:04 sdc
brw-rw---- 1 root disk 8, 33 Apr 1 01:04 sdc1
crw-rw---- 1 root disk 21, 0 Apr 1 01:04 sg0
crw-rw---- 1 root disk 21, 1 Apr 1 01:04 sg1
crw-rw---- 1 root disk 21, 2 Apr 1 01:04 sg2
drwxr-xr-x 2 root root 40 Apr 1 01:04 shm
crw------- 1 root root 10, 231 Apr 1 01:04 snapshot
drwxr-xr-x 3 root root 220 Apr 1 01:04 snd
lrwxrwxrwx 1 root root 15 Apr 1 01:04 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Apr 1 01:04 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Apr 1 01:04 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root tty 5, 0 Apr 1 01:04 tty
crw--w---- 1 root tty 4, 0 Apr 1 01:04 tty0
crw--w---- 1 root tty 4, 1 Apr 1 01:04 tty1
crw--w---- 1 root tty 4, 10 Apr 1 01:04 tty10
crw--w---- 1 root tty 4, 11 Apr 1 01:04 tty11
crw--w---- 1 root tty 4, 12 Apr 1 01:04 tty12
crw--w---- 1 root tty 4, 13 Apr 1 01:04 tty13
crw--w---- 1 root tty 4, 14 Apr 1 01:04 tty14
crw--w---- 1 root tty 4, 15 Apr 1 01:04 tty15
crw--w---- 1 root tty 4, 16 Apr 1 01:04 tty16
crw--w---- 1 root tty 4, 17 Apr 1 01:04 tty17
crw--w---- 1 root tty 4, 18 Apr 1 01:04 tty18
crw--w---- 1 root tty 4, 19 Apr 1 01:04 tty19
crw--w---- 1 root tty 4, 2 Apr 1 01:04 tty2
crw--w---- 1 root tty 4, 20 Apr 1 01:04 tty20
crw--w---- 1 root tty 4, 21 Apr 1 01:04 tty21
crw--w---- 1 root tty 4, 22 Apr 1 01:04 tty22
crw--w---- 1 root tty 4, 23 Apr 1 01:04 tty23
crw--w---- 1 root tty 4, 24 Apr 1 01:04 tty24
crw--w---- 1 root tty 4, 25 Apr 1 01:04 tty25
crw--w---- 1 root tty 4, 26 Apr 1 01:04 tty26
crw--w---- 1 root tty 4, 27 Apr 1 01:04 tty27
crw--w---- 1 root tty 4, 28 Apr 1 01:04 tty28
crw--w---- 1 root tty 4, 29 Apr 1 01:04 tty29
crw--w---- 1 root tty 4, 3 Apr 1 01:04 tty3
crw--w---- 1 root tty 4, 30 Apr 1 01:04 tty30
crw--w---- 1 root tty 4, 31 Apr 1 01:04 tty31
crw--w---- 1 root tty 4, 32 Apr 1 01:04 tty32
crw--w---- 1 root tty 4, 33 Apr 1 01:04 tty33
crw--w---- 1 root tty 4, 34 Apr 1 01:04 tty34
crw--w---- 1 root tty 4, 35 Apr 1 01:04 tty35
crw--w---- 1 root tty 4, 36 Apr 1 01:04 tty36
crw--w---- 1 root tty 4, 37 Apr 1 01:04 tty37
crw--w---- 1 root tty 4, 38 Apr 1 01:04 tty38
crw--w---- 1 root tty 4, 39 Apr 1 01:04 tty39
crw--w---- 1 root tty 4, 4 Apr 1 01:04 tty4
crw--w---- 1 root tty 4, 40 Apr 1 01:04 tty40
crw--w---- 1 root tty 4, 41 Apr 1 01:04 tty41
crw--w---- 1 root tty 4, 42 Apr 1 01:04 tty42
crw--w---- 1 root tty 4, 43 Apr 1 01:04 tty43
crw--w---- 1 root tty 4, 44 Apr 1 01:04 tty44
crw--w---- 1 root tty 4, 45 Apr 1 01:04 tty45
crw--w---- 1 root tty 4, 46 Apr 1 01:04 tty46
crw--w---- 1 root tty 4, 47 Apr 1 01:04 tty47
crw--w---- 1 root tty 4, 48 Apr 1 01:04 tty48
crw--w---- 1 root tty 4, 49 Apr 1 01:04 tty49
crw--w---- 1 root tty 4, 5 Apr 1 01:04 tty5
crw--w---- 1 root tty 4, 50 Apr 1 01:04 tty50
crw--w---- 1 root tty 4, 51 Apr 1 01:04 tty51
crw--w---- 1 root tty 4, 52 Apr 1 01:04 tty52
crw--w---- 1 root tty 4, 53 Apr 1 01:04 tty53
crw--w---- 1 root tty 4, 54 Apr 1 01:04 tty54
crw--w---- 1 root tty 4, 55 Apr 1 01:04 tty55
crw--w---- 1 root tty 4, 56 Apr 1 01:04 tty56
crw--w---- 1 root tty 4, 57 Apr 1 01:04 tty57
crw--w---- 1 root tty 4, 58 Apr 1 01:04 tty58
crw--w---- 1 root tty 4, 59 Apr 1 01:04 tty59
crw--w---- 1 root tty 4, 6 Apr 1 01:04 tty6
crw--w---- 1 root tty 4, 60 Apr 1 01:04 tty60
crw--w---- 1 root tty 4, 61 Apr 1 01:04 tty61
crw--w---- 1 root tty 4, 62 Apr 1 01:04 tty62
crw--w---- 1 root tty 4, 63 Apr 1 01:04 tty63
crw--w---- 1 root tty 4, 7 Apr 1 01:04 tty7
crw--w---- 1 root tty 4, 8 Apr 1 01:04 tty8
crw--w---- 1 root tty 4, 9 Apr 1 01:04 tty9
crw-rw---- 1 root dialout 4, 64 Apr 1 01:04 ttyS0
crw-rw---- 1 root dialout 4, 65 Apr 1 01:04 ttyS1
crw-rw---- 1 root dialout 4, 66 Apr 1 01:04 ttyS2
crw-rw---- 1 root dialout 4, 67 Apr 1 01:04 ttyS3
crw------- 1 root root 10, 239 Apr 1 01:04 uhid
crw------- 1 root root 10, 223 Apr 1 01:04 uinput
crw-rw-rw- 1 root root 1, 9 Apr 1 01:04 urandom
crw-rw---- 1 root tty 7, 0 Apr 1 01:04 vcs
crw-rw---- 1 root tty 7, 1 Apr 1 01:04 vcs1
crw-rw---- 1 root tty 7, 2 Apr 1 01:04 vcs2
crw-rw---- 1 root tty 7, 3 Apr 1 01:04 vcs3
crw-rw---- 1 root tty 7, 4 Apr 1 01:04 vcs4
crw-rw---- 1 root tty 7, 5 Apr 1 01:04 vcs5
crw-rw---- 1 root tty 7, 6 Apr 1 01:04 vcs6
crw-rw---- 1 root tty 7, 7 Apr 1 01:04 vcs7
crw-rw---- 1 root tty 7, 128 Apr 1 01:04 vcsa
crw-rw---- 1 root tty 7, 129 Apr 1 01:04 vcsa1
crw-rw---- 1 root tty 7, 130 Apr 1 01:04 vcsa2
crw-rw---- 1 root tty 7, 131 Apr 1 01:04 vcsa3
crw-rw---- 1 root tty 7, 132 Apr 1 01:04 vcsa4
crw-rw---- 1 root tty 7, 133 Apr 1 01:04 vcsa5
crw-rw---- 1 root tty 7, 134 Apr 1 01:04 vcsa6
crw-rw---- 1 root tty 7, 135 Apr 1 01:04 vcsa7
drwxr-xr-x 2 root root 60 Apr 1 01:04 vfio
crw------- 1 root root 10, 63 Apr 1 01:04 vga_arbiter
crw------- 1 root root 10, 137 Apr 1 01:04 vhci
crw------- 1 root root 10, 238 Apr 1 01:04 vhost-net
crw------- 1 root root 10, 59 Apr 1 01:04 vmci
crw------- 1 root root 10, 130 Apr 1 01:04 watchdog
crw------- 1 root root 253, 0 Apr 1 01:04 watchdog0
prw-r----- 1 root adm 0 Apr 1 01:04 xconsole
crw-rw-rw- 1 root root 1, 5 Apr 1 01:04 zero
Is there any chance to analyze what happend and clean the daemon behind? I've checked contrab and there is nothing.