[olug] example intrusion detection

[Eric Pierce]

What is a rootkit comprised of?  It sounds like it is some kind of package that
has modified binaries (like 'ps' and 'top' in your case).

Or do they go further than that and have scripts that are run to wipe out log
files, etc.?  Is that what a script kiddie is; someone who gains access to a
system and runs a simple script to "set-up" the computer so they won't be
detected easily (w/o really knowing what they are actually doing)?

Anyway, interesting read.
Eric Pierce

--- Adam Haeder <adamh at omaha.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Recently, I had to do some investigation on a server for a company I do some
> consulting with that was experiencing some odd
> behavior. The only initial indication was that some processes weren't running
> that should be, and attempting to start them
> gave an error (Socket already in use). I did some initial analysis and
> discovered that the system had been compromised and a
> rootkit installed. This will warrant a much longer writeup later, but for the
> time being I thought I'd share some initial
> steps I took to determine exactly what was wrong with the system.
> 
> The first thing that usually happens in a breakin like this is that the log
> files get wiped. However, that's not always the
> case (especially if you're dealing with a script-kiddie) so it's always worth
> while to check there first. `last -a` will give
> you the last logins and where they came from. Also, if you fear a root
> compromise, check the .bash_history file in the /root
> directory. This will log every command that the root user types. This ended
> up helping me out quite a bit, as the intruder
> apparently didn't think about this file or didn't know about it.
> 
> Anyway, since this system did not have any kind of file system integrity tool
> like tripwire or AIDE, I needed to determine what
> (if anything) had been changed on the system. A common occurence in these
> cases is to install some trojaned binaries, that either
> hide certain information or have some sort of back door. Since this was an
> rpm-based distribution (Redhat 9.0), I used the -V
> option to rpm to verify all the installed packages on the system:
> 
> # rpm -Va
> 
> This will report on any file that belongs to an rpm package that was been
> modified since the package was installed. Modified
> can mean different checksum, different owner or group, or different
> permissions. Obviously, some things like configuration
> files are going to show up here, but what we're really concerned about is
> binaries. This search showed that /bin/ps and
> /usr/bin/top where different from the original rpm version. These programs
> are usually trojaned in order to hide the
> existence of certain processes. This system had the 'apt' command installed
> from www.freshrpms.net, so it was a simple
> matter to fix this. First, we find out what rpm package gave us /bin/ps:
> 
> # rpm -q --whatprovides /bin/ps
> procps-3.2.0-1.1
> 
> The we use the apt command to reinstall procps:
> # apt-get install procps --reinstall
> 
> Repeat the above steps for /usr/bin/top.
> 
> A side note: initially, this process failed, because the /bin/ps file was not
> able to be overwritten. It had nothing to do with
> regular filesystem permissions, because I was root and root owned the file.
> However, the filesystem is ext3, and there are some
> ext3 specific flags that can be set. You can view these flags with the
> 'lsattr' command and change them with the 'chattr' command.
> Read the man pages on each command to find out what the flags are. In this
> case, /bin/ps had been set to 'undeletable'. I had to
> first issue this command:
> 
> # chattr -suSiadAc /bin/ps
> 
> to turn all of the ext3 options off. Then I could overwrite it.
> 
> Now that we have verified all of the rpm packages on the system, it's time to
> approach the problem from the other way: we have
> verified that all the files we know SHOULD be on the system are good. How
> about the files on the system that AREN'T part of an
> rpm package? In the absence of a file system integrity tool, this is the only
> way to find out what else might have been installed
> on the system.
> 
> A note on timestamps: some of you may be saying "Just use the find command
> and search for files with a creation date of the last
> week or so". That doesn't work, because most of the rootkits and trojans that
> get installed have a creation date months or years
> in the past. So you can't trust timestamps.
> 
> The '--whatprovides' option to rpm will return "File /bin/whatever is not
> owned by any package" if that file isn't associated
> with an rpm. So we can run this command against every file on the system to
> determine which files are not owned by an rpm package.
> Now, before we start this, we have to know a little bit more about the
> system. Obviously, if it has a large amount of data on it,
> none of those files are going to owned by an rpm package. So we're not going
> to test every single file. We're going to look in the
> most likely places, and ignore the stuff we know is data. Looking back on
> this, I really need to make this command smarter.
> If we're just looking for binaries that don't belong, we can narrow our
> search to that and not waste so much of rpm's time.
> Anyway, here is the command I used:
> 
> # for file in `find usr/ -print`; { rpm -q --whatprovides /$file; } | grep
> "not owned by any package"
> 
> I did this for the directories /usr, /sbin, /bin, /boot, and /lib. It told me
> what was not associated with any rpm package.
> I ended up finding a number of files that weren't supposed to be there,
> including a completely different ssh server.
> 
> Did that find every potential malicious file on the system? Nope. Something
> nasty could be lurking in /home or /dev or /etc.
> Now that I trust /bin/ps, I can examine the process list and ensure that I
> know exactly what each of the running processes is for.
> 
> So I've verified all the rpms, and I've done my best to examine the files on
> the system that aren't in an rpm package. The final
> step? The kernel. `lsmod` didn't show me any modules that I was unsure of,
> but I'm not familiar with the various rootkits, so I
> can no longer trust this kernel. It's due for an upgrade anyway, it's
> 2.4.20-8, which has a local root vulnerability (which is how
> I'm assuming the attacker got root in the first place). So I use apt-get to
> upgrade to the latest kernel for RH9 (because `apt-get
> upgrade` will do all packages EXCEPT kernel packages) and then I reboot to
> the new kernel.
> 
> Is this good enough? No way. The only option here is to reload the system
> from scratch. However, these steps at least gave me
> some confidence that nothing is currently running on the system that I don't
> know about.
> 
> So exactly what happened? I'm not 100% sure. I know that originally the
> system was running an old version of apache, which is how
> I assume the initial attacker got in. Or just simple password guessing; there
> are a lot of accounts on this box and ssh was
> allowed from everywhere. The logs didn't show anything of this nature, but we
> already discussed the trustworthiness of the logs.
> Once in, a local user could download an exploit for the older kernel and use
> that to get root.
> 
> Steps I took:
> - - upgraded apache to the latest version
> - - verified the rpms
> - - made a stab at searching for non-rpm-owned files
> - - restricted ssh to their local network and one of my external IPs (so I
> could still log in)
> - - upgraded the kernel
> 
> The system was only down for the time that I rebooted the kernel. Unless the
> attacker is using something other than ssh to log
> in, I don't know how else they could get in now, seeing as all packages are
> up to date. All in all, an interesting experience.
> I've scheduled a complete reload of this box for this week.
> 
> Anyone else have interesting forensic analysis stories to share?
> 
> - --
> Adam Haeder
> Vice President of Information Technology
> AIM Institute
> adamh at omaha.org
> (402) 345-5025 x115
> PGP Public key: http://www.haederfamily.org/pgp.html
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
> 
> iD8DBQFBYaqubHC3IXlHqBQRAs3gAJ0ct6r8gANXGeSZpRbDBvVlCDzzEgCfeSiZ
> sAkRYjzfm4uE4bPDnlHwRMc=
> =93CG
> -----END PGP SIGNATURE-----
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> http://lists.olug.org/mailman/listinfo/olug
> 



		
_______________________________
Do you Yahoo!?
Declare Yourself - Register online to vote today!
http://vote.yahoo.com