Normally this sort of thing comes and goes and really doesn’t cause much nuisance, but when they broke the 50,000 page load mark in less than 24 hours I started to get annoyed.

My initial reaction was to complain to the ISP and simply block them with a .htaccess file. I thought about it and realized that this isn’t a very good solution simply because that method still delivers to them my “Go away” page.

So I asked an admin buddy of mine what he thought I should do beyond the .htaccess or iptables. Specifically if there was anything fun we could do back. Something subtle that isn’t a direct hack back, but rather just something to get the message across.

He came up with this;

route add -host {incoming.annoying.bots.ip} gw 127.0.0.1

Evidently this will route the bot directly back to itself ~ localhost. Which just amuses the hell out of me. What’s more is, it will likely take about 5 minutes per hit for the TCP/IP layer to time out. So it will really slow their bot down, which I’m pretty sure the rest of the internet appreciates.

My old server was FedoraCore2.. *shudder* I wasn’t given a choice there.. it was core2 or nothing. And since it was out of date pretty much the day I was given it, it was doomed to be hacked at some point. I knew that going into it.. Eh what can ya do.

Well, it happened. Sometime last week someone rooted the box. They did a pretty thorough job of it too, though the more I looked at it the more I thought it was a script kiddie instead of someone that’s actually clever..

They went through and added a program that listened on port 50030 for some sort of command before it went out and did it’s misdeeds.. Then went through the trouble of replacing every tool that you would normally use to detect these things with their own custom version which hid the running process..

That’s not even the sneaky part.. they also went through and set the attrib of those files to make them undeletable even by root. Annoying.

So instead of trying to save the box, and undo the fuckery that they had installed. I just wiped it. I installed a much later version of the OS and all new stuff.. restored the database, and all things are now happy.

It was a good learning experience and interesting for the most part. It really went a long way to point out that I’m a software engineer, not a server admin. And it really seems like the more you are one, the less you are the other.. I’m trying to learn this stuff, but its a lot of magic to me at this point. I mean there is so much to know, I don’t see how anyone could know it all.

Add to the fact that the internet is by it very nature a warzone and this server/hacker thing is an eternal arms race and I begin to think, why the hell bother? I want a server is the obvious answer..

As always I don’t blame the hacker .. its a game really.. I wish I knew how they got in, not so I could hack other boxes but so I could make it safe on mine — this is something I’ll likely never learn.

So this is what I did once I got a fresh install on the box — I put this here for me so I can do it quickly and easily next time;

less /etc/inittab — make sure we are in rc level 3, if not change it and reboot

chkconfig –list | grep 3:on — Shows run level of services managed through init.d make sure everything is on you want on, and off that you don’t want.

Only do this if you want sendmail off, which you likely wont want because PHP needs it.

chkconfig –level 3 sendmail off — turn off sendmail so it doesn’t start on boot and annoy me constantly
service sendmail stop — turn off sendmail (smtp port 25 is now closed)

cd /etc/cron.daily — turn off the annoying logwatch email spam which is again an annoyance.

netstat -pan | grep LIST — Show a list of all open ports/listeners
nmap localhost — does the same thing, make sure they don’t conflict

useradd [newuser]
passwd [newuser]

visudo — Add [newuser] to the sudoers list
log out as root and relog in as [newuser]

sudo yum install vim — install VIM because I prefer it as my remote editor
sudo vim /etc/ssh/sshd_config — set PermitRootLogin to no so that root can’t SSH into the box

sudo yum list all > ~/rpm-avail.txt — get a list of everything installable, its easier to get a grep on the file

sudo yum -y install httpd php mysql mysql-server php-mysql — Install traditional LAMP setup

sudo /sbin/service mysqld start — Start up mysqld
sudo /sbin/service httpd start — Start up apache

sudo /sbin/chkconfig –level 3 mysqld on — Make sure mysqld starts on reboot
sudo /sbin/chkconfig –level 3 httpd on — Make sure apache starts on reboot

We don’t want people hitting our MySQL from out side so we need to make sure its only listening to the internal port.. Not sure why this isn’t the default setting.

sudo vim /etc/my.cnf — Add “bind-address=127.0.0.1″ to the [mysqld] block
mysqladmin -u root password ‘new-password’

We want to keep people from using our apache as a proxy, so;
sudo vim /etc/httpd/conf/httpd.conf
Comment out LoadModule proxy_{anything} and restart apache
Change the AllowOverride None to AllowOverride ALL in the root directory (this lets .htaccess changes work)

Now create the various databases and restore those bad boys using the database user/passwords appropriate for those.

Restore the files to their correct places and make sure the virtual host information is correct in httpd.conf

Also, change the logrotate.conf to rotate the httpd logs daily instead of weekly.. otherwise they get biggish..

aaaannd.. GO!