So I have (had, now) a webserver hosting this site at home. Out of the blue, weird things started going on with my server, and I think it's been hijacked to serve another purpose. What purpose? I don't know. By whom? Still no idea. How? Even less of an idea. In the end, I wound up backing up my website (which was strangely left untouched), and rebuilt my server with a new OS and better security, and got my website up and running.
When I rebuilt the server, I used a new hard drive so that I could rebuild my server and get my website back up. The old hard drive is sitting on my desk, waiting to be plugged in for further analysis.
When things went down, I am honestly not fully sure. But I noticed that slowly but surely, some things would stop working, yet others remained functional without issue.
It all started a few weeks ago when I was trying to set up a VPN server on the webserver. Things were fine, all connectivity to the server seemed fine, but the VPN processes wouldn't seem to start. One of the steps I was following said to run 'ifconfig', which I did. Only I got an error when I tried to do so:
ifconfig: command not found
WTF? I know that just the day before that command worked just fine. In the end, I was unable to get that VPN going. Since this website is my test-bed, as well as personal advertisement, I feared further damaging the server, so I left it all alone while I worked to find something to replace that OS with. Eventuially I found something worthwhile (and theoretically a little more secure), and that is what is running now.
While I was looking for this new OS, I continued to try to troubleshoot my server, and that is when I noticed something else very odd...
So first it was my SSH access. I couldn't get access at all. The server would immediately close the connection, and I couldn't figure out why. At first I thought it was maybe because the phone apps I was using were having problem, but no. I tried the laaaptop (which itself runs Linux as well), and even that wasn't working, and I was getting the exac same 'Server closed the connection' message. So then I tried for physical access.
Now my "server" is really an old desktop computer running a LAMP stack (Linux, Apache, MySQL, and PHP); so it isn't like some dedicated server hardware or anything fancy like that... But since it was a desktop, I decided to use a Linux distro that has a some kind of nice User Interface. The windows manager wound up being OpenBox... But I digress here. Anyways, where I am going with this is that to continue my troubleshooting, I just tried logging in locally, using a direct keyboard/mouse/monitor. While the password was accepted, the windows manager didn't load at all. None of the keyboard exit shortcuts worked. I couldn't get a terminal session. I couldn't do anything, I was dead in the water.
I rebooted the server, booted into the "safe-mode" (or diagnostics, or fail-safe... I don't recall the EXACT words), and that's when things seemed REALLY messed up... I figured I'd start doing a tcpdump on the server to see what was happening with the SSH connections.
"tcpdump: command not found"
What...? I KNOW I have used this command many MANY times with much success. Well, let's take a look at the interfaces
"ifconfig: command not found"
WTF!? Now I know this happened when everything was booted normally, but this was some kind of fail-safe mode, how could this be? Well, let's maybe try to connect out from the server. I tried to SSH to my Raspberry Pi.
"ssh: command not found"
Ok, now things are really starting to seem messed up. What about ping?
"ping: command not found"
Now I'm at a total loss. I decided to reboot into "jnormal" mode, and see what can be done about my website. When it loaded back up, I loaded the URL and Voila, website's fine.
Sure, I may just be paranoid or playing (or imagining) things up on what may otherwise be a plain old system failure. That thought is always on my mind, that this just may be some kind of system failure. And while I am happy to accept that possibility, I just can't help but think that it was pseudo-maliciously hacked. Let's think of this scenario: it isn't simple to determine if a particular server is running headless or not. But blocking such access could be easily scripted. And my web services were all left alone. So if simply left as-is, had I not tried to set up a VPN, whatever had been done would have gone unnoticed. For week, months even. This, I think, is the main reason I think it was hijacked, as opposed to system failure.
So now that I have the old hard drive removed, and my site mostly back up and running, it is time to attempt some kind of forensics analysis to determine what happened. I will journal my work and findings here. Please offer suggestions if you have any.
