Early this year, I discovered my server was acting... kinda wonky (details here). In short, I feared that my server had been hijacked for some other, nefarious purposes. After that, I decided to start a forensics analysis, detailed here. 8 months later, after much additional research, I am looking back at my notes and writing up my "final word" on the matter.
Despite the lack of updates on my site here on this topic, I have still been looking over the files, and trying to determine what I can about what I see. The lack of updates is evidence to the fact that I was unable to glean any further details from any of the files I had already looked over. Which is nice on the one hand, seeming to indicate that I didn't really miss anything the first time around, but on the other hand, it is frustrating as I have insufficient evidence to indicate a true root cause.
So my intial thoughts, in the end, still stand. It looks like it really is 1 of 2 possibilities:
- Server was hijacked
- Failed update
I think that the 2 options are mutually exclusive, so it really is one or the other.
Personally, I still think that the server was hijacked... By whom and for what purpose, I will likely never really know. And while it is certainly possible that a failed or interrupted update may have caused some issues, the range of issues, what was affected and in what manner, seems to me to be a little too coincidental for a mere update. But then who knows...
I did learn a few things though... Like the fact that my "backup" process was far from ideal. I also learned that there are not just tools for forensic analysis, like SleuthKit, but there are also live Linux Distros, specially designed for forensics analysis, like DeftLinux. Even Kali Linux has a forensics mode for this very thing. Suffice to say, next time I run into this kind of issue, I will be taking a slightly different approach.
In the end, this was a great learning experience. While I hope to never have to do it with my equipment again, I kinda hope that the chance to do a forensics analysis comes my way again. Should you have want of any such analysis, I will be happy to discuss the matter with you.
