I love a good story. Almost as much as I enjoy securing websites. Put them together and you’ve got suspense, intrigue, and plenty of encoded gibberish. But no happy ending this time, in this case the smartest decision was to “pull it” and rebuild. The site was just wasted — completely riddled with malicious code. Without current backup data, it would’ve been “game over” for the site, and possibly the business.
There are several lessons in this sordid tale of hack-recovery, including a reminder of perhaps the most important of all: backup thy data. Without a good backup policy, you’re just begging for data loss.
Act I: The Warning
It’s always first thing in the morning. Out of bed, sit in chair, check email. Uh-oh, what’s this.. someone needs help with a hacked website. Thankfully it’s not anything of mine (just sayin’). So I jump up out of Mail and scramble to the Chrome-mobile. Immediately I’m stopped by the worst:
Ugh.. that’s no good, but don’t panic, let’s take a closer look and gather any clues:
Diagnostic page: “hacked.example.com contains malware”:
- Site is listed as suspicious – visiting this web site may harm your computer.
- Part of this site was listed for suspicious activity 28 time(s) over the past 90 days.
Of the 39 pages we tested on the site over the past 90 days, 16 page(s) resulted in malicious software being downloaded and installed without user consent. The last time Google visited this site was on 2012-06-02, and the last time suspicious content was found on this site was on 2012-05-21.
- Malicious software is hosted on 2 domain(s), including ayy.bee.cee/, dee.eee.eff/.
- This site was hosted on 1 network(s) including 1234567 (CODENAME).
If you are the owner of this web site, you can request a review of your site using Google Webmaster Tools. More information about the review process is available in Google’s Webmaster Help Center.
This is your basic intercept page from Google, designed to alert visitors to potential danger. It does in fact offer some clues about the situation, including:
- Number of reports of malicious activity
- Number of pages tested, number of malicious events
- Date of most recent visit, date of recent malicious activity
- Location of any externally hosted malicious software
- Where to find more information and help
Fortunately it’s not always as bad as it looks. Unfortunately, this time it was..
Act II: The Diagnosis
Upon connecting to the hacked server via SFTP, it’s time to search for hacked files, beginning with one of the most widely used obfuscation methods:
So I download the current theme directory, make a backup, and begin scanning.. almost immediately hundreds of results fill the screen — and the count:
504 items found in 848 documents.
Ewwww… that’s a lot, by any measure. Looking at the code was like watching one of those bloody scenes from an old gangster movie.. in the theme’s
archive.php file, for example, there were 24 instances of malicious code — literally packed into a file that’s normally a mere 50 lines tall. And with the red syntax highlighting for all the encoded gibberish, it just looked like a bloody mess. Essentially DOA. Here is a screenshot to help visualize the situation:
The site was riddled with hundreds of instances of this obfuscated PHP code.
Act III: The Payload
When successfully introduced into its target, malicious code is often referred to as “payload”, referring to both manner of delivery and potential of production. Once payload is discovered, understanding its purpose reveals useful information regarding:
- the severity and scope of the attack
- additional files and/or code involved
- the method or point of entry of the attack
- clues to the identity/location of the attacker
Whatever the payload, it behooves the investigator to examine it thoroughly. In this case, most of the code had been obfuscated using a common method,
eval(base64_decode(. There are numerous ways to decode encrypted PHP code, but the easiest is to copy/paste the encoded string into your favorite online PHP decoder. It’s like, “click, click, and done.”
I can’t display the actual code here on this web page because search engines and anti-virus software will flag my site as “suspiciously malicious” or something equally scary. But it is safe to share the code via external text files; so in the name of “peace and safety,” here is the principle payload, in both encoded and decoded flavours:
Throughout the site, hundreds of instances of this malicious code were discovered. And not entirely in WordPress core and theme files — many of the client’s custom/3rd-party scripts were also infected. Indeed the situation was grim, so time to clean up the mess, lock things down, and back in time for lunch..
Act IV: The Verdict
After assessing the severity of the damage, the verdict was to nuke the site and restore from backups. This is a prime example of why backing up your data is so important. If the client had not a recent set of backups, it would’ve been game over. It’s easy enough to delete everything and reinstall WordPress (or whatever other software), but with many custom/3rd-party scripts and files, rebuilding from scratch takes WAY more time than re-uploading from a recent, complete set of backup files.
Another option might have been to try “cleaning” the infected files with a plugin, script, or even manually. There are several good reasons why cleaning is ineffective:
- No guarantees. Unless you’re using a script or service that scans files and compares against originals, you can’t be sure that all traces have been removed completely.
- Malfunction. Payload could disrupt/alter functionality of plugins or other auto-scanning services, and make things worse.
- Waste of time. It’s just waay faster and more convenient to restore from backup rather than clean.
Regardless of whether you clean or restore, the most important thing is to patch any security holes. How did the attacker get in? Were there multiple points of entry? You’ve got to scan the logs, follow the tracks, and ultimately lock things down. Here is my basic process:
- Put client site in maintenance mode if necessary
- Duplicate hacked site locally and investigate offline
- Restore clean site from most recent working backup
- Patch any/all security vulnerabilities
- Test thoroughly
I go further into depth on this process of cleaning up a hacked website, so give it a read for more information and tips.
Act V: Epilogue
So case closed. After the diagnosis, there were plenty of details to work out. How to improve security, for example. Should the client find a new host, like maybe something that is not on a shared server. A private or virtual private server would lower the chances of another attack. And becuase you’re already starting over with a restore, what better time to find better hosting. And you know that there’s better hosting out there. And more than anything else you do online, finding reliable & secure hosting is worth your time and money. A more secure server gives you a better foundation for securing your online assets.
Lessons learned from this sordid tale of virtual intrigue:
- Always keep dependable recent backups — files, database, email, etc.
- Don’t try “cleaning” a hacked site — restore from recent backup instead
- There’s always better hosting — and it’s your mission to find it
While I’m on the soapbox: take the time to test that your backups actually work. It’s pretty easy to know that files will work, but database imports fail frequently enough to be concerned, so you want to check that your exports are solid, otherwise what’s the point? Might as well not even bother with backups at all. Perhaps a career in the service industry..?
Act VI: Encore (Log Analysis & Security Tips)
Alas, the missing chapter — thanks to Vladimir for furthering the story with some excellent insight and advice into the better securing of your site, particularly those of the WordPress variety :)