Tale of a Hacked Website

Updated June 26, 2018 • 22 comments

[ Prise de la Bastille, by Jean-Pierre-Louis-Laurent Houel (Detail) ] I love a good story. Almost as much as I enjoy securing websites. Put them together and you’ve got suspense, intrigue, and plenty of encoded gibberish. But no happy ending this time, in this case the smartest decision was to “pull it” and rebuild. The site was just wasted — completely riddled with malicious code. Without current backup data, it would’ve been “game over” for the site, and possibly the business.

There are several lessons in this sordid tale of hack-recovery, including a reminder of perhaps the most important of all: backup thy data. Without a good backup policy, you’re just begging for data loss.

Act I: The Warning

It’s always first thing in the morning. Out of bed, sit in chair, check email. Uh-oh, what’s this.. someone needs help with a hacked website. Thankfully it’s not anything of mine (just sayin’). So I jump up out of Mail and scramble to the Chrome-mobile. Immediately I’m stopped by the worst:

[ Google warning: visiting this site may harm your computer ]

Ugh.. that’s no good, but don’t panic, let’s take a closer look and gather any clues:

Diagnostic page: “hacked.example.com contains malware”:

Site is listed as suspicious – visiting this web site may harm your computer.
Part of this site was listed for suspicious activity 28 time(s) over the past 90 days.

Of the 39 pages we tested on the site over the past 90 days, 16 page(s) resulted in malicious software being downloaded and installed without user consent. The last time Google visited this site was on 2012-06-02, and the last time suspicious content was found on this site was on 2012-05-21.

Malicious software is hosted on 2 domain(s), including ayy.bee.cee/, dee.eee.eff/.
This site was hosted on 1 network(s) including 1234567 (CODENAME).

If you are the owner of this web site, you can request a review of your site using Google Webmaster Tools. More information about the review process is available in Google’s Webmaster Help Center.

This is your basic intercept page from Google, designed to alert visitors to potential danger. It does in fact offer some clues about the situation, including:

Number of reports of malicious activity
Number of pages tested, number of malicious events
Date of most recent visit, date of recent malicious activity
Location of any externally hosted malicious software
Where to find more information and help

Fortunately it’s not always as bad as it looks. Unfortunately, this time it was..

Act II: The Diagnosis

[ Paratroopers of the 50’s @ http://home.hiwaay.net/~magro/paraww2.html ] Upon connecting to the hacked server via SFTP, it’s time to search for hacked files, beginning with one of the most widely used obfuscation methods: eval(base64_decode.

So I download the current theme directory, make a backup, and begin scanning.. almost immediately hundreds of results fill the screen — and the count:

504 items found in 848 documents.

Ewwww… that’s a lot, by any measure. Looking at the code was like watching one of those bloody scenes from an old gangster movie.. in the theme’s archive.php file, for example, there were 24 instances of malicious code — literally packed into a file that’s normally a mere 50 lines tall. And with the red syntax highlighting for all the encoded gibberish, it just looked like a bloody mess. Essentially DOA. Here is a screenshot to help visualize the situation:

[ Screenshot of encoded gibberish (obfuscated payload) ]

The site was riddled with hundreds of instances of this obfuscated PHP code.

Act III: The Payload

When successfully introduced into its target, malicious code is often referred to as “payload”, referring to both manner of delivery and potential of production. Once payload is discovered, understanding its purpose reveals useful information regarding:

the severity and scope of the attack
additional files and/or code involved
the method or point of entry of the attack
clues to the identity/location of the attacker

[ SPACE ART: Space Shuttle Payload Bay Concept @ http://galaxywire.net/2009/04/16/space-art-space-shuttle-payload-bay-concept-1970s/ ] Whatever the payload, it behooves the investigator to examine it thoroughly. In this case, most of the code had been obfuscated using a common method, eval(base64_decode(. There are numerous ways to decode encrypted PHP code, but the easiest is to copy/paste the encoded string into your favorite online PHP decoder. It’s like, “click, click, and done.”

I can’t display the actual code here on this web page because search engines and anti-virus software will flag my site as “suspiciously malicious” or something equally scary. But it is safe to share the code via external text files; so in the name of “peace and safety,” here is the principle payload, in both encoded and decoded flavours:

Before: Encoded payload
After: Decoded payload

Throughout the site, hundreds of instances of this malicious code were discovered. And not entirely in WordPress core and theme files — many of the client’s custom/3rd-party scripts were also infected. Indeed the situation was grim, so time to clean up the mess, lock things down, and back in time for lunch..

Act IV: The Verdict

[ Scene from ‘The Verdict’ ] After assessing the severity of the damage, the verdict was to nuke the site and restore from backups. This is a prime example of why backing up your data is so important. If the client had not a recent set of backups, it would’ve been game over. It’s easy enough to delete everything and reinstall WordPress (or whatever other software), but with many custom/3rd-party scripts and files, rebuilding from scratch takes WAY more time than re-uploading from a recent, complete set of backup files.

Another option might have been to try “cleaning” the infected files with a plugin, script, or even manually. There are several good reasons why cleaning is ineffective:

No guarantees. Unless you’re using a script or service that scans files and compares against originals, you can’t be sure that all traces have been removed completely.
Malfunction. Payload could disrupt/alter functionality of plugins or other auto-scanning services, and make things worse.
Waste of time. It’s just waay faster and more convenient to restore from backup rather than clean.

Regardless of whether you clean or restore, the most important thing is to patch any security holes. How did the attacker get in? Were there multiple points of entry? You’ve got to scan the logs, follow the tracks, and ultimately lock things down. Here is my basic process:

Put client site in maintenance mode if necessary
Duplicate hacked site locally and investigate offline
Restore clean site from most recent working backup
Patch any/all security vulnerabilities
Test thoroughly

I go further into depth on this process of cleaning up a hacked website, so give it a read for more information and tips.

Act V: Epilogue

So case closed. After the diagnosis, there were plenty of details to work out. How to improve security, for example. Should the client find a new host, like maybe something that is not on a shared server. A private or virtual private server would lower the chances of another attack. And becuase you’re already starting over with a restore, what better time to find better hosting. And you know that there’s better hosting out there. And more than anything else you do online, finding reliable & secure hosting is worth your time and money. A more secure server gives you a better foundation for securing your online assets.

Lessons learned from this sordid tale of virtual intrigue:

Always keep dependable recent backups — files, database, email, etc.
Don’t try “cleaning” a hacked site — restore from recent backup instead
There’s always better hosting — and it’s your mission to find it

While I’m on the soapbox: take the time to test that your backups actually work. It’s pretty easy to know that files will work, but database imports fail frequently enough to be concerned, so you want to check that your exports are solid, otherwise what’s the point? Might as well not even bother with backups at all. Perhaps a career in the service industry..?

Act VI: Encore (Log Analysis & Security Tips)

Alas, the missing chapter — thanks to Vladimir for furthering the story with some excellent insight and advice into the better securing of your site, particularly those of the WordPress variety :)

About the Author

Jeff Starr = Fullstack Developer. Book Author. Teacher. Human Being.

22 responses to “Tale of a Hacked Website”

Peter Michael 2012/06/12 7:55 pm

Did you find out how the payload was delivered?

Jeff Starr 2012/06/12 10:33 pm • Post Author

Unfortunately no, but other sites on the same shared server were also infected with the same code.

Vladimir 2012/06/12 8:37 pm

I think you miss another important part – log analysis.

The problem with all hacked sites are usually that they contain a vulnerability that allowed the attacker to compromise the site. In this case even if you restore the site from the backup, chances are that the site can be hacked again.

When theme files are changed, there are two most probable scenarios:

1. The theme or a plugin has a vulnerability that allowed the attacker to upload an arbitrary file (usually php file) and run it (the most famous case is probably the vulnerability in TimThumb).

If this is the case, you can probably find the references to that file in the access logs of the web server. This will give you the list of IP addresses the attacker used. Then you need to search the log for those IP addresses – this will give you (probably very big) list of requests used by the attacker. Your goal is to find any unusual requests (e.g., containing any patterns specific to XSS or SQL injection attacks etc) or suspicious POST requests (POST is used to upload the file). This will give you the idea which plugin/file was used to compromise the site. Then you either get rid of it or install an update (if any and if you still trust the author of the plugin).

2. The theme was altered using Theme Editor. This is worse because it uaully means that the attacker has administrative access to your site. You will need to change the admin password and check whether there are unwanted users with administrative privileges.

There are two most common ways to get the admin access:
* brute force the password – to fight this you will need a plugin that block the user/IP after several failed login attempts or use CAPTCHA to log into the admin panel etc. If this was the case, you will see a lot of POST requests to /wp-login.php
* use SQL injection attack – the attacker used a vulnerability in a plugin or the theme to create a user with admin privileges. Check the logs for the traces of SQL injection attacks.

Things that can make attacker’s job more difficult:
* your great blacklist – the only addition I would make is logging: instead of making Apache return 403 immediately, I would use mod_rewrite to redirect the request to a script that logs all useful information somewhere (IP, User-Agent, query string, cookies, …) and then returns 403 Forbidden. This information can be used a) to file an abuse report to the attacker’s ISP, b) to report the attacker to, say, blocklist.de or a similar site, c) the information can be used by a plugin that will deny access to the admin panel if there is any recent malicious activity from that IP. Or the script can set a cookie to mark that bot/user as an attacker and use that cookie during, say, plugins_loaded phase to do funny things;
* remove write access from the directories that should not be written to: wp-admin, wp-includes or even themes and plugins – you can always give back write access with your (S)FTP client;
* disable PHP script execution in wp-content/uploads and other similar directories (especially if they are writable – cache, thumbnails etc) – just create .htaccess with

php_value engine off

Jeff Starr 2012/06/12 10:43 pm • Post Author

Incredibly insightful and useful input, Thank you Vladimir! I’ve updated the tale with the missing act: Act VI: Encore (Log Analysis & Security tips) — it basically just points back here to your comment :)
- Dan Farrow 2012/08/06 9:45 am
  
  Just to let you know that, now there’s more than one page of comments, the link to Vladimir’s comment has changed to:
  
  https://perishablepress.com/tale-of-a-hacked-website/comment-page-1/#comment-87574
  
  Cheers! Dan
N Atta Kusi Adusei 2012/06/13 3:21 am

Thanks 4 the add, Vlad. On point number two, I would advise disabling theme and plugin editors in the dashboard. I think it would go a great way to eliminate some of these possibilities. You could always find snippets for achieving these on http://www.wpfunction.me/

Jon 2012/06/12 11:28 pm

Man, I had the exact same thing happen to my shared server. Noticed the warnings in my browser, then the encoded payload.

I didn’t know how to / that you could decode that stuff, thanks for sharing.

Luckily, it hadn’t been to long since I backed up my server, but like you said it could have been game over.

oren 2012/06/13 1:15 am

very nice writing

N Atta Kusi Adusei 2012/06/13 3:22 am

Great, Jeff, as always. Thanks :-)

JP 2012/06/13 5:22 am

You wrote “Always keep dependable recent backups” — do you have any recommended methods or plugins for doing so? Anything you know has worked when a site has been hacked?

Jon 2012/06/13 9:11 pm

I’m a lot newer to this stuff than the author, but i use WordPress Database Backups ( http://austinmatzko.com/wordpress-plugins/wp-db-backup/ ) for my databases and keep my theme files backed up locally on my computer with git.

Andy W 2012/06/13 5:53 am

Nothing is 100% secure/effective; but there are products available that “prevent” contagion on shared servers e.g. 1H Hive. My hosting provider kualo.com (good value, reliable, support above and beyond) has used this since early 2011 and it is being used by an increasing number of hosts.

Right, I’m off to back-up my WordPress sites.

Peter Mumford 2012/06/24 8:45 am

Jeff,
I saw some odd files on some WP-run sites last week. I posted on the WP forum about these files, but no one seemed the least bit interested:
http://wordpress.org/support/topic/malicious-wp-installs?replies=12

It’s possible that these files show a vulnerability that would be useful to know about.

Vladimir 2012/06/25 11:50 am

Peter, those files are now removed and it is difficult to tell somehting about them.

If you tried to access them with Googlebot user agent (or Google SERP referrer) you might see something different.

Judging by reports from scumware (e.g., http://www.scumware.org/report/69.163.140.152) they look like malicious files. However, without server access logs it is difficult to tell how they were uploaded.

Peter Mumford 2012/06/28 5:07 am

Jeff, Vladimir,
I just got another email. It contained the following (probably malicious) url:
http://rus.ctn.com.kg/rehrt.html
Although I don’t know if this is a wordpress installation, its from the same email account that sent the last ones.

Vladimir 2012/06/30 3:35 pm

Peter, rus.ctn.com.kg is simply a redirector, it redirects to online-article24.com. That site, however, is often referenced in spam emails (weight loss) – http://multirbl.valli.org/lookup/online-article24.com.html

It could be that the malicious files uploaded to WordPress are/were used to send spam.

This also can be interesting: http://www.urlvoid.com/scan/online-article24.com/

spencer 2012/07/11 8:29 pm

The My site don’t show pictures upload by user. why?
many thanks

Comments are closed for this post. Something to add? Let me know.

Peter Michael 2012/06/12 7:55 pm

Did you find out how the payload was delivered?
- Jeff Starr 2012/06/12 10:33 pm • Post Author
  
  Unfortunately no, but other sites on the same shared server were also infected with the same code.
Vladimir 2012/06/12 8:37 pm

I think you miss another important part – log analysis.

The problem with all hacked sites are usually that they contain a vulnerability that allowed the attacker to compromise the site. In this case even if you restore the site from the backup, chances are that the site can be hacked again.

When theme files are changed, there are two most probable scenarios:

1. The theme or a plugin has a vulnerability that allowed the attacker to upload an arbitrary file (usually php file) and run it (the most famous case is probably the vulnerability in TimThumb).

If this is the case, you can probably find the references to that file in the access logs of the web server. This will give you the list of IP addresses the attacker used. Then you need to search the log for those IP addresses – this will give you (probably very big) list of requests used by the attacker. Your goal is to find any unusual requests (e.g., containing any patterns specific to XSS or SQL injection attacks etc) or suspicious POST requests (POST is used to upload the file). This will give you the idea which plugin/file was used to compromise the site. Then you either get rid of it or install an update (if any and if you still trust the author of the plugin).

2. The theme was altered using Theme Editor. This is worse because it uaully means that the attacker has administrative access to your site. You will need to change the admin password and check whether there are unwanted users with administrative privileges.

There are two most common ways to get the admin access:
* brute force the password – to fight this you will need a plugin that block the user/IP after several failed login attempts or use CAPTCHA to log into the admin panel etc. If this was the case, you will see a lot of POST requests to /wp-login.php
* use SQL injection attack – the attacker used a vulnerability in a plugin or the theme to create a user with admin privileges. Check the logs for the traces of SQL injection attacks.

Things that can make attacker’s job more difficult:
* your great blacklist – the only addition I would make is logging: instead of making Apache return 403 immediately, I would use mod_rewrite to redirect the request to a script that logs all useful information somewhere (IP, User-Agent, query string, cookies, …) and then returns 403 Forbidden. This information can be used a) to file an abuse report to the attacker’s ISP, b) to report the attacker to, say, blocklist.de or a similar site, c) the information can be used by a plugin that will deny access to the admin panel if there is any recent malicious activity from that IP. Or the script can set a cookie to mark that bot/user as an attacker and use that cookie during, say, plugins_loaded phase to do funny things;
* remove write access from the directories that should not be written to: wp-admin, wp-includes or even themes and plugins – you can always give back write access with your (S)FTP client;
* disable PHP script execution in wp-content/uploads and other similar directories (especially if they are writable – cache, thumbnails etc) – just create .htaccess with

php_value engine off
- Jeff Starr 2012/06/12 10:43 pm • Post Author
  
  Incredibly insightful and useful input, Thank you Vladimir! I’ve updated the tale with the missing act: Act VI: Encore (Log Analysis & Security tips) — it basically just points back here to your comment :)
  - Dan Farrow 2012/08/06 9:45 am
    
    Just to let you know that, now there’s more than one page of comments, the link to Vladimir’s comment has changed to:
    
    https://perishablepress.com/tale-of-a-hacked-website/comment-page-1/#comment-87574
    
    Cheers! Dan
- N Atta Kusi Adusei 2012/06/13 3:21 am
  
  Thanks 4 the add, Vlad. On point number two, I would advise disabling theme and plugin editors in the dashboard. I think it would go a great way to eliminate some of these possibilities. You could always find snippets for achieving these on http://www.wpfunction.me/
Jon 2012/06/12 11:28 pm

Man, I had the exact same thing happen to my shared server. Noticed the warnings in my browser, then the encoded payload.

I didn’t know how to / that you could decode that stuff, thanks for sharing.

Luckily, it hadn’t been to long since I backed up my server, but like you said it could have been game over.
oren 2012/06/13 1:15 am

very nice writing
N Atta Kusi Adusei 2012/06/13 3:22 am

Great, Jeff, as always. Thanks :-)
JP 2012/06/13 5:22 am

You wrote “Always keep dependable recent backups” — do you have any recommended methods or plugins for doing so? Anything you know has worked when a site has been hacked?
- Jon 2012/06/13 9:11 pm
  
  I’m a lot newer to this stuff than the author, but i use WordPress Database Backups ( http://austinmatzko.com/wordpress-plugins/wp-db-backup/ ) for my databases and keep my theme files backed up locally on my computer with git.
Andy W 2012/06/13 5:53 am

Nothing is 100% secure/effective; but there are products available that “prevent” contagion on shared servers e.g. 1H Hive. My hosting provider kualo.com (good value, reliable, support above and beyond) has used this since early 2011 and it is being used by an increasing number of hosts.

Right, I’m off to back-up my WordPress sites.
Peter Mumford 2012/06/24 8:45 am

Jeff,
I saw some odd files on some WP-run sites last week. I posted on the WP forum about these files, but no one seemed the least bit interested:
http://wordpress.org/support/topic/malicious-wp-installs?replies=12

It’s possible that these files show a vulnerability that would be useful to know about.
Vladimir 2012/06/25 11:50 am

Peter, those files are now removed and it is difficult to tell somehting about them.

If you tried to access them with Googlebot user agent (or Google SERP referrer) you might see something different.

Judging by reports from scumware (e.g., http://www.scumware.org/report/69.163.140.152) they look like malicious files. However, without server access logs it is difficult to tell how they were uploaded.
Peter Mumford 2012/06/28 5:07 am

Jeff, Vladimir,
I just got another email. It contained the following (probably malicious) url:
http://rus.ctn.com.kg/rehrt.html
Although I don’t know if this is a wordpress installation, its from the same email account that sent the last ones.
Vladimir 2012/06/30 3:35 pm

Peter, rus.ctn.com.kg is simply a redirector, it redirects to online-article24.com. That site, however, is often referenced in spam emails (weight loss) – http://multirbl.valli.org/lookup/online-article24.com.html

It could be that the malicious files uploaded to WordPress are/were used to send spam.

This also can be interesting: http://www.urlvoid.com/scan/online-article24.com/
spencer 2012/07/11 8:29 pm

The My site don’t show pictures upload by user. why?
many thanks

12 • Newer Comments »