Even More Server Drama
Several months ago, we encountered some problems with our hosting company and decided to switch servers. Then, after spending countless hours transferring our army of domains, the new server crashed and our databases were deleted. Further, after the transfer we quickly realized the inferior technological quality of our new host. Thus disgusted, we transferred everything back to our old server and hoped for the best.
Chaos, crashed websites
For the past several months since then, our original server has been running as well as could be expected, all things considered. All functions were running smooth, the error logs were empty, and all was well and good in cyberspace. Then, suddenly, about two weeks ago, someone gained access and indiscriminately hacked every index file on the server. Expectedly, this created chaos, crashed websites, and left our hard-working server techs scrambling to fix the hole and secure the server.
After the attack
Immediately after the attack, we began repairing our websites, uploading files, restoring databases, and troubleshooting errors. Then, just as we began to make some progress, the entire server crashed, wiping out all traces of every domain on the server. Deluged with “help tickets” from many customers, our hosting company responded with a form letter indicating the problem and reaffirming us of their support (copied verbatim):
Dear Velued Customer,
today we have experienced 4 hours of downtime due to broken cPanel/RHEL update. Everyday we receive updates from cPanel and RedHat, they are automatically installed on the server. Those are critical patches, software updates etc. Today’s nightly upgreade broken whole server due to incompatibility in Bind (Name servers) library. All techs have been working on the issue, it took us some time to locate the problem. In the meantime we find out that more hosting companies has such problems. Finally we were able to fix the issue and the servers are back to normal. If you experience any problems accessing your domain names it may be because you tried to access the server when DNS was down and your local ISPs DNS server couldn’t cache the IP address. It may take few hours until your local ISP’s DNS server refresh the DNS zone.
You can check that your web site is up and working properly through 3rd party proxy server ie. www.the-cloak.com
The issue affected ALL hosting companies which uses cPanel, for more information regarding the issue please check cPanel forums at:
http://forums[…].com (edited)
We understand your frustration and how it harmed your business however we would like to assure you that we are here 24 hours a day and 7 days a week and if there is anything wrong we will do our best to fix the issue as soon as possible.
Please accept our appology and we hope to offer you best hosting services possible.
Best regards,
Customer Service Manager
Apparently, during the process of cleaning up the aftermath of the server attack, it became necessary to upgrade various components of cPanel and other server software. Unfortunately, the upgrade produced conflicts and subsequently crashed the entire system. Ahhh yeah. Thanks for that form letter.
Several days later, after great stress and concern, the domains were once again online and accessible, enabling customers access to (once again) begin work on the restoration process. Things were finally looking up..
Cracking exploits
Well almost. After all of our websites had been restored and the dust had settled, several key applications were no longer functional. After an unsuccessful troubleshooting session, we broke down and submitted a help ticket. As it turns out, two vital PHP functions, passthru() and exec(), had been disabled due to security concerns. In other words, thanks to the cracking exploits some mindless showoff, the generous scripting privileges customers once enjoyed have now been restricted.
The good news is that, aside from the loss of a few key functions, everything else is once again up and running considerably well. Looking back, we see how the difficult, stressful, even frustrating events serve as priceless learning experiences. Indeed, managing websites is definitely a challenging endeavour, requiring great patience, flexibility, and determination.
