Drupal/server optimization may matter little if you've got the leeches

During the course of administering a server full of various sites which have been Farked, Dugg, and StumbledUpon'd, I've learned first hand the value of optimizing a Drupal site/server to handle large amounts of traffic. I've also learned that eventually it's likely that the level of optimization for one's Drupal site/server will be rendered mostly irrelevant by frequent, and (mostly) malicious, circumstances.

Malicious and/or misdirected requests
Compared to more popular subjects such as Drupal optimization, Apache, MySQL, and/or PHP optimization - the subject of malicious requests gets a rare mention. Despite losing the popularity contest to those sexier subjects, rest assured, there is a lot more to running a site than just tuning the former items. All of those things can be working really, really well and your site/server can still be hammered to a state of dysfunction - even with very few users coming through the site.

Welcome to the wonderful world of denial of service attacks and/or server spam. Broadly defined this includes anything that is requesting something from your server that is of a malicious (usually the case) and/or misguided origin. Make no mistake it's a problem which anyone who is running a medium to large size web site will contend with, whether they know it or not.

Server spam/DDS attacks are NOT uncommon problems limited only to large or 'unlucky' websites. If you have a server and/or VPS with a frequently trafficked site, or especially one with several frequently trafficked sites you will be amazed at just how much processor, memory, and bandwidth can be allocated to malicious and/or wrongly directed requests at any given point in time. The exact of amount of resources that these leeches suck up varies greatly depending on many factors, but on the high-end I can share that several times over the last 6 months alone I've personally witnessed, and fixed, crippling issues stemming from server spam for sites that I system admin for. Likewise several months ago I was happy to be able to help resurrect Drupal.org one day when it was suffering from a particularly nasty malbot. (How many people has this happened to that never realize what was going on, and who in turn just blamed their host and/or Drupal??)

So once one realizes that server spam is a real, and not theoretical, problem how does one confirm the issue(s) on their own site/server and what can they do about it?

For starters I recommend taking a look at this article from our archives. There are some advanced issue/techniques not covered in the article, which I plan to cover in a future article within the next couple weeks, but in the meantime it's a great overview of the battleground and includes some practical tools/techniques you can use right now to successfully help navigate liabilities which will otherwise drain your patience, site, server, and wallet.

23 September, 2007

Comments

Some very good points...and right on the money. I especially liked the section in your archived article, "How to block if their IP is not constant/spoofed?" I've often wondered how best to block a spammer that's spoofing the IP.

Very rarely do I find the IP not being spoofed. Lately, I've seen though where the "human" will later come by to a site and do a search for key words or phrases that are placed by the spambots. They're looking to see if they're spambots have broken through a site and as far as I can tell their IP is usually consistent and not being spoofed...so it's usually an IP that can be reliably blocked.

I've also noticed lately a decrease in spambots visiting my sites. Has anyone else noticed this downward trend...or am I just being overly optimistic?

...is that the weasels seems to fluctuate in number and intensity.

The most recent issue I've seen is bots/people hitting invalid urls (I don't want to give an example in case inspired weasels are reading). These are particularly expensive if they are not blocked because the front page gets bootstapped, rather than just the 404 page. (an issue that seems somewhat fixed for Drupal 6)

This sounds like something that might be hurting my site, WallStreetOasis.com

I recently noticed a slow down of my site and am in the process of trying to diagnose it further. Through some optimization from a known Drupal expert, the load times improved to make it usable, but certain pages are still very slow compared to a few weeks ago.

It is very frustrating and has already cost me thousands.
:(

I will keep at it, but I can see how these issues could cripple a site completely. I will read up on the article. thank you.