Caleb Gilbert's blog

Announcing the Advanced Theme Construction Kit (atck)

I am pleased to announce the official release of the Advanced Theme Construction Kit (atck) (project page):

atck was developed in a production environment for purposes of quickly building css and xhtml valid Drupal themes from scratch without having to un-theme an existing Drupal theme to do it.

atck components include:

1. WYSIWYG grid builder
A browser-based grid builder which produces starter code for a page.tpl.php file. The grid builder itself was originially built by Christos Constandinou and it uses a modified version of Yahoo Grids which is more flexible and easier to customize. With Christos' permission it, and the supporting css, has been customized so that it works for Drupal themes and so that it is css3 valid. Access the builder online (FF only!), or download it and run it locally. [builder cannot be on Drupal.org because of MIT license]

2. style.css [separate download because of BSD license]
style.css contains only the css needed to support what the grid-builder outputs, as well as some general 'resets'. The only modifications one should need to make here would be if they want to make their layout a fixed width and/or a different width (default widths are in %).

3. page-layout.css and template.php
These files are where the visual styling of the site happens. The source of these files is from a combination of code from the Hunchbaque theme and some changes/additions which I added in order to provide baseline settings I prefer and/or provide more granular settings. The beauty of these files is the simplicty of them - they include as little markup/styling as possible to avoid complexity, while at the same time putting many helpful tools/comments at your finger tips so that you can accomplish what you want.

4. page.tpl.php
A sample page.tpl.file is included as a reference for finishing your own page.tpl.php file using code output from the grid builder. Note particularly the variable names for the blocks, regions, menus, etc. (at this point the builder does not include those items)

5. fix-ie-6.css and fix-ie-7.css
These files are included for purposes of providing IE-only css to each of the respective versions. By using conditional comments like this we keep the main css files hack-free and atck css3 valid.

Please keep in mind that this is an initial release - comments, contributions, etc welcome! (please file such things at the project issues queue though)

21 October, 2007

Advanced server/spam bot blocking

Published in: 

As promised in an earlier article about blocking server spam, here are some advanced tips on shutting the door to these resource leeches:

#1: Non-existent urls getting hammered:
This is can be a major problem, one which I believe has been at least somewhat cured in Drupal 6, but for Drupal 5 and below a request to a non-existent page such as http://yoururl.com/node/vote/ does not trigger a 404 page as you might expect. Instead the entire front page loads up. Annoying enough as it is, but when combined with a confused/malicious bot that continually hammers the non-existent url, the resource load can be enough to weigh heavily even on dedicated server, let alone a shared-hosting account. [note: there is an update in the comments below with more specific information about the versions of Drupal which are affected by this problem]

What to do about it:
Certainly putting any paths you see that are getting hit this way in your robots.txt file is a good idea, but that does not always solve the problem. Sometimes more drastic measures are needed. Below is snippet from an actual .htaccess file that has on several cured malbot instances that were causing significant server slowdowns - feel free to use and append yours appropriately (be sure you do not have a line break before the [OR] if you copy this - and also be sure your last line does not have an [OR]):

### Forbid access to bot-beaten non-pages
RewriteCond %{REQUEST_URI} ^/node/forward($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/blog/comment($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/blog/node/forward($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/blog/blog($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/storylink/forward($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/node/blog($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/_vti_bin/404.html($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/categories/node($|/.*$) [OR]
RewriteCond %{REQUEST_URI} ^/node/accessories($|/.*$)
RewriteRule .* - [F,L]

#2: Have an IP? Awesome. Now keep bots from even reaching apache
If there is anything good that can be said about server malbots, as compared to their comment spamming cousins, it's that typically a server-spamming bot will have a static ip address instead of a (dreaded) dynamic one. This makes banning it much easier, of course.

16 October, 2007

Drupal/server optimization may matter little if you've got the leeches

During the course of administering a server full of various sites which have been Farked, Dugg, and StumbledUpon'd, I've learned first hand the value of optimizing a Drupal site/server to handle large amounts of traffic. I've also learned that eventually it's likely that the level of optimization for one's Drupal site/server will be rendered mostly irrelevant by frequent, and (mostly) malicious, circumstances.

Malicious and/or misdirected requests
Compared to more popular subjects such as Drupal optimization, Apache, MySQL, and/or PHP optimization - the subject of malicious requests gets a rare mention. Despite losing the popularity contest to those sexier subjects, rest assured, there is a lot more to running a site than just tuning the former items. All of those things can be working really, really well and your site/server can still be hammered to a state of dysfunction - even with very few users coming through the site.

Welcome to the wonderful world of denial of service attacks and/or server spam. Broadly defined this includes anything that is requesting something from your server that is of a malicious (usually the case) and/or misguided origin. Make no mistake it's a problem which anyone who is running a medium to large size web site will contend with, whether they know it or not.

Server spam/DDS attacks are NOT uncommon problems limited only to large or 'unlucky' websites. If you have a server and/or VPS with a frequently trafficked site, or especially one with several frequently trafficked sites you will be amazed at just how much processor, memory, and bandwidth can be allocated to malicious and/or wrongly directed requests at any given point in time. The exact of amount of resources that these leeches suck up varies greatly depending on many factors, but on the high-end I can share that several times over the last 6 months alone I've personally witnessed, and fixed, crippling issues stemming from server spam for sites that I system admin for. Likewise several months ago I was happy to be able to help resurrect Drupal.org one day when it was suffering from a particularly nasty malbot. (How many people has this happened to that never realize what was going on, and who in turn just blamed their host and/or Drupal??)

So once one realizes that server spam is a real, and not theoretical, problem how does one confirm the issue(s) on their own site/server and what can they do about it?

23 September, 2007

Komodo IDE and Drupal/PHP development - a combo built upon mutual appreciation

After spending 3 days trying to get Elipse PDT and the Zend debugger working on Mac OS X, my nerves were very frayed, indeed. Apparently, there has been an ongoing problem with the Zend debugger not stopping at breakpoints on Mac Intel machines...something that has plagued Eclipse through 3 different PHP extensions. (don't even get me started on how crazy it is that Eclipse has seen three completely separate PHP plugins within less than a year)

In the end all was not lost. On the contrary, after enough scratching around Google I discovered what I needed to know. Komodo has become the semi-de-facto IDE of choice for many Drupal developers. A fact confirmed for me when I saw many familiar names from the Drupal community on the ActiveState website (itself a Drupal site).

Suffice to say with Komodo I got local and remote debugging up and running within just a couple hours, and it's been a total dream to use.

So now I have a proper debugger, integrated svn, and last but not least, Drupal api code compeletion/documentation is included. And if that's not enough, it uses 100mb+ less memory than Eclipse did.

Komodo is going to set me back a few bucks ($295) after my trial runs out, but even at that price it's a no-brainer for some who makes their living with Drupal and PHP. Kudos to ActiveState on their outreach to the Drupal community.

Some useful links if your interested in checking out Komodo:
Link 1
Link 2
Link 3
Link 4

13 September, 2007

AutoPilot released - Change management just got a whole lot easier

Perhaps, one of the most important contributed Drupal projects ever released, was released today (at least as far as professional Drupal development goes).

AutoPilot is out and for anyone who has a Drupal site for which they need to worry about syncing development, staging, and or production versions - it's magnificent. I've had privilege of seeing it work in person - and it's unlike anything available to Drupal developers before now.

Though I haven't set up my own local instance of this yet - the version I saw allowed one to be able to simply click a button in order to update changes between a development environment and a staging/production environment.

Congratulations, WorkHabit on such a valuable contribution, and for everyone else - grab this at your earliest convenience and let's work to integrate this as a core part of a Drupal development toolbox, asap. (ala the devel.module)

[Please direct any questions about AutoPilot to WorkHabit and/or the AutoPilot project on Drupal.org.]

21 August, 2007

Drupal 6: Benchmarking and Block Cache Performance Revisited

As a follow up to an earlier article I posted about Drupal 6 performance, and please bear with my learning curve for a moment, I figured out by 'accident', and a lot of investigation, that it matters very much the order one uses when they 'generate content' with the devel module for benchmarking purposes. My previous tests were done incorrectly - I inadvertently created a bunch of nodes that weren't assigned to any terms or users and vice versa. The result of correcting this error means that a no-cache-enabled-baseline takes much longer to complete than when I had things setup incorrectly.

...happily, the point of this article isn't that I'm a total goof.

No, the good news out of this ordeal is that now when block-cache-disabled performance is compared to block-cache-enabled performance the results are MUCH more substantial than previously noted (and thus Drupal 6 is going to be that much faster than it's predecessor Drupal 5 for authenticated users):

2489.69 ms (request time for auth user, no-caching of any kind)
-878.09 ms (request time for auth user, block-caching on)
--------
1,611.6 (difference) / 2489.69 =
64.73% improvement w/ block cache on

With the block caching on, the mean processing time is 876 ms with a sd of 91.9 ms while the base install results in 2481 ms mean processing time and sd of 91.9. Even at the upper end of the standard deviation, the block-cached processing time is 967.9 ms, which is far below the low end of the standard deviation (2080.1 ms) for the non-block-cached test. Looks like a clear improvement - 64.7 percent improvment using just the means.

The benchmarks are posted here so that everyone can do their own math. If you'd like to check the validity of my installation/numbers - feel free to download a tarball which includes all the files and a db dump. Username/pass for user 1 = superadmin

BENCHMARKS

Benchmarks using 10,000 nodes, 5000 comments, 15 categories, 250 terms, 2000 users and with the following blocks enabled:

BLOCKS ENABLED
Recent comments 
User login
Navigation
Active forum topics
New forum topics
Who's online

=====

13 August, 2007

Pages

Subscribe to RSS - Caleb Gilbert's blog