RocketModule - performance http://rocket.local/categories/performance en Example Varnish VCL for a Drupal / Pressflow site http://rocket.local/blog/example-varnish-vcl-drupal-pressflow-site <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><img src="http://highervisibilitywebsites.com/files/varnish-logo-red-64.gif" style="float:right; margin: 3px;" alt="" />A few months ago I set up Varnish on my Macbook Pro and have deployed it for a production site which serves anonymous and (a lot of) authenticated users. Initially, I spent a couple months just running it in my local environment, including backporting the <a href="http://drupal.org/project/Varnish">Varnish.module</a> to Drupal 5. In retrospect, I'm glad that I spent the time to learn how Varnish and it's configuration file works before deploying it, as it's paid off in a big way as our production site now has something which is equivalent to:</p> <ul><li>...an in-memory static file server for all users (e.g., the equivalent of hooking up something like nginx or <a href="/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining">lighttpd</a> as a front end to Apache (or whatever you're using).</li> <li>...an in-memory boost.module in terms of database-relief for anonymous users.</li> </ul><p>Contrary to popular belief the two items above are in no way an automatic benefit of simply installing Varnish. If the configuration file, and Drupal installation, is not massaged with care one definitely won't get the database relief from anonymous page caching, and the benefits from Varnish-as-a-static-file server will not nearly be optimized. Bottom line Varnish can be a temperamental piece of software. It only gives back what you put into it.</p> <p>To this end, the settings in the Varnish VCL file can make or break whether you get a substantial benefit from it. Below is an example VCL file, which was formed from a good amount of research and a lot of trial and error:</p> </div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item even"><a href="/categories/varnish" typeof="skos:Concept" property="rdfs:label skos:prefLabel">varnish</a></div><div class="field-item odd"><a href="/categories/pressflow" typeof="skos:Concept" property="rdfs:label skos:prefLabel">pressflow</a></div></div></div> Tue, 18 May 2010 20:14:39 +0000 Caleb Gilbert 103 at http://rocket.local http://rocket.local/blog/example-varnish-vcl-drupal-pressflow-site#comments Scaling Drupal: HTTP pipelining and benchmarking revisited http://rocket.local/blog/scaling-drupal-http-pipelining-and-benchmarking-revisited <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><strong>UPDATE: I've updated some of the numbers below to reflect corrections for a testing error. Let's just say to be sure not to benchmark with any external links in your test pages (because if you do use external links you'll obviously be benchmarking the external server too, which is not what we want in this case). To summarize the effect of these corrections - having lighttpd in front of Apache and pipelining actually provide a substantially larger boost in performance than I had indicated before. Other than that the results are the same.</strong></p> <p>So things with my <a href="http://highervisibilitywebsites.com/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining">first attempt at benchmarking HTTP pipelining</a> did not go exactly as planned. It turns out that if two different domains/subdomains you are using for content on your site are pointing to the same IP, <a href="http://highervisibilitywebsites.com/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining">based on previous testing</a>, it looks like browsers (at least FireFox) will not pipeline requests (e.g., create more concurrent requests to your site) because it considers the requests as being from the same origin. In order for a browser to pipelining requests at all, they seem to require two domains/subdomains which are using <em>two separate/unique IPs</em>. If you read the <a href="http://en.wikipedia.org/wiki/Hostname">Wikipedia entry for hostnames</a> this all makes sense, as it indicates domains are associated with IP's, and <a href="http://www.browserscope.org/">browserscope's</a> testing of browsers checks for "Connections per Hostname", not "Connections per Domain".</p> <p>After figuring out how to get requests to pipeline correctly, I re-benchmarked all the configurations from the <a href="http://highervisibilitywebsites.com/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining">first article </a>. Everything from that article regarding lighttpd is still holds true, so without covering those aspects again, here's the updated benchmarks and notes for browser request pipelining:</p> <ul><li>Once the conditions for request pipelining was setup correctly there were discernable performance implications. Some of them I definitely wasn't expecting. On the one end of the spectrum, with browser pipelining working (via string replacement of domains within the rendered HTML) and lighttpd serving the static files there was an 11% increase in throughput vs not using the pipelining methods. So static file serving ='s good, and static file serving + HTTP pipelining ='s a little better. <p>This is not where the story ends with pipelining however, as there was a net performance <em>decrease</em> by enabling pipelining with all configurations which did not use a separate static file server! (in my case lighttpd on the same machine)</p></li></ul></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item even"><a href="/categories/optimization" typeof="skos:Concept" property="rdfs:label skos:prefLabel">optimization</a></div><div class="field-item odd"><a href="/categories/lighttpd" typeof="skos:Concept" property="rdfs:label skos:prefLabel">lighttpd</a></div></div></div> Wed, 27 Jan 2010 23:37:28 +0000 Caleb Gilbert 96 at http://rocket.local http://rocket.local/blog/scaling-drupal-http-pipelining-and-benchmarking-revisited#comments Scaling Drupal: Benchmarking static file serving with lighttpd and browser pipelining http://rocket.local/blog/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><img src="/files/light_logo_170px.png" style="display:inline;margin:5px 15px 3px 0pt;float:left;" alt="" />I finally had a chance to investigate an optimization which I've been wondering about for a while now - serving static files of a site from somewhere else. As a side, but related, experiment I also tested the claim that serving files from a static file server/separate domain/subdomain will speed things up because it results in browsers opening more concurrent requests than they would from a single domain.</p> <p>For my tests I used lighttpd (pron. <em>lighty</em>) as a static file server for Apache. The idea is that lighttpd, which is acclaimed as being fast and light on memory, will serve the non-dynamic pages of the site (images, CSS, Javascript, etc), which should thereby help relieve Apache of some of its workload. This arrangement involves changing the paths, either on the backend or frontend, to these static resources so that they no longer get served by Apache.</p> <p><strong>The pieces</strong><br /> All tests took place on my Macbook Pro and involved two pages on a large Drupal 5 site running Pressflow. For the static file server itself, I installed lighttpd using Macports. Two separate pages of the site were tested, the smaller page's number of static files was in the category of 'average' for most sites. The larger page of the two, was very large - 39 CSS files, 23 Javascript files, and 46 image files.</p> <p><strong>Methods tested and benchmarked</strong><br /> I implemented and benchmarked the following methods of path modification in order to enable static file serving:</p></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/apache" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Apache</a></div><div class="field-item even"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item odd"><a href="/categories/lighttpd" typeof="skos:Concept" property="rdfs:label skos:prefLabel">lighttpd</a></div></div></div> Mon, 25 Jan 2010 17:50:09 +0000 Caleb Gilbert 95 at http://rocket.local http://rocket.local/blog/scaling-drupal-benchmarking-static-file-serving-lighttpd-and-browser-pipelining#comments Convert your MySQL database from MyISAM to InnoDB, and get ready for Drupal 7 at the same time http://rocket.local/blog/convert-your-mysql-database-myisam-innodb-and-get-ready-drupal-7-same-time <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><img src="/files/innodb-trans.png" alt="" style="float:left;margin:5px 15px 3px 0;" />If you haven't already heard, <a href="http://drupal.org/node/301362">Drupal 7 will default to using the InnoDB storage engine instead of MyISAM</a> for MySQL (though a MyISAM database will continue to work just fine in Drupal 7). This is fairly substantial change within Drupal core, and as the thread in the issue queue I linked to shows, there were a lot of questions and apprehension about it. However...</p> <p>...we are going to just skip over a lot of that apprehension and get down to point of this article - there's no good reason not to hop right into using InnoDB today on your Drupal 5 or Drupal 6 site. The rewards are; a possibly significant improvement in performance, a definite improvement in scalability (most highly trafficked Drupal sites have been using InnoDB for some time now because of this), and you'll start getting used to working with what will be more and more common in your Drupal-life, InnoDB.</p> <p><strong>My experience</strong><br /> I came to the conclusion about how great InnoDB is after researching the experiences of others, and after converting a large Pressflow-driven Drupal 5 site from InnoDB vs MyISAM. This change resulted in a 14% increased throughput during load tests performed in JMeter. That's a very substantial increase, and while everyone's mileage will vary based on their own site, server, and any number of variables it's clear enough to me that there's nothing to be afraid of as far as InnoDB goes (quite the contrary).</p> <p><strong>Converting your database to InnoDB</strong><br /> Before you go any further <strong>backup your database</strong> before doing any steps below. If you 'splode your database for any reason, you'll need it.</p> <p>Here are the steps:</p> <blockquote><p> 1. <em>Shutdown MySQL</em></p> <p>2. <em>Move/copy/change the name of ib_logfile0 and ib_logfile1 files</em>. (find where MySQL exists on your system - locations can vary greatly). MySQL will recreate these files when you start it up again. Not anytime you change the innodb_log_file_size parameter you will need to shutdown MySQL, move these files, and start up MySQL again.</p> <p>3. <em>Tune it up a bit</em><br /> Based on a lot of searching around and benchmarking with JMeter I arrived at the setting below for running on my Macbook Pro. See the links at the end of this post for articles which can help you determine what to adjust these numbers to for other machines (ones with more RAM/CPU, for instance. The production server for this particular site ended up with 5000M setting for innodb_buffer_pool_size. So settings will, and should, vary greatly just depending).</p></blockquote></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/mysql" typeof="skos:Concept" property="rdfs:label skos:prefLabel">MySQL</a></div><div class="field-item even"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item odd"><a href="/categories/innodb" typeof="skos:Concept" property="rdfs:label skos:prefLabel">InnoDB</a></div></div></div> Mon, 18 Jan 2010 18:28:18 +0000 Caleb Gilbert 94 at http://rocket.local http://rocket.local/blog/convert-your-mysql-database-myisam-innodb-and-get-ready-drupal-7-same-time#comments Speeding up svn / ssh transfer speed in terminal for Mac OS 10.5, Leopard http://rocket.local/blog/speeding-svn-ssh-transfer-speed-terminal-mac-os-105-leopard <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>After struggling with upload/commit transfer speeds that were absolutely crippling, I've finally managed to get an exponential speed boost for svn merging and svn committing large numbers of files.</p> <p>For svn merging, see <a href="http://www.workhabit.org/subversion-merge-easy-way-copying-changes-trunk-prod-tag">this post by the good folks at WorkHabit.org</a>.</p> <p>For general SVN transfer speed help - hop into your /etc/ssh_config file and add these lines to your file:</p> <div class="codeblock"><code># Host *<br />Compression no<br />FallBackToRsh yes<br />KeepAlive yes</code></div> <p>Enjoy not seeing upload times of 2k/sec anymore. :-)</p> </div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item odd"><a href="/categories/ssh" typeof="skos:Concept" property="rdfs:label skos:prefLabel">ssh</a></div><div class="field-item even"><a href="/categories/svn" typeof="skos:Concept" property="rdfs:label skos:prefLabel">svn</a></div></div></div> Tue, 01 Jul 2008 22:23:24 +0000 Caleb Gilbert 76 at http://rocket.local http://rocket.local/blog/speeding-svn-ssh-transfer-speed-terminal-mac-os-105-leopard#comments Speeding up Drupal Forums http://rocket.local/blog/speeding-drupal-forums <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>The Drupal forum.module has become, well, somewhat infamous for less than awesome scalability. Recently I had a chance to see this firsthand, and track down a solution for managing the long page load times for a <a href="http://wallstreetoasis.com">client</a> who has a highly trafficked forum. This was not a case of a site that was un-tuned - actually this particular site had a lot of good work and performance enhancements already done to it, including block caching and even some modifications to the forum module that were allowing to work better than it would have without them. But still 5-6 second page load times on /forum persisted.</p> <p>As this was my first time working on the site, I began by reviewing all of the main configuration files for Apache, MySQL, and PHP, since they are the foundation for everything else. After making some adjustment there, I headed over to the Drupal admin interface and check /admin/setting/performance/ and made sure all was happy there as well. Finally, I went to the block admin page and double-checked all of the blockcache settings, which as it turns out were set a bit to aggressively, resulting in slow form submission times (every time anyone submitted anything a gazillion blocks were being re-cached whether they need to be or not).</p> <p>With the foundation of the site now looking good and everything, except the forum pages flying along (and the tracker, but that's a story for a later article) - it was just down to forum.module.</p> <p>Here are the steps that led to cutting the page load time in half from what they were:</p> <p>1. Disable the forum.module, rename it to something like "forum.module.orig", make a copy and then rename the copy "forum.module".</p> <p>2. Download the <a href="http://drupal.org/project/advcache">advcache module</a> and apply the forum patch includes to your new copy of the forum.module.</p> <p>3. If want the cleanest solution and are comfortable with coding/debugging at all, instead of just copying the forum module and working on it directly (and thus having a hacked Drupal core file around all the time) - name the copied file something completely different than forum.module and edit all of the hook/function calls with in it with new name and place it where you keep the rest of your contributed modules.</p> <p>4. (Note: this step is an option only if all your forums are public)<br /> Open the module and remove all references to db_rewrite_sql. This will keep Drupal from doing a lot of expensive and uneccesary queries in order to check access rights. (<a href="http://2bits.com/articles/how-drupals-nodeaccess-table-can-negatively-impact-site-performance.html">thanks Khalid</a>)</p></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/mysql" typeof="skos:Concept" property="rdfs:label skos:prefLabel">MySQL</a></div><div class="field-item even"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item odd"><a href="/categories/tuning" typeof="skos:Concept" property="rdfs:label skos:prefLabel">tuning</a></div><div class="field-item even"><a href="/categories/forums" typeof="skos:Concept" property="rdfs:label skos:prefLabel">forums</a></div></div></div> Thu, 28 Feb 2008 18:41:41 +0000 Caleb Gilbert 71 at http://rocket.local http://rocket.local/blog/speeding-drupal-forums#comments Drupal 6: Benchmarking and Block Cache Performance Revisited http://rocket.local/blog/drupal-6-benchmarking-and-block-cache-performance-revisited <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>As a follow up to an <a href="http://highervisibilitywebsites.com/drupal-6-performance-and-developer-drupal-release">earlier article</a> I posted about Drupal 6 performance, and please bear with my learning curve for a moment, I figured out by 'accident', and a lot of investigation, that it matters very much the order one uses when they 'generate content' with the devel module for benchmarking purposes. My previous tests were done incorrectly - I inadvertently created a bunch of nodes that weren't assigned to any terms or users and vice versa. The result of correcting this error means that a no-cache-enabled-baseline takes much longer to complete than when I had things setup incorrectly.</p> <p>...happily, the point of this article isn't that I'm a total goof.</p> <p>No, the good news out of this ordeal is that now when block-cache-disabled performance is compared to block-cache-enabled performance the results are MUCH more substantial than previously noted (and thus Drupal 6 is going to be that much faster than it's predecessor Drupal 5 for authenticated users):</p> <blockquote><p>2489.69 ms (request time for auth user, no-caching of any kind)<br /> -878.09 ms (request time for auth user, block-caching on)<br /> --------<br /> 1,611.6 (difference) / 2489.69 =<br /><strong>64.73% improvement w/ block cache on</strong> </p></blockquote> <p>With the block caching on, the mean processing time is 876 ms with a sd of 91.9 ms while the base install results in 2481 ms mean processing time and sd of 91.9. Even at the upper end of the standard deviation, the block-cached processing time is 967.9 ms, which is far below the low end of the standard deviation (2080.1 ms) for the non-block-cached test. Looks like a clear improvement - 64.7 percent improvment using just the means.</p> <p>The benchmarks are posted here so that everyone can do their own math. If you'd like to check the validity of my installation/numbers - feel free to download a tarball which includes all the <a href="http://highervisibilitywebsites.com/downloads/benchmark-files.tgz">files</a> and a <a href="http://highervisibilitywebsites.com/downloads/benchmark-db.sql.tgz">db dump</a>. Username/pass for user 1 = superadmin</p> <p><strong>BENCHMARKS</strong></p> <p>Benchmarks using 10,000 nodes, 5000 comments, 15 categories, 250 terms, 2000 users and with the following blocks enabled:</p> <div class="codeblock"><code>BLOCKS ENABLED<br />Recent comments  <br />User login <br />Navigation <br />Active forum topics <br />New forum topics <br />Who's online</code></div> <p>=====</p></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div><div class="field-item even"><a href="/categories/block-cache" typeof="skos:Concept" property="rdfs:label skos:prefLabel">block cache</a></div><div class="field-item odd"><a href="/categories/tuning" typeof="skos:Concept" property="rdfs:label skos:prefLabel">tuning</a></div></div></div> Mon, 13 Aug 2007 01:49:17 +0000 Caleb Gilbert 59 at http://rocket.local http://rocket.local/blog/drupal-6-benchmarking-and-block-cache-performance-revisited#comments Drupal 6: The Performance and Developer Drupal release? http://rocket.local/blog/drupal-6-performance-and-developer-drupal-release <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>[The information in this post has been <a href="http://highervisibilitywebsites.com/drupal-6-benchmarking-and-block-cache-performance-revisited">greatly updated</a> - basically, Drupal 6 will enjoy even more of a performance advantage over Drupal 5 than is indicated here]</p> <p>The more one learns about Drupal 6 the more there is to like. A while ago, right before Drupal 5 was released, I heard someone reference it being a 'developer's release' (think maybe it was <a href="http://buytaert.net">Dries</a>). That seems like an apt way to describe Drupal 6 these days. If the anti-CMS, ya-gotta-roll-your-own-or-it-stinks critics have a last breath left, this might be what snuffs it out once and for all. The list of new features, optimizations, and/or technologies introduced in Drupal 6 seems destined to light up the antennae of developers everywhere.</p> <p><strong>Performance, performance, performance</strong> </p> <p>If there is any doubt about whether there will be performance improvements for Drupal 6, wonder no longer. Benchmarks, made using <a href="http://drupal.org/node/79237">these standards</a>, shows that the current, pre-beta, version of Drupal 6 performs 19.5% faster than Drupal 5.2 does when NO page caching is on. A healthy improvement by any standard, but even more so when one considers the following:</p> <ul><li>Drupal page caching is never active for logged in users, so any gains to non-cached speeds are pure gains for everyone that is logged in. This is a big deal when you consider that up till now it has taken 10x longer to serve authenticated/logged users than it does to server anonymous site visitors.</li> <p></p> <li>With the <a href="http://drupal.org/node/80951">block cache patch</a> which looks destined for core applied, Drupal 6 becomes 32% faster than Drupal 5.2 for authenticated users. Community site system admins rejoice.</li> </ul><p>Without quibbling over the details of specific numbers and/or benchmarking methods - one thing is very clear.</p> <p><em>Speed</em> is coming to Drupal.</p> <ul><li>Download benchmarks for Drupal 6, which includes benchmarks for block caching <a href="http://highervisibilitywebsites.com/downloads/drupal6_and_blockcache_benchmarks.txt">here</a>.</li> <p></p> <li>Download benchmarks for Drupal 5-2 <a href="http://highervisibilitywebsites.com/downloads/drupal_5-2_benchmarks.txt">here</a>.</li> </ul><p></p></div></div></div><div class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above"><div class="field-label">Categories:&nbsp;</div><div class="field-items"><div class="field-item even"><a href="/categories/drupal" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal</a></div><div class="field-item odd"><a href="/categories/drupal-6" typeof="skos:Concept" property="rdfs:label skos:prefLabel">Drupal 6</a></div><div class="field-item even"><a href="/categories/performance" typeof="skos:Concept" property="rdfs:label skos:prefLabel">performance</a></div></div></div> Fri, 10 Aug 2007 21:37:04 +0000 Caleb Gilbert 58 at http://rocket.local http://rocket.local/blog/drupal-6-performance-and-developer-drupal-release#comments