Drupal

Drupal 8 architecture decisions. Standing on the shoulders of giants.

(Below are the two cents of someone who came to Drupal nearly nine years ago knowing only HTML and CSS, who worked hard to become a full fledged PHP developer, and has continued their programming education vis-a-vis rolling their own JavaScript framework and platform architecture for a large web application)

Until recently I've been less than enthusiastic about Drupal 8, but I've changed my tune (and of course nothing is perfect; I am still a bit concerned/sad that Drupal 8 could possibly be leaving some smaller sites behind, but this is a ship that has likely sailed, so c'est la vie).

What led to this change of heart? Simply, my understanding of just how much, and in what ways, Drupal 7 and previous versions of Drupal differ from widely established/modern techniques and/or best practices of higher level computer science/programming. Arguing whether this is true is comparable to arguing whether square wheels work better than round ones. All of the best programmers, across many different languages, and frameworks don't end up at the same basic conclusions for no reason. This is empirically evident for anyone who has looked around, studied, and experimented with solving problems that, as it turns out, have already been solved before. Drupal 8 will stand on the shoulders of these giants of programming.

To wit, a Drupal 8 programmer will not need to feel behind the times to anyone developing in Java, Rails, Django, or you name it. In fact, a Drupal 8 programmer that embraces the changes and learns the fundamentals underneath the implementations will have a much easier time being able to grasp those other languages/frameworks because Drupal will share more fundamental paradigms with them than it differs by. The reverse is true as well, of course (e.g., programmers from other languages will have an easier time coming to Drupal).

So, to keep this short I'll just close with this - embrace Drupal 8 and the future by starting to learn all the design patterns and OOP fundamentals you can (I recommend not just reading, but making something medium to large sized from scratch). The truth is this; if you think it's time to ditch Drupal and reach for anything other than WordPress -- objects, interfaces, design patterns, dependency injection, event models, etc will more than likely still be waiting for you there (and if it isn't I'd recommend passing, anyway). Drupal's architecture is just catching up with the rest of the more advanced computing world. Hop on board.

P.S., the scale of refactoring that Drupal 8 is undergoing to shed it's imperative/procedural/tightly-coupled skin is truly a monumental task. Allocate your criticism and judgement appropriately, and give things some time to expand and evolve.

11 September, 2013

Build your own JavaScript framework

I've happily reached a significant milestone in the development of my project, finishing a foundational layer that by itself could be valid to open source at some point.

Also:

  • In my last post I explained my reasons for passing on the current crop of JavaScript frameworks. This is a decision that I've been extremely pleased with, as I've managed to conceive/implement my own MVC/MVVM pattern for the project and the endeavor has pushed me much further into my JavaScript education.
  • I have refactored the codebase from using 100% jQuery to about 80% raw JavaScript to 20% jQuery. It is nice to be able to know how to step outside of jQuery and/or deepen one's appreciation for what jQuery is doing under the hood. Also, if you've spent any time on jsperf.com at all you probably know that jQuery is much slower than raw JavaScript for many things. When building an almost-entirely client-side app, speed starts to matter much more than when you have just a couple jQuery snippets on a page, and I want this app to be whip quick. It was a little ackward getting started with plain JS, but once underway I caught the hang of it fairly quickly, and the end result I'm definitely happy with.
  • Currently, I am using localStorage for my data store, and it's been working great for development purposes - it's basically a mock mongoDB for me (I'm storing nested objects in localStorage using a small jQuery library called jStorage). I'm purposely putting off the server side of this application because I want to be able to use the Symfony 2.3 LTS release, which is scheduled for May as middleware between the client-side and mongoDB. The temptation to use node.js is strong, and arguably would be the most performant option, but the reasons for picking Symfony are probably obvious to any Drupal developer.
  • Finally, don't forget - if you are using regex to parse the DOM, you're doing it brain-meltingly wrong!

Cheers all, I'm off to DrupalCamp Sacramento for the weekend.

13 April, 2013

Javascript Framework Redux

Published in: 

Part II in a series of articles about an online web application I am working on. Read Part I

I'm am currently in the process of making my own web service, which will have a single-page-app (SPA) type of structure/interface. This is exactly the type of project that the bevy of JavaScript frameworks like Backbone.js, Angularjs, Ember.js, Knockout.js, and Meteor are build for, thus I've taken it for granted that I would use one of these frameworks as the client-side infrastructure for my app. After much review and fiddling around it turns out that I will not be using one of them. At least not for now. Below are some of reasons why (note these reasons may not apply to you, you may rightly decide that a Javascript framework is a great choice for your project(s) - these are one person's opinions only):

Using any of the frameworks requires more commitment from my code base than I want to give right now

One of my biggest surprises/disappointments after spending a lot of time with Backbone.js and Angularjs was realizing that my (imho) perfectly awesome JavaScript/jQuery foundation layer, which I just slaved over creating, would not only have to be modified - it would have to be thrown out the window. This is because none of these frameworks are a container for your JavaScript - they are a replacement for it. Sure you might use a little bit of JavaScript/jQuery within any one of the frameworks, but overall you'll be doing-Backbone, or doing-Angular, or doing-Ember, or whatever. It's even common advice for Angular experts to tell newbies to not use jQuery at all while they are getting started, so that they can learn the 'Angular way'. Thanks, but no thanks.

I don't want to abandon my universally usable and performant JavaScript/jQuery that I can take and integrate easily into almost anything (well almost anything but these frameworks, apparently). Maybe in the future as one of the JS frameworks becomes more mature and dominant I will reconsider. Honestly, I think a platform would have be as ubiquitous as jQuery has grown to be, for me to volunteer myself for platform lock-in on the level these frameworks require now.

I want to continue building my expertise in JavaScript/jQuery themselves and don't want to have it abstracted away from me

This is a big one for me. I can do what needs to be done with JavaScript, even reasonably well on a case by case basis. But I want to really master it. I don't want to just know Angular, or Backbone. I want to come up with solutions for the real-world, big-picture DOM/JS obstacles that I run into, presumably like the makers of the Javascript frameworks did. I want to eventually have the knowledge that I *could* make my own framework. And at that point, even if I decide to use someone else's framework I will have much more appreciation for my choice, and I will have more knowledge for how to leverage it.

Who are the frameworks a good choice for?

Don't get me wrong I am really glad I spent the time I did looking into Backbone.js and Angularjs (I actually looked at others, but I spent the most time with Backbone and Angular). They are demystified for me now, and there are a lot of good bits, patterns, and big-picture issues that I'm more informed about now than I was previously. I can easily imagine that someone who isn't a JavaScript expert and who just wants to commit to learning the idiosyncrasies of one of the frameworks can build some really great stuff. I can also imagine that true JavaScript masters might be taking their knowledge and one of these frameworks to bring their projects to an even higher level of organization and efficiency (though I'd actually be interested in hearing from the John Resigs of the world regarding their thoughts on current framework implementations).

1 April, 2013

A Side Project: Part 1, Getting Started

"A Side Project" is a series of articles about an online web application I am working on. In Part 1 I share some of the thoughts and motivations which brought me to starting this project in the first place.


The seed for what is new, has some old roots
In 2005 I became involved with Drupal after discovering it in the course of working on an ambitious side project of mine. It was a discovery that would change my life, professionally and otherwise, much for the better. Ironically, the very thing, which got me into Drupal in the first place (e.g., online side projects) had became one of the unfortunate casualties in more recent years as my professional Drupal activities increased. Sure, I had little projects - went to meetups, cons, made some patches here, contributed this there - but building my own online site/service from the ground up with the intention/hope of achieving something big, that went by the wayside.

A couple months ago I realized that this situation of no-large-side-projects needed to change for many reasons. A few of which are:
1. Side projects are fun. I personally enjoy having something to obsess over and tinker with in my free time.
2. Side projects provide an opportunity to check out new (or at least new to me) technologies.
3. It's one thing to develop web technologies. It's another thing to be responsible for managing a real live website, its content, its community, and everything else that comes with it. It's my belief that experience with the latter helps the former.

So once I convinced myself that I really needed a project of my own to work on again, the only thing left was to figure out WHAT I would work on. For this I took some time to consider things, as I realized that building one more version of a TodoMVC is probably not going to change the world, and probably wouldn't teach me much either. I wanted to come up with an idea for something that would really matter to me, something I can get excited about, and actually use. Something REAL.

And you know what? I have found that kind of idea for myself (more on that later), and I've been working very hard on bring my ideas to life. I'm not at a launchable point yet, but I'm feel like the decision to start this project is validated every day I wake up right now. Besides just feeling generally inspired and motivated, I've been learning a TON of stuff (more on this later, too!).

Thanks for reading Part 1 of this series.

19 March, 2013

Example Varnish VCL for a Drupal / Pressflow site

A few months ago I set up Varnish on my Macbook Pro and have deployed it for a production site which serves anonymous and (a lot of) authenticated users. Initially, I spent a couple months just running it in my local environment, including backporting the Varnish.module to Drupal 5. In retrospect, I'm glad that I spent the time to learn how Varnish and it's configuration file works before deploying it, as it's paid off in a big way as our production site now has something which is equivalent to:

  • ...an in-memory static file server for all users (e.g., the equivalent of hooking up something like nginx or lighttpd as a front end to Apache (or whatever you're using).
  • ...an in-memory boost.module in terms of database-relief for anonymous users.

Contrary to popular belief the two items above are in no way an automatic benefit of simply installing Varnish. If the configuration file, and Drupal installation, is not massaged with care one definitely won't get the database relief from anonymous page caching, and the benefits from Varnish-as-a-static-file server will not nearly be optimized. Bottom line Varnish can be a temperamental piece of software. It only gives back what you put into it.

To this end, the settings in the Varnish VCL file can make or break whether you get a substantial benefit from it. Below is an example VCL file, which was formed from a good amount of research and a lot of trial and error:

# If you're running a single site on a server, or else want all sites
# on a server to go through Varnish you'd only need one of the following backends.
# Showing different possibilities for those who have sites that they
# don't want to run Varnish on. In this example file, Varnish is assumed to
# be running on port 80, and Apache (or whatever) on port 8080.

backend default {
.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.max_connections = 800;
}

backend sitea {
.host = "xxx.xx.xxx.103";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.max_connections = 800;
}

backend siteb {
.host = "xxx.xx.xxx.104";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.max_connections = 800;
}

sub vcl_recv {

# Now we use the different backends based on the uri of the site. Again, this is
# not needed if you're running a single site on a server
if (req.http.host ~ "sitea.com$") {
set req.backend = sitea;
} else if (req.http.host ~ "siteb.com$") {
set req.backend = siteb;
} else {
# Use the default backend for all other requests
set req.backend = default;
}

# Allow a grace period for offering "stale" data in case backend lags
set req.grace = 5m;

remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;

# Properly handle different encoding types
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}

# Force lookup if the request is a no-cache request from the client
if (req.http.Cache-Control ~ "no-cache") {
return (pass);
}

## Default request checks
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
# Non-RFC2616 or CONNECT which is weird.
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
# We only deal with GET and HEAD by default
return (pass);
}

## Modified from default to allow caching if cookies are set, but not http auth
if (req.http.Authorization) {
/* Not cacheable by default */
return (pass);
}
## This would make varnish skip caching for this particular site
if (req.http.host ~ "internet-safety.yoursphere.com$") {
return (pass);
}

# This makes varnish skip caching for every site except this one
# Commented out here, but shown for sake of some use cases
#if (req.http.host != "sitea.com") {
#   return (pass);
#}

## Remove has_js and Google Analytics cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");

## Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

## Remove empty cookies.
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}

## Pass cron jobs
if (req.url ~ "cron.php") {
return (pass);
}

# Pass server-status
if (req.url ~ ".*/server-status$") {
return (pass);
}

# Don't cache install.php
if (req.url ~ "install.php") {
return (pass);
}
 
# Cache things with these extensions
if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
    return (lookup);
}

# Don't cache Drupal logged-in user sessions
# LOGGED_IN is the cookie that earlier version of Pressflow sets
# VARNISH is the cookie which the varnish.module sets
if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID|LOGGED_IN)") {
return (pass);
}

}

sub vcl_fetch {

  # Grace to allow varnish to serve content if backend is lagged
  set obj.grace = 5m;

# These status codes should always pass through and never cache.
if (obj.status == 404 || obj.status == 503 || obj.status == 500) {
set obj.http.X-Cacheable = "NO: obj.status";
set obj.http.X-Cacheable-status = obj.status;
return (pass);
}
   
  if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
   unset obj.http.set-cookie;
}

if (req.url ~ "^/files/") {
unset req.http.Set-Cookie;
set obj.cacheable = true;
}

if (req.url ~ "^/sites/") {
unset req.http.Set-Cookie;
set obj.cacheable = true;
}

if (!obj.cacheable) {
set obj.http.X-Cacheable = "NO: !obj.cacheable";
return (pass);
}
else {
# From http://varnish-cache.org/wiki/VCLExampleLongerCaching
/* Remove Expires from backend, it's not long enough */
unset obj.http.expires;

# These TTLs are based on the specific paths and may not apply to your site.
# You could just set a single default TTL if you want.
if (req.url ~ "(.js|.css)$") {
set obj.ttl = 60m; // js and css files ttl 60 minutes
}
else if (req.url ~ "(^/articles/)|(^/tags/)|(^/taxonomy/)") {
set obj.ttl = 10m; // list page ttl 10 minutes
}
else if (req.url ~ "^/article/") {
set obj.ttl = 5m; // article ttl 5 minutes
}
else {
set obj.ttl = 45m; // default ttl 45 minutes
}

/* marker for vcl_deliver to reset Age: */
set obj.http.magicmarker = "1";

# All tests passed, therefore item is cacheable
set obj.http.X-Cacheable = "YES";
}

return (deliver);
}

sub vcl_deliver {

  # From http://varnish-cache.org/wiki/VCLExampleLongerCaching
  if (resp.http.magicmarker) {
     /* Remove the magic marker */
     unset resp.http.magicmarker;

     /* By definition we have a fresh object */
     set resp.http.age = "0";
   }

   #add cache hit data
   if (obj.hits > 0) {
     #if hit add hit count
     set resp.http.X-Cache = "HIT";
     set resp.http.X-Cache-Hits = obj.hits;
   }
else {
     set resp.http.X-Cache = "MISS";
   }

}

sub vcl_error {

if (obj.status == 503 && req.restarts < 5) {
set obj.http.X-Restarts = req.restarts;
restart;
}

}

# Added to let users force refresh
sub vcl_hit {

if (!obj.cacheable) {
pass;
}

if (req.http.Cache-Control ~ "no-cache") {
# Ignore requests via proxy caches,  IE users and badly behaved crawlers
# like msnbot that send no-cache with every request.
if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE")) {
set obj.ttl = 0s;
return (restart);
}
}

deliver;

}

Some links from places which helped me arrive at this VCL:
* VCL Examples
* Lazy sessions with PF5 and Varnish problem
* Configure Varnish for Pressflow
* Varnish config - default.vcl

18 May, 2010

Some highlights from Drupalcon San Francisco

Published in: 

Drupalcon San Francisco was great. Here is some of what stood out to me, in no particular order:

  • Tim O'Reilly's keynote was great (see link for it in comments). It put perspective on where Drupal fits into the larger world and reminded me that I actually came to Drupal for a reason, not just for technology or Drupal itself. While listening to the keynote, I couldn't help but think that Dries' personal push for RDF in core (which I've seen some head-scratching about from some) might owe part of its genesis to his own talks with O'Reilly.
  • The presentation YOU SHALL NOT PASS: Managing Expectations and Boundaries of Clients is one of the hidden gems of Drupalcon. If you're not project managing your clients/work the way they describe, you're doing it wrong (or at least for less money, more aggravation, and less satisfaction!).
  • Larry Garfield's session Objectifying PHP was a wonderful session for anyone who is ready to move to the next level with object oriented methods (pun intended) and practices.
  • The money, as well as the number of people, and type of people involved with Drupal these days has definitely changed the feel of the conference as compared to earlier ones I've attended. This is true even comparing it to DC which was only last year, let alone the first one I went to 4 years ago (OSCMS). There is no good or bad implied here, just worth noting since it has implications (and things will likely continue in this direction for a while).
  • For the past year or so it's been cliche to say, 'No one person one can know everything about Drupal anymore'. For me the new saying is, 'No one person would even *want* to know everything about Drupal anymore'. There are vast, vast knowledge areas within Drupal that have their own following, experts, and activity levels. Don't get me wrong, it's *all* extremely interesting, but gazooks...
  • For now videos/screencasts for presentations (they don't all seem to have them) can be found by going to this page and clicking through to the individual presentation page you're interested in. UPDATE: Videos posted here too.
  • Hottest Drupal-related career: Project/QA manager. The need for developers and themers, is plenty hot of course, but I'd be willing to bet that at least some of the larger Drupal-based companies/shops would choose a competent project/QA manager over a competent developer if forced to choose only one.
  • The way Drupal 7's $page array, hook_page_alter, and drupal_render works, simultaneously makes me want to use Drupal 7 RIGHT DARN NOW, as well as makes me wonder just how horrifically these features will be abused. (who needs a custom theme anymore - Garland with a bunch of hook_page_alters should work just fine, right) ;)
  • If you want the sneak peak for the next North American Drupalcon, look no further.

If you have a highlight you'd like to share please leave a comment.

22 April, 2010

Pages

Subscribe to RSS - Drupal