Wednesday, November 28, 2012

CiviCRM developer secret handshake

I'm at a CiviCRM developer training and have to share my discoveries. Before yesterday, I always felt like a CiviCRM hacker, without really knowing why, and now I think I do. For me, the difference is between knowing a few tricks to accomplish what I want to do, compared with actually understanding the code as a whole and feeling like I could responsibly contribute stuff and not just cross my fingers that I'm not missing something important. So, no, it's not a secret handshake, but here's  what I've learned:

1. Dispatcher and permissions

In Drupal, we have a bunch of code called the "menu system" which takes care of (among other things, like the actual visible menu), how a particular url maps onto a specific piece of code to be executed - i.e. which callback function gets invoked. This mapping of urls to callbacks is implemented by modules in the hook_menu function and stored in a table that gets rebuilt when modules get added and rebuilt or the menu cache gets cleared. This function in Drupal also takes care of the top level of permissioning - i.e. who gets to call which urls.

CiviCRM, because it's CMS agnostic, doesn't use Drupal's menu for it's individual urls, but instead just exposes to Drupal a single callback path "civicrm" and then deals with the dispatching and permissioning itself.

So here's what I learned: the way CiviCRM implements dispatching and permissioning is similar to Drupal, but it's done via xml files rather than implementing a hook. So, dive into the CiviCRM code base, pick your favourite module (e.g. Events) and look in there for xml files. It's got a pretty obvious structure.

2. Module Extensions

CiviCRM now has a thing equivalent to a Drupal module, it's called a Module Extension.

"Equivalent to" is an expression that gets a lot of abuse by sales types, so let me be more specific: in Drupal, there are really three parts to "core", namely:
a) the really core essential stuff that mostly lives in the includes directory + the index.php file.
b) the modules that you can't disable
c) the other modules that are distributed with Drupal
What makes module development work in Drupal is the discipline this structure imposes on the code as a whole - even modules that you can't disable are built with the same tools as modules that you can contribute, the same hooks, etc. So, at least in theory, a contributed module can do things as powerful as the core.

What we get with a Module Extension is access to the same tools and structure that CiviCRM core is built with. Okay, maybe not quite, not yet, but that's the promise and I'm going to hold them to it.

What this means for CiviCRM developers is: your standard tools for customizing CiviCRM look like this:

a. Custom tpl and php directories - difficult to maintain, limited in what it can do and easy to implement, or difficult to go beyond the really basic stuff. In other words, a documented way to hack core. Roughly "equivalent" to using the theme layer in Drupal to do your customizations.

b. Drupal module - use the civicrm api to expose custom callback functions that  access the civicrm code, db, etc. A nice development environment, but also limited in how much it can sensibly accomplish. Also not portable to other CMSs. Feels kind of like throwing rocks, involves some crossing of fingers.

c. The CiviCRM module extension. Can now replace my big hacks that currently use a grab bag of 1. and 2. - makes them more easily shareable across installations and CMSs.

3. Development Tools

Okay, so we've now got access to the same mechanism that builds CiviCRM functionality (with the usual disclaimers), but we're not core developers so we don't know the code base as well. With Drupal we have the api documentation, books and other documentation. We don't have quite the same level of maturity with CiviCRM, but here's what I've found:

a. the api explorer. You get this automatically with your own civicrm installation, it's a bit beta, but a cool interface and has the basics. A real important difference with Drupal is that the actual api code is a separate, thin layer over the actual code, so most of the time you need to look at the core code to figure out arguments.

b. the core code. See it here:, it's linked with the warning "code level documentation".

c. the list of hooks. CiviCRM adopted Drupal's hook idea a while ago. See:

d. Eileen's civicrm_developer module (the civicrm version of the Drupal devel module). I'm going on hearsay from Tim and respect for Eileen's past work, but I doubt you'll be disappointed.

e. Tim's civix. This should almost be first on the list - it's module extension generator. The only non-obvious thing I ran into was that you first create the module extension, then you use a separate subsequent command to add stuff for different types of things you want your module to do. For example, if you just want to use the hooks to intervene on some existing pages, you can just use the basic scaffolding. If you want to create a new url with functionality, then you add in a "page". Yes Tim, it's documented, I just didn't read it carefully.

Conclusion: many thanks to Tim for explaining all this, and to Joe and Louis-Charles for getting this developer training to happen.

Wednesday, October 17, 2012

Time only goes forward, or does it? A time turner for civimail bad links!

Recently a client sent out an email with a bad link in it.

Normally, I'd sympathise with them but tell them that the horse has bolted and there's no point in closing the barn doors (because I like to use old fasioned expressions).  (And yes - if the link was on my server, I could give them an apache redirect.)

But, if the message went out via civimail and you're tracking click-throughs of links (actually this trick also works with simplenews if you're using simplenews_stats), then there's still hope.

This is because the actual link embedded in the mail isn't the final destination, but a civicrm url with a magic trackable url id, which then redirects to the final url, which is stored in a civicrm table. The rest you can probably do as a simple homework exercise, but here's the details since I've got more room for this post.

Start with loading up a mysql client and use your civicrm database, then find out the right id using this:

select * from civicrm_mailing_trackable_url order by mailing_id desc limit 30;

The first thing you'll want to do is fix that with something like

update civicrm_mailing_trackable_url set url = 'your-corrected-url' where id = the-id-of-the-bad-url;

If you caught it early enough, then you're done, but you might want to try this query as well:

select distinct contact_id from civicrm_mailing_event_trackable_url_open o inner join civicrm_mailing_event_queue q on o.event_queue_id = where trackable_url_id = the-id-of-the-bad-url;

Which will give you a list of the contact ids of people who have already clicked through your bad url - in which case you can resend them the corrected version and some humble pie.

Friday, October 12, 2012

CiviCRM user and administrator training in Toronto, November 14-15 2012

A CiviCRM user and administration training in Toronto has just been announced for Nov 14-15, here:

If, like most people including myself, you wish you understood a bit more about this multi-headed beast called CiviCRM, then this should be a great opportunity.

If you've grown a little complacent with your CiviCRM install, this is a great place to learn about all the new things you can do with it, and hear what others like you are already doing. Non-profit types already know that community is an essential part of success and sustainability, and events like this put that into practice within the CiviCRM ecosystem.

In addition, there's a developer training and a 'code sprint', for the weeks following.

And it reminds me that it's past time to publish a new civicrm integration module I wrote this year.

Thanks to Joe Murray and Louis-Charles Lavallee for getting this going!

Thursday, October 04, 2012

Drupal Commerce IATS

As faithful readers of this blog will know, I'm a fan of IATS as a payment processor solution for non-profits. Many years ago I wrote a CiviCRM payment plugin that is still in wide use by most of my clients and many others as well.

Although my Drupal module contributions have been minimal over the recent past, I'm happy to announce a new one to integrate IATS payments into the new Drupal Commerce module/distro/framework/thingy.


Many thanks to Karin Gerritson, my colleague at Blackfly Solutions, and her project to develop a new site for the school she sends here kids to, here:

What this means for the rest of the world: non-profits have a nice solution to get e-commerce going on Drupal 7 with a minimum of cost and hassle.

Drupal 7 migration: a script for image + image_attach

Inspired by this post:

I thought I'd dig out and contribute a script I wrote for a similar issue I had with a number of old sites.

Specifically - in the old days before imagecache, one way to safely enable your site editors to add pictures to posts was using a combination of the image module with the image attach module. Functionally, most of the sites I've seen it on were doing the same thing as the imagecache, only not as well, and now that D7 is mainstream I can't prolong keeping it going any longer.

The main difference with Laura's post above is trying to deal with the image attachment bit - i.e. getting the images into the fields of the posts to which the original image nodes where being 'attached'.

The other key difference - this script was written for D6, so do this before you do your D7 upgrade. It's making use of the D6 table structure, it's not written using high-level api's! D7's table structure is much more complicated, so trying to do this after the D7 upgrade would be more difficult I suspect.

Also: doing this only with a script would be too general a tool I think. In any case, I simplified the problem by manually creating the new image fields and then hard coding those field names into my script.

Conclusion: feel free to recycle my script below, but make sure you

1. create the new fields first, and then modify my "$convert" array below.
2. try it on a dev copy first. Also, you'll see some commented out code that I uncomment on the first run as a way of checking out what it will do.

Notes: because an image node image can be attached to multiple nodes, i'm copying the original image files, not reusing them in their same location. So when you're done, you can remove the files/image directory where the image module put it's stuff.

// code to convert image module use to imagecache

$convert = array(
  'newsletter_article' => 'field_newsletter_image',
  'page' => 'field_page_image',
  'press' => 'field_press_image',

// you can run this code first to see what sizes were being attached to which node types
/* foreach(node_get_types() as $type => $info) {
  $image_attach = variable_get("image_attach_$type",'');
  if ($image_attach) {
    echo "\n $type";   
    echo db_result(db_query('DESC content_type_'.$type));
} */
$sizes = variable_get('image_sizes','');
// print_r($sizes); die();

foreach($convert as $type => $image_field) {
  //echo $image_field;
  $iff = $image_field.'_fid';
  $config = db_fetch_object(db_query("SELECT * FROM {content_node_field_instance} WHERE field_name = '%s'",$image_field));
  $settings = unserialize($config->widget_settings);
  $dir = file_create_path($settings['file_path']);
  // print_r($config);
  // print_r($settings);
  $status = file_check_directory($dir,(FILE_CREATE_DIRECTORY | FILE_MODIFY_PERMISSIONS));
  if (!$status) {
    die("Unable to create directory!!!");
    // TODO: for rfc, i don't have directory create permission, need to run initialization as nobody
  // for each image, move the _original one to the new directory, 
  // remove the other ones, and update the db
  $result = db_query("SELECT n.vid,ia.*,i.fid FROM {node} n INNER JOIN {image_attach} ia ON n.nid = ia.nid INNER JOIN {image} i on ia.iid = i.nid AND i.image_size = '_original' WH
ERE n.type = '%s'",$type);
  while ($ia = db_fetch_object($result)) {
    $file = db_fetch_object(db_query("SELECT * FROM files where fid = %d",$ia->fid));
    $name = basename($filepath = $file->filepath);
    // print_r($ia);
    // die("$old_path -> $new_path");
    // get the current fid!
    $good = file_copy($filepath,$settings['file_path'],FILE_EXISTS_REPLACE);
    if (!$good) {
      print_r($file); print_r($ia); die($filepath);
    else {
      db_query("INSERT INTO {files} (uid,filename,filepath,filemime,filesize,status,timestamp) VALUES (%d,'%s','%s','%s',%d,%d,%d)",$file->uid,$name,$filepath,$file->filemime,$fil
      $fid = db_last_insert_id('files','fid');
      if (!db_result(db_query("SELECT vid FROM {content_type_$type} WHERE vid = %d",$ia->vid))) {
        db_query("INSERT INTO {content_type_$type} (nid,vid,$iff) VALUES (%d,%d,%d)",$ia->vid, $ia->nid, $fid);
      else {
        db_query("UPDATE {content_type_$type} SET $iff = %d WHERE vid = %d",$fid, $ia->vid);
    db_query('DELETE FROM {image_attach} WHERE nid = %d AND iid = %d',$ia->nid,$ia->iid);
    // print_r($ia); die("$old_path $new_path");
  // echo $status ? ': good' : ': bad';
// print_r(array_keys($types));

Wednesday, March 28, 2012

Varnish and Drupal

Varnish is front-end web proxy that I've been using with my Drupal sites for 18 months now. I talked about it in a presentation at Drupal camp with Khalid.

A front-end web proxy means that instead of visitors directly accessing the Drupal site, they go through a caching layer. I like to think of varnish as a protective bubble around Drupal.

Using Varnish has a number of benefits, and in my presentation I was particularly interested in demonstrating how it enables a site to survive huge spikes of sudden traffic, beyond what a web server would normally be able to handle. So that's one of the benefits.

Another important benefit is to reduce the load on the server, enabling better response. Or more cynically, it would enable you to pile more sites onto your server. But to explore this more closely: by moving most of your 'easy' page loads out of the php/apache stack, you can tune your stack better to what it's got to do - and you're not wasting 128Mb or more of RAM when serving an image, which is what happens by default on a server with apache and mod_php.

But even if you've got a pretty ordinary site and no server load issues, here's a reason why Varnish benefits you: your visitors' experience of speed. The key is that the traffic that is most sensitive to the load speed of your site is anonymous traffic to your front page. That's because new visitors (who usually end up first on your front page) are the ones you have to engage immediately or they go away (technically known as a "bounce"). A typical site might have a bounce rate of 50%, so any slowness in loading that front page by anonymous visitors is going to make a difference to whether that visitor continues to explore the site, or just leaves out of the tedium of waiting (who wants to wade through a slow loading site?). And here's why Varnish is the perfect solution: it does its best caching of your most visited pages to anonymous visitors!

And finally, some numbers. I was curious just how much traffic varnish might divert from a 'regular' drupal site, and haven't found anybody else's numbers.  Here are my last 2 months on one of my servers. The short answer is: about 30% of the visits go entirely to Varnish, more than half the hits (mainly because varnish caches images/css/js files for up to 2 weeks, even for logged-in users), and about half the total bandwidth.

Anybody got any comparable statistics?


MonthUnique visitorsNumber of visitsPagesHits
Feb 2012471448530546285476603111.79 GB
Mar 2012399636941748503083841612.76 GB


MonthUnique visitorsNumber of visitsPagesHits
Feb 201260082118672449906226430724.89 GB
Mar 20125171397369452394211089024.44 GB