Twitter this

Powered by MariaDB Powered by nginx ...

W3 Total Cache plugin

W3 Total Cache

This post describes my experiences with the W3 Total Cache (w3tc) plugin in relation to my nginx install. I have bumped against a few hiccups along the way that I want to share. It all started when I took up ab, the apache benchmark tool. I wanted to see how this nginx powered website performed with the spawn-fci and mariaDB setup. Along the way I wanted to exploit the amazing ability of nginx to deliver static files. It's just really good at that so I wanted to start caching the thing after seeing very poor performance due to the regular load of php cgi,  I had a mission all the sudden.

My slick nginx powered wordpress is slow

Some simple benchmarks taken at various points, I tried 3 different cache plugins, W3TC as the third and it's still running now (although I reinstalled it once).   The first benchmark below, is the 'before cache' one, no improvements yet done, besides running on mariaDB tables instead of innodb,  switching off revisions in wordpress and running on nginx.  Notice the number of failed ones...

Before the makeover

1
2
3
4
5
6
7
8
9
10
glenn@crashy:~$ ab -c5 -n200 http://byte-consult.be/
Finished 200 requests
Concurrency Level:      5
Time taken for tests:   62.473 seconds
Complete requests:      200
Failed requests:        45
Requests per second:    3.20 [#/sec] (mean)
Time per request:       1561.826 [ms] (mean)
Time per request:       312.365 [ms] (mean, across all concurrent requests)
Transfer rate:          99.25 [Kbytes/sec] received

This is just plain insulting, 3.2/s is crap!  Admittedly  I'm on the Linode 768 plan but still, these guys are great at what they do so I believe I can do better here with their/my resources.

I tried SuperCache plugin with static pages.  Configuring this for nginx proved to be a pain in the butt.  I took a benchmark with static file caching right before un-installing this.  I didn't like the way it worked for me. No nginx support.   Still, the performance was about the same as with W3TC.

With SuperCache plugin

1
2
3
4
5
6
7
8
9
Finished 200 requests
Concurrency Level:      5
Time taken for tests:   6.438 seconds
Complete requests:      200
Failed requests:        0
Requests per second:    31.07 [#/sec] (mean)
Time per request:       160.946 [ms] (mean)
Time per request:       32.189 [ms] (mean, across all concurrent requests)
Transfer rate:          1242.66 [Kbytes/sec] received

I'm not going to mention the other cache plugin, it was total crap, let's not waste our time on it.  So time to move to WT3C.  After installation, switching the page cache with disk advanced mode to on.

With wt3c plugin

1
2
3
4
5
6
7
8
9
10
glenn@crashy:~$ ab -c5 -n200 http://byte-consult.be/
Finished 200 requests
Concurrency Level:      5
Time taken for tests:   6.376 seconds
Complete requests:      200
Failed requests:        0
Requests per second:    31.37 [#/sec] (mean)
Time per request:       159.403 [ms] (mean)
Time per request:       31.881 [ms] (mean, across all concurrent requests)
Transfer rate:          1256.03 [Kbytes/sec] received

As you can see here, there is not too much difference between the two, but we aren't done yet.  When logging into the WP Admin page I see a Performance section at the left bottom under Settings.  That is new.  I like seeing everything belonging to a single goal in the same place.  I've had a big issue in getting this page to load by the way but that is food for later.  Once I put a workaround in place for this issue I could use it.

I see lot's of support for goodies I like.   For example memcached support.  For this small site using memcached seems like a done deal, I have memory left over I can put to use.  So I decided I want object/query caching done in memcached, I'll select that, put some bigger values in the caching (5 minutes=300sec) and Deploy it.  Let's pimp the tests up a bit ...

With more threads

1
2
3
4
5
6
7
8
9
10
glenn@crashy:~$ ab -c50 -n1000 http://byte-consult.be/
Finished 1000 requests
Concurrency Level:      50
Time taken for tests:   20.244 seconds
Complete requests:      1000
Failed requests:        0
Requests per second:    49.40 [#/sec] (mean)
Time per request:       1012.204 [ms] (mean)
Time per request:       20.244 [ms] (mean, across all concurrent requests)
Transfer rate:          1977.86 [Kbytes/sec] received

Caching on for objects/queries with memcached

1
2
3
4
5
6
7
8
9
10
11
glenn@crashy:~$ ab -c50 -n200 http://byte-consult.be/
Benchmarking byte-consult.be (be patient)
Finished 200 requests
Concurrency Level:      50
Time taken for tests:   4.005 seconds
Complete requests:      200
Failed requests:        0
Requests per second:    49.94 [#/sec] (mean)
Time per request:       1001.295 [ms] (mean)
Time per request:       20.026 [ms] (mean, across all concurrent requests)
Transfer rate:          1923.44 [Kbytes/sec] received

Professionally, I have had issues with memcached with trying to store larger values that what most typical uses would.  I've used it outside the web context for years now but I've seen some issues with it along the way.  Still, I don't expect any problems when using it here, for what it was intended for in the first place, which is beefing up sites like they are Duke Nukem's on steroids.  I'm at 50/s now using Page + Object + Query caching.

From a faster connection with tuning/bugfixes

1
2
3
4
5
6
7
8
9
10
obosql003:~# ab -c50 -n200 http://byte-consult.be/
Finished 200 requests
Concurrency Level:      50
Time taken for tests:   1.982 seconds
Complete requests:      200
Failed requests:        0
Requests per second:    100.90 [#/sec] (mean)
Time per request:       495.548 [ms] (mean)
Time per request:       9.911 [ms] (mean, across all concurrent requests)
Transfer rate:          3963.52 [Kbytes/sec] received

Much better!

That looks really good there, 100/s!!!  Does everyone agree that this benchmark is totally not complete nor do I claim it is.  It's just that.... erm.   The difference is overwhelming.  It's a factor 50 faster than before the makover....  Even the worst benchmarks get some of it right.

Now about those bugs I fixed along the way

The timout problem

A hanging Performance tab, only on nginx.  When I clicked anything in that new Performance tab, the whole thing would hang for a while and then time out.   I've investigated this at the mysql level,  at the php-cgi level and the nginx level.  With the latter two I have strace 'd those processes. If you have this issue after installing the plugin you need to put this workaround in the following file : wp-content/plugins/w3-total-cache/lib/W3/Plugin/TotalCache.php

1
2
3
if (!$this->test_rewrite_pgcache()) {
...
}

Find this line (for me at line 1781), and change this to

1
2
3
if (1==0) {
...
}

You don't want to have it call $this->test_rewrite_pgcache() . It made things time-out and I don't know why.  What it do know is that it makes a special call to the nginx webserver that looks like this GET /w3tc_rewrite_test. It's a test.

In the nginx access log file I see :

1
2
178.79.156.25 - "GET /w3tc_rewrite_test HTTP/1.1" 504 183 "-" "W3 Total Cache/0.9.2.3"
178.79.156.25 - "GET /w3tc_rewrite_test HTTP/1.0" 200 2 "-" "Wget/1.12 (linux-gnu)"

This is where it get's interesting, I noticed that right after that call to nginx it goes south. (you  see the 504 there).  When I call this from my browser or with wget, it works perfectly(as per status 200).  All it does is return 'OK', which sounds really easy to handle.  But for some reason that whole thing doesn't get back.  So it just doesn't work when W3TC requests it coming from PHP, it does work on every other occasion, by hand this time.  As stated previously, it's just a test, a check to see if you have the rewrite rules in place on nginx.  The request matches the following rewrite rule from the nginx config:

Download
      # BEGIN W3TC Page Cache core
      rewrite ^(.*\/)?w3tc_rewrite_test$ $1?w3tc_rewrite_test=1 last;
 

So now it knows you have that rule there.  What it concludes from there on is too much trouble to find out and boring too once I fixed this call anyway.  The panel loaded in a blink of an eye after that. I have an example php-cgi strace dump here for the curious. Check out what happens after this line:

Download
send(7, "GET /w3tc_rewrite_test HTTP/1.1\r"..., 107, MSG_NOSIGNAL) = 107
 

Anyway, I managed to get the rules written to the config before that panel started locking up (you can run it once before you switch page caching on). But I'm pretty confident I have every needed rule in my nginx.conf already now. I just wanted that panel to work so I can configure the rest of the cache.  Thus,  I switched off that test by changing those particular lines.

Memcache md5 problem

Another issue I noticed, and this one is I think a bug is the definition of the memcached instance key, basically to make your key = value pair unique.   Take a look at this code from wp-content/plugins/w3-total-cache/lib/W3/Cache.php. This is what happens when you implode a nested array. Implode doesn't handle them as per php documentation.  Now it becomes unpredictable, and it sure looks like it needs to be predictable.

1
2
// this doesn't work on nested arrays
$instance_key = sprintf('%s_%s', $engine, md5(implode('', $config)));

You get this error if you try:

1
2
3
Notice: Array to string conversion in
/usr/share/wordpress/wp-content/plugins/w3-total-cache/lib/W3/Cache.php on line 52 Array (
[servers] => Array ( [0] => 127.0.0.1:11211 ) [persistant] => 1 )

So I replaced this with json_encode , perfectly designed to handle this kind of an Array

1
$instance_key = sprintf('%s_%s', $engine, md5(json_encode($config)));

I've tried to find if there is any other place in the code that would generate this and one other file popped up: /usr/share/wordpress/wp-content/plugins/w3-total-cache/lib/W3/Cdn.php. You should fix this issue in there too. Another warning message will be gone which spawn-fcgi doesn't like here.

Other fixes

I also fixed all calls to deprecated functions in lots of plugins I have running.  I've also fixed a bug in the Raindrop template I'm currently using as well that made my permalinks to "/tag/%s/"  URL locations to be wrongfully generated due to a wrong function call to an non-existing function.   And lastly I also fixed an issue in PiWik, the analytics software I'm using that threw a warning on every hit.  In the plugin wp-content/plugins/piwik-analytics/piwikanalytics.php line: 256

1
2
3
4
5
if (function_exists("get_option")) {
//if ($wp_siteid_takes_precedence)
$options  = get_option('PiwikAnalyticsPP');
$siteid = $options['siteid'];
}

Since $wp_siteid_takes_precedence is nowhere to be found in the code it arrives there uninitialised and throws a warning.  Finding from where that variable should come and why it's unknown at that point in the code is going to take me too much time so I commented it out, which looks like the right choice. Bit off topic but while we are at it we might as well dump all.

Lots like these :

1
2
3
Notice: has_cap was called with an argument that is deprecated since version 2.0!
Usage of user levels by plugins and themes is deprecated. Use roles and capabilities
instead. in /usr/share/wordpress/wp-includes/functions.php on line 3321

and those

1
2
Notice: attribute_escape is deprecated since version 2.8! Use esc_attr() instead.
in /usr/share/wordpress/wp-includes/functions.php on line 3237

Although I don't think the last one is related to this plugin. It was part of a bunch of fixes.

The rewrite rules for nginx

W3TC Page Cache

Download
      # BEGIN W3TC Page Cache cache
      location ~ /wp-content/w3tc/pgcache.*html$ {
         add_header X-Powered-By "W3 Total Cache/0.9.2.3";
         add_header Vary "Accept-Encoding, Cookie";
      }
 
      location ~ /wp-content/w3tc/pgcache.*gzip$ {
         gzip off;
         types {}
         default_type text/html;
         add_header X-Powered-By "W3 Total Cache/0.9.2.3";
         add_header Vary "Accept-Encoding, Cookie";
         add_header Content-Encoding gzip;
      }
      # END W3TC Page Cache cache
 

W3TC Browser Cache

Download
      # BEGIN W3TC Browser Cache
      gzip on;
      gzip_types text/css application/x-javascript text/richtext image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;
      location ~ \.(css|js)$ {
            add_header X-Powered-By "W3 Total Cache/0.9.2.3";
      }
      location ~ \.(html|htm|rtf|rtx|svg|svgz|txt|xsd|xsl|xml)$ {
            add_header X-Powered-By "W3 Total Cache/0.9.2.3";
      }
      location ~ \.(asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|swf|tar|tif|tiff|wav|wma|wri|xla|xls|xlsx|xlt|xlw|zip)$ {
            add_header X-Powered-By "W3 Total Cache/0.9.2.3";
      }
      # END W3TC Browser Cache
 

W3TC Page Cache core

Download
      # BEGIN W3TC Page Cache core
      rewrite ^(.*\/)?w3tc_rewrite_test$ $1?w3tc_rewrite_test=1 last;
      set $w3tc_rewrite 1;
      if ($request_method = POST) {
            set $w3tc_rewrite 0;
      }
      if ($query_string != "") {
            set $w3tc_rewrite 0;
      }
      set $w3tc_rewrite2 1;
      if ($request_uri !~ \/$) {
            set $w3tc_rewrite2 0;
      }
      if ($request_uri ~* "(sitemap\.xml(\.gz)?)") {
         set $w3tc_rewrite2 1;
      }
      if ($w3tc_rewrite2 != 1) {
         set $w3tc_rewrite 0;
      }
      set $w3tc_rewrite3 1;
      if ($request_uri ~* "(\/wp-admin\/|\/xmlrpc.php|\/wp-(app|cron|login|register|mail)\.php|wp-.*\.php|index\.php)") {
         set $w3tc_rewrite3 0;
      }
      if ($request_uri ~* "(wp\-comments\-popup\.php|wp\-links\-opml\.php|wp\-locations\.php)") {
         set $w3tc_rewrite3 1;
      }
      if ($w3tc_rewrite3 != 1) {
         set $w3tc_rewrite 0;
      }
      if ($http_cookie ~* "(comment_author|wp\-postpass|wordpress_\[a\-f0\-9\]\+|wordpress_logged_in)") {
         set $w3tc_rewrite 0;
      }
      if ($http_user_agent ~* "(W3\ Total\ Cache/0\.9\.2\.3)") {
         set $w3tc_rewrite 0;
      }
      set $w3tc_ua "";
      if ($http_user_agent ~* "(2\.0\ mmp|240x320|alcatel|amoi|asus|au\-mic|audiovox|avantgo|benq|bird|blackberry|blazer|cdm|cellphone|danger|ddipocket|docomo|dopod|elaine/3\.0|ericsson|eudoraweb|fly|haier|hiptop|hp\.ipaq|htc|huawei|i\-mobile|iemobile|j\-phone|kddi|konka|kwc|kyocera/wx310k|lenovo|lg|lg/u990|lge\ vx|midp|midp\-2\.0|mmef20|mmp|mobilephone|mot\-v|motorola|netfront|newgen|newt|nintendo\ ds|nintendo\ wii|nitro|nokia|novarra|o2|openweb|opera\ mobi|opera\.mobi|palm|panasonic|pantech|pdxgw|pg|philips|phone|playstation\ portable|portalmmm|\bppc\b|proxinet|psp|qtek|sagem|samsung|sanyo|sch|sec|sendo|sgh|sharp|sharp\-tq\-gx10|small|smartphone|softbank|sonyericsson|sph|symbian|symbian\ os|symbianos|toshiba|treo|ts21i\-10|up\.browser|up\.link|uts|vertu|vodafone|wap|willcome|windows\ ce|windows\.ce|winwap|xda|zte)") {
         set $w3tc_ua _low;
      }
      if ($http_user_agent ~* "(acer\ s100|android|archos5|blackberry9500|blackberry9530|blackberry9550|blackberry\ 9800|cupcake|docomo\ ht\-03a|dream|htc\ hero|htc\ magic|htc_dream|htc_magic|incognito|ipad|iphone|ipod|kindle|lg\-gw620|liquid\ build|maemo|mot\-mb200|mot\-mb300|nexus\ one|opera\ mini|samsung\-s8000|series60.*webkit|series60/5\.0|sonyericssone10|sonyericssonu20|sonyericssonx10|t\-mobile\ mytouch\ 3g|t\-mobile\ opal|tattoo|webmate|webos)") {
         set $w3tc_ua _high;
      }
      set $w3tc_ref "";
      if ($http_cookie ~* "w3tc_referrer=.*(google\.com|yahoo\.com|bing\.com|ask\.com|msn\.com)") {
         set $w3tc_ref _search_engines;
      }
      set $w3tc_ssl "";
      if ($scheme = https) {
         set $w3tc_ssl _ssl;
      }
      set $w3tc_enc "";
      if ($http_accept_encoding ~ gzip) {
         set $w3tc_enc .gzip;
      }
      if (!-f "$document_root/wp-content/w3tc/pgcache/$request_uri/_index$w3tc_ua$w3tc_ref$w3tc_ssl.html$w3tc_enc") {
         set $w3tc_rewrite 0;
      }
      if ($w3tc_rewrite = 1) {
         rewrite .* "/wp-content/w3tc/pgcache/$request_uri/_index$w3tc_ua$w3tc_ref$w3tc_ssl.html$w3tc_enc" last;
      }
      # END W3TC Page Cache core
 

What it does well for me

  • Static page caching is really good.  I like the pre-zipped support, doing page caching is the main gain of every cache plugin, this is why I chose it since it creates static files which nginx serves well in the webserver realm.
  • Object + DB query caching using memcached.  Stats of memcached prove it's useful.
1
2
3
4
5
6
stats
STAT cmd_get 3202
STAT cmd_set 3034
STAT cmd_flush 543
STAT get_hits 3202
STAT get_misses 3155

These don't really tell us anything on cache efficiency as with a 5 minute limit on the caching of subsequent data, it all depends on the expiration of those items from the cache. What it does tell is is what would have gone to the backends in case it wasn't there at all. You will want to balance site dynamism vs caching efficiency.

What it doesn't do well for me

  • That Preview/Deploy thing, there is something about it that I find irritating.  It doesn't seem to notice that the deployed and the previewed config are the same so it keeps on displaying that you need to Deploy. Even after you have just deployed.  Aargh,  It's irritating, I don't really test-drive it either, I just apply and test the whole site.  I like the Deploy facility to apply all my changes at once though.  But I want to just activate and test the lot.
  • I have my issues with the Browser caching functionality, I create php applications for a living and I know for a fact that getting all those browsers to respect caching directives (Headers) is hard work.  I took a look under the hood using Web developer plugin to see what kind of expiration tricks they do and I saw some strange stuff going on that I switched this part off for now.  I haven't investigated this in depth yet.  Once I do I will make this more complete. I can just say that if you want the browser to cache the pages, you cannot send a no-cache header. that's the opposite of browser caching, more on it later I hope.

Conclusion of my experiences

nginx isn't hard to configure, with the aid and support of plugin developpers.  This plugin gets extra points for nginx support, translating these .htaccess rules/files from apache to nginx would be troublesome for most people, myself included, these are a lot of rules as we have seen here.  I list them up per module as they are written in the nginx config by this plugin, I didn't invent those.  These people understand that making it as low-entry as possible is good marketing.

As a PHP developer or even as a troubleshooter,  try to just use E_ALL for a change when running php code.  Switch every  error / trace / log warning in the php.ini file to on you can find,  you will be amazed on how much crap gets left behind.  I've noticed of small issues that are a nobrainer to fix in the code so I guess a lot of people turn their PHP warnings off .... even when creating nice wordpress plugin's.

Future of nginx

I've seen lots of people talking about speeding up their Apache driven wordpress blogs by using nginx in front of it. Although nginx does that well too, I believe it's so wrong that setup. Honestly. Just get rid of Apache right now. It's over, it earned it's place in history, that's where it belongs now. Replace it with nginx.

I'm going to predict that if nginx keeps up like it's doing now, it will replace Apache as a default web-server in some distributions just as mariaDB will be replacing mysql in the near future. It's inevitable. GPL is at work here. Better stuff just survives. Evolution works.

On the subject, make sure you check out the benchmarks mentioned before, they are unreal if you glance at the G-WAN ones.
It confirms a few things for me, there's always a new kid on the block, G-WAN is insane by the looks of it, I'm not sure I want to use it into production environments at this stage or if I can get it to work with php, in fact, it's new to me too.   This also confirms that Apache is on it's way out for sure.  More exciting things are about to come.

Some sources I visited to feed my hunger

http://wp-performance.com/2010/10/nginx-reverse-proxy-cache-wordpress-apache/
http://tentblogger.com/w3-total-cache/
http://wordpress.org/extend/plugins/w3-total-cache/
http://nbonvin.wordpress.com/

2 comments to W3 Total Cache plugin

  • E_ALL is one of the most important things to switch on during development. Actually I had the same issue with Drupal developers which caused issues on a production system. People who are only experienced in developing plugins or modules aren’t ‘real’ developers in my opinion.

    Always develop without leaving any issues during E_ALL.

    I remember during my perl development days that there are a lot of cgi scripts used in many sites which were very insecure. Because they didn’t :

    use warnings;
    use strict;

    The first one to switch on the warnings and the other one to make sure you declare variables.

  • void()

    Hi, thanks for your article. I’ve did some experiments with nginx / php-fpm and wordpress w3-cache before. the E_ALL idea is great. I have to look into it, did you experience any performance improvements because of solving this annoyances?

    Related to performance: Take a look at nginx fastcgi_cache module. You can use this to implement page-caching at the nginx level.

    A (probably somewhat screwed) ab test from localhost:

    # ab -c5 -n200 http:///
    This is ApacheBench, Version 2.0.40-dev apache-2.0
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Copyright 2006 The Apache Software Foundation, http://www.apache.org/

    Benchmarking (be patient)
    Completed 100 requests
    Finished 200 requests

    Server Software: nginx/0.7.67
    Server Hostname:
    Server Port: 80

    Document Path: /
    Document Length: 7439 bytes

    Concurrency Level: 5
    Time taken for tests: 0.385456 seconds
    Complete requests: 200
    Failed requests: 0
    Write errors: 0
    Total transferred: 1523800 bytes
    HTML transferred: 1487800 bytes
    Requests per second: 518.87 [#/sec] (mean)
    Time per request: 9.636 [ms] (mean)
    Time per request: 1.927 [ms] (mean, across all concurrent requests)
    Transfer rate: 3860.36 [Kbytes/sec] received

    Connection Times (ms)
    min mean[+/-sd] median max
    Connect: 0 0 0.0 0 0
    Processing: 0 9 56.4 0 378
    Waiting: 0 9 56.3 0 377
    Total: 0 9 56.4 0 378

    Percentage of the requests served within a certain time (ms)
    50% 0
    66% 0
    75% 0
    80% 0
    90% 0
    95% 1
    98% 352
    99% 366
    100% 378 (longest request)

    my nginx-config for wordpress:

    location ~ \.php$ {
    root html;
    set $do_not_cache “”;

    if ($http_cookie ~* “comment_author_|wordpress_(?!test_cookie)|wp-postpass_”) {
    set $do_not_cache “Y”;
    }

    set $my_cache_key “$scheme://$host$uri$query_string$is_args$args”;

    fastcgi_no_cache $do_not_cache;
    fastcgi_pass_header Set-Cookie;
    fastcgi_cache_use_stale error timeout invalid_header http_500;
    fastcgi_cache_key $my_cache_key;
    fastcgi_cache one;
    fastcgi_buffers 256 16k;
    fastcgi_buffer_size 32k;
    fastcgi_cache_valid 301 302 200 3h;
    fastcgi_read_timeout 300;
    fastcgi_pass unix:/tmp/php.sock;
    fastcgi_index index.php;
    fastcgi_ignore_headers Cache-Control Expires;
    fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
    include fastcgi_params;

    I used APC instead of memcached and gained a little speed improvement, also using unix-sockets instead of tcp/ip for speaking with php-fpm gained a little speed.

    However the biggest obstacle for me was the database queries from wordpress.

    enabling the query_cache and rising mysql-buffers helped somewhat but often and especially with plugins a lot of queries are generated.. to get ahold of them I found a great plugin:

    http://wordpress.org/extend/plugins/debug-queries/

    I still think my setup is somewhat slow, I guess I have to dig deeper into certain aspects of object caching as it seems somewhat complex and error-prone to me.

    greetings
    void()

recruitment
recruitment
recruitment
recruitment