Magento performance optimisation: APC, Memcached and Varnish

We’re always trying to improve the performance of the Magento Community stores that we look after.

Once we’re done streamlining the application itself and making sure that we’ve done everything we can, we look at the platform itself. We do it this way to make sure that we don’t miss anything out. Server extensions such as Varnish make quick sites use less resources, not neccessarily slow sites quick – although that can also be the case!

All the software mentioned in this post can be set up to work concurrently on a Magento installation. There are a lot of different configuration options that can work, and some that will no doubt work better depending on your configuration. We’re used to Red Hat Linux servers running standard LAMP stacks. We’ve played with NGinx and PHP-FPM (both with Apache and NGinx) but that’s for a later post.

Using APC as an OPCode Cache module

Starting with the PHP 5.5 branch, Zend Optimizer+ is directly built in, so there will be no need for this step moving forward. In the meantime, if you’re running an earlier version, you’ll want to install an OPCode Caching module. Our favourite is Alternative PHP Cache – APC.

PHP is an interpreted language. This means that each time a PHP snippet is requested, it reads all of the files and “compiles” them into something that the server can execute – OpCode. OpCode caching caches this generated OpCode, leading to decreased timings when repeat requests are made.

Magento often has hundreds of included files in a single request meaning some real savings can be made.

OpCaching with APC is system-wide. There is no need to set up Magento to use it – PHP will use it by default.

If you want to know more about OpCaching, I’d recommend reading through this https://support.cloud.engineyard.com/entries/26902267-PHP-Performance-I-Everything-You-Need-to-Know-About-OpCode-Caches. It has some interesting information and a couple of really good diagrams to explain how PHP caches the OpCode and how it uses it.

Installing APC

Depending on your platform, you should be able to install APC using your package manager.

yum install pecl-php-apc or through PECL pecl install apc

You can look at the apc.ini to fine-tune your settings then. The only one you have to make sure of, is actually enabling it!

It’s worth looking at a value between 64 and 256Mb for apc.shm_size configuration value (depending on your server). It’s also worth increasing apc.num_files_hint to 15000 or maybe even more. Magento contains upward of 20k PHP/PHTML files. APC will cache only 1000 files by default.

Remember to restart Apache/HTTPd or NGINX when you’re done so that it loads the module.

service httpd restart

service nginx restart

Memcached

“Memcached is a high-performance, distributed memory object caching system, generic in nature, but originally intended for use in speeding up dynamic web applications by alleviating database load.” – http://memcached.org/about. Simply put, Memcached alleviates Magento’s heavy hits on your database by caching data away from it.

You’ll need to install Memcache (a PHP extension), and install Memcached – the Memcache Daemon on your server. Across a load balanced / multi server solution, Memcached only has to reside on one server, and the others can all use it. This provides a centralised source for cache management.

yum install pecl-php-memcache pecl-php-memcached or through PECL pecl install memcached memcache

As per APC, remember to restart Apache/HTTPd when you’re done so that it loads the module.

Memcache integration with Magento is much closer – remember, with APC, we’re closer linked to PHP. So, we have to make Magento aware of Memcached. How do we do this?

Magento offers a few examples on their site of how to configure this, but no definitive by-the-book example really. The key to this is a change to your local.xml file. Within your local.xml file, you should notice the following block:

Within this global section, we need to add the following block.

Memcached can run on any number of ports, 11211 is the default. If you’re using a shared cache storage, change the file_name_prefix per store to something more relevant, or run memcached instances on seperate ports per store; or both. If you’ve got a lot of front-end servers, dedicate one of them to be your memcached server and make all the others connect to it. That way, they’ll all store their data in one memcached instance. This makes the user bouncing between a number of servers a lot smoother.

It’s worth pointing out that Memcache only supports one level of key-value relationships. This means that it can’t actually store Magento cache tags. This means that you should either specify a slow back-end to maintain the cache content tag relationship, or, alternatively, don’t define one – purposefully break this.

Redis WILL allow cache tags, and can be used as a direct replacement for Memcached.

Varnish

Varnish is a caching reverse proxy. It’s actually also a http accelerator, it can act as a load balancer and a fail-over system. As a basic concept, it will cache the generated HTML from a page, and serve that to a user rather than making a call to the Magento Framework. This leads to some pretty speedy load times. Magento is very slow to initialise, and if you want your first byte speed down to anything under one second, let alone under half a second – then full page caching is the only way to achieve this.

As mentioned at the beginning of this article, your store should already be fast and you should be looking to reduce server resource – however, Varnish is a great way to reduce your speed to < 300ms. Varnish requires at least a single page view in order to prime the cache, so it means that your un-cached performance still needs to be high. There are cache-warmers out there (crawlers that will run over your site and prime the cache ready for customer visits) – but we’ve not found any we’d recommend thus far.

Usually, Magento stores fall into two categories, and these will give you an idea of how to configure Varnish.

If your site has relatively low traffic, then you’re going to have a bigger trade-off. Either your TTLs need to be high to ensure a greater hit-rate, or you get up-to-date content. It’s a fine line between them – obviously you can’t have aggressive caching, but frequent changes to content. They don’t play well together. You could, in theory run some sort of cache primer, but I’d expect that it wouldn’t be worth it due to having potentially thousands of URLs.

If you’ve got a busy site, then you should be able to get away with low TTLs, and still see a good hit-rate from Varnish. This also means that you can update content more frequently.

So, let’s install Varnish.

yum install varnish

By default, if installed successfully, once started, varnish will listen on port 6081.

netstat -taupen | grep :6081

Next, you’ll need to configure /etc/varnish/default.vcl

There are a couple of Varnish plugins for Magento, and the one we like is: http://www.magentocommerce.com/magento-connect/pagecache-powered-by-varnish.html – by default, this comes with a VCL that can be used alongside Magento.

Download and install the plugin, then grab the provided VCL. Backup your default.vcl and copy this one over in it’s place.

You’ll need to move your back-end onto the port that you’re going to ask Apache to listen on. So, you’d usually expect that:

OUTSIDE WORLD –> APACHE (RUNNING ON PORT 80)

But you’re then going to create:

OUTSIDE WORLD –> VARNISH (RUNNING ON PORT 80) –> APACHE (RUNNING ON PORT 8080)

Thus, anything that Varnish can’t serve, Apache will.

So, modify the port n the back-end default and admin sections to listen on port 8080


backend default {
.host = "127.0.0.1";
.port = "8080";
}

backend admin {
.host = “127.0.0.1”;
.port = “8080”;
.first_byte_timeout = 18000s;
.between_bytes_timeout = 18000s;
}

Now change your httpd.conf to listen on port 8080 instead of 80, and change your NameVirtualHost definition to port 8080 too.


NameVirtualHost *:8080
Listen 8080

You may also have to change your virtual host definition for your domain if it includes the port number.

Having set these up, cycle apache and varnish.

service httpd restart && service varnish restart

You should now be running through Varnish. You can test this by hitting the front-end of your site and taking a look at your headers. You should also be able to run varnishstat from a command line and see traffic.

Now that it’s set up, enable the module in Magento and it will start hitting Varnish.

Any questions on the above, feel free to drop me a comment below.

By Douglas Radburn. at 3:39PM on Friday, 13 Dec 2013

Doug is our Senior Open Source Web Developer since bringing his knowledge and skills to Branded3 in 2009. A founding developer of our Twitition and Competwition platforms, Doug has also been lead in our Open Source Projects on Magento ecommerce solutions and Wordpress CMS platforms. Follow Douglas Radburn on Twitter.

comments

  • http://www.paginasweb.pe Diseño Paginas Web

    good post, APC its been used for a long time with excelent results

  • Ammar

    Great post. I wasn’t clear on whether you were recommending either Memcache or Varnish or proposing both. What would the difference be?

    • Douglas Radburn

      Hi Ammar, I’d recommend both. Varnish will take the main hit, then when a request needs Apache, Memcached should take the hit and reduce DB load.

      • Rostol

        Thanks for this post Douglas, you covered all the basics in one. If i understood correctly you recommend all three, Varnish serving HTTP pages, made with cached opcodes from APC and cached DB objects from memcached. or is APC no longer needed with varnish ? (i know its supposed to be on its way out, and i am using 5.5.3-1ubuntu2.1 )

        • Douglas Radburn

          Hi Rostol – Exactly. I’d always suggest throwing APC into the mix, however, if you’re using PHP 5.5, you’ll have Zend’s code optimiser built in now, so there’s probably little point in having APC installed.

  • http://magento.ocodewire.com/ shanaya sharma

    Knowledgeable post!!

  • http://www.northcutt.com/ Ben Ustick

    Nice post. It’s good to see the focus on Varnish. Haven you ever used the Turpentine extension from Nexcess? It provides Varnish configuration files to modify Magento’s behavior to improve the cache hit
    rate. The stats from the their whitepaper at the Magento Conference were pretty impressive.

    • Douglas Radburn

      I’m aware of it, but we’ve not used it in any production environments yet. WIll have to check out the whitepaper – do you have a link?

  • Daniel Passov

    Douglas, we are having issues with Varnish not caching our site. Do you have an email address that I can contact you to maybe help us out. Thanks

    • Douglas Radburn

      Hi Daniel, sure thing, douglas [dot] radburn [at] branded3.com

  • https://www.surface604.com/ Surface 604

    Good post, Douglas! However, how to deal with the Varnish and SSL? For example I want to have an SSL across all my Magento store. How should I do this if Varnish does not support SSL?

    Please help.

    • Douglas Radburn

      Hi! I believe you emailed me recently, and I only replied this week. For anyone else wondering, The only experience we’ve had unfortunately is behind a load balancer where the SSL terminates (meaning that the servers actually respond on port 80)