Varnish: the secret weapon for boosting page speed

Before We Begin

Request processing (especially dynamic) with web servers such as Apache can be a heavy strain on system resources.

That’s why more and more people started using lightweight web servers like nginx or lighttpd; they can work as a frontend for static content paired with Apache or independently. But there is another way to improve your site: using a web server accelerator such as Varnish Cache – a solution from Varnish Software.

There are a few ways to learn all the specifics of Varnish Cache processing: reading the Varnish Book, the Official documentation and a lot of practice and experimentation. We highly recommend starting with the first two. This article is based on all three, materials found at stackoverflow and other professional sites, and personal experience.

What is Varnish?

VARNISH is an open source caching reverse HTTP proxy, sometimes referred to as a web application accelerator. A reverse proxy is a proxy server that appears to clients as an ordinary server. It stores (caches) files or fragments of files in memory that are used to reduce the response time and network bandwidth consumption.

Varnish is available under a two-clause BSD licence. Commercial support is available from Varnish Software. Version 1.0 of Varnish was released in 2006, Varnish 2.0 in 2008, Varnish 3.0 in 2011 and Varnish 4.0 in 2014.

Varnish is flexible because you can configure it by writing your own caching policies in its Varnish Configuration Language (VCL). VCL is a domain specific language based on C. VCL is then translated to C code and compiled.

How does Varnish work?

Let’s imagine that the client sends a request to your server. Varnish receives it and tries to find an appropriate response in the cache storage. If it finds one and its content is still valid, then we have a response we can deliver to the client. Otherwise Varnish will forward the request to the backend, fetch the response, store it in the cache and send the response to the client. Next time the same request is made, Varnish will have the response in cache storage.

It’s easy to represent this with a simple diagram:

Installation and basic configuration of Varnish

You can easily find Varnish binary packages or compile it for yourself from source code. Use the official site for grabbing a current version. Installation can be done with the system package manager, but please note that this might not be the latest version of Varnish. For example under Debian 8 (“jessie”) do the following as root:

[code]

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.1" >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

[/code]

If compiling from source, make sure that you already have dependencies installed. Please check the official documentation for a list.

The next step is configuration. First, let’s look at the configuration file
/etc/default/varnish (Debian)

/etc/sysconfig/varnish (Red Hat)

This holds a lot of useful and important settings for the Varnish daemon, such as locked shared memory, default TTL, numbers of threads, etc.

Here is a simple example:

[code]

NFILES=131072
MEMLOCK=82000
NPROCS="unlimited"
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=6081
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=10
VARNISH_MAX_THREADS=50
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_SIZE=128M
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -p esi_syntax=0x2 \
             -p cli_buffer=16384 \
             -p http_resp_hdr_len=131072 \
             -p thread_pools=4 \
             -p thread_pool_add_delay=2 \
             -p thread_pool_min=15 \
             -p thread_pool_max=50 \
             -p thread_pool_timeout=3600 \
             -u varnish -g varnish \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"

[/code]

In VARNISH_VCL_CONF we define a path to a second file, which describes request handling and the caching process in the Varnish Configuration Language (VCL). VCL is translated to C and compiled into binary code which is then executed when requests arrive. This feature makes Varnish a powerful HTTP processor not only for caching, since VCL has inherited a lot from C and it reads much like a simple C or Perl script. We will look on it closer in next chapter.

A look inside

Let’s look inside request processing in Varnish v.4!

To start, here’s a simplified diagram of the Varnish Finite State Machine and a brief description of each of its processes:

VCL_RECV

The start of all processes, called just after the request has been received and parsed. It determines the next steps for incoming requests – will they be modified, cached or not, should anything be forwarded for processing by the backend or simply return a response from cache, etc.

VCL_HASH

Called after vcl_recv to create a hash value for the request that will be used as a lookup key for the object in Varnish. Passes control to the functions, depended on vcl_recv result.

VCL_HIT

Called when a cache lookup is successful. Its result determines how to handle the rest of the request.

VCL_MISS

Called after a cache lookup if the requested document was not found in cache or if vcl_hit returned fetch.

VCL_PASS

Called upon entering pass mode. In this mode, the request is passed on to the backend, and the backend’s response is passed on to the client, but is not entered into the cache.

VCL_PIPE

Called upon entering pipe mode. In this mode, the request is passed on to the backend, and any further data from both the client and backend is passed on unaltered until either end closes the connection.

Backend servers

In the descriptions above, we referred to the backend several times. In Varnish terminology “backend” refers to any server(s) that provide cacheable content for Varnish, the one(s) that Varnish accelerates.

To tell Varnish where it can find a backend, define it in the VCL file like this:

[code]

backend default {
        .host = "127.0.0.1";
        .port = "8080";
}

[/code]

In real life we could encounter server crashes or other problems in the backend. To ensure continuity of work we will define a set of parameters called “probe” for checking the backend’s state.

[code]

backend default{
        .host = "127.0.0.1";
        .port = "8000";
        .probe = {
                .url = "/alive/";
                .timeout = 2s;
                .interval = 3s;
                .window = 10;
                .threshold = 9;
        }
}

[/code]

Now Varnish will send a simple GET request to “/alive/” url every 3 seconds, with a timeout limit of 2 seconds. Since we set “window” to 10 and “threshold” to 9, the backend will be considered healthy if at least 9 of the last 10 requests succeeded. Varnish will not send traffic to hosts that are marked as unhealthy.

Great, now when Varnish needs content it will know where to get it by using a connection to a default backend. But what if you have multiple backend servers or maybe you have some special policy for some requests? We could define a new backend called ‘’magic’’ and add a new rule for processing requests:

[code]

backend magic {
        .host = "127.0.0.1";
        .port = "8000";
}
...
sub vcl_recv {
        if (req.url ~ "^/magic/") {
                set req.backend_hint = magic;
        } else {
                set req.backend_hint = default;
        }
}

[/code]

You are free to define your own method of specifying a backend for a particular request, based on rules, parameters or randomly. You can also group several backend servers together into a “director”. This requires you to load a VMOD, a Varnish module, and then call certain actions in vcl_init.

[code]

import directors;    # load the directors
 
backend default {
        .host = "127.0.0.1";
        .port = "8080";
}
backend magic {
        .host = "127.0.0.1";
        .port = "8000";
}
 
sub vcl_init {
        new foobar = directors. round_robin();
        foobar.add_backend(default);
        foobar.add_backend(magic);
}
 
sub vcl_recv {
        set req.backend_hint = foobar.backend();
}

[/code]

This is a good way to use Varnish for load balancing. Here we are using a round-robin director, but there is also a random director which distributes requests randomly.

Let’s talk about Edge Side Includes (ESI)

Sometimes we need to display unique information to each user, like a user’s name, contents of their cart or a list of past orders etc. Caching such data can be a huge problem, that’s why one of the best things implemented in Varnish are ESI blocks.

Using ESI we can break our page up into smaller pieces and decide which of them should be cached or not, then assembling them into a response. These fragments can have individual cache policies, for example if you have a list of the 5 most popular products in your online store, this list can probably be cached for a time and included in other pages, while other fragments might never be cached and must always be loaded dynamically.

With this strategy you can increase your hit rate and reduce the load on your servers.

ESI only implements a small subset of the full Varnish functionality, but it is enough. You can use three ESI statements:

1. esi:include
2. esi:remove
3. <!–esi …–>

Varnish will not process ESI instructions in HTML comments.

Let’s look at some examples.

ESI:include

We will make a simple example. This is user.php file:

<?php
…
// Including some libraries
..
$user = $this->getUser();
echo $user->getName();
?>

Now some HTML file that has an ESI include statement. This is test.html:

<HTML>
    <BODY>
        Hello <esi:include src="user.php"/>!
    </BODY>
</HTML>

And the last step before ESI will work is activation ESI processing in VCL.

[code]

sub vcl_backend_response {
        if (bereq.url == "/test.html") {
               set beresp.do_esi = true;
               set beresp.ttl = 24 h;   
        } elseif (bereq.url == "/user.php") {
               set beresp.ttl = 1m;
        }
}

[/code]

ESI:remove and <!–esi … –>

But what should we do if ESI is not available? For that there are the <esi:remove> and <!–esi … –> constructs. You can use it to present content whether or not ESI is available, for example you can include content when ESI is available or link to it when it is not.

The ESI processors will remove the starting “<!–esi” and the ending “-→” tags when the page is processed, while still processing the contents. If the page is not processed, it will remain intact, becoming a HTML/XML comment tag. ESI processors will remove <esi:remove> tags and all content contained in them, allowing you to only render the content when the page is not being ESI-processed. For example:

<esi:remove>
  <a href="http://www.example.com/LICENSE">The license</a>
</esi:remove>
<!--esi
<p>The full text of the license:</p>
<esi:include src="http://example.com/LICENSE" />
-->

Who uses Varnish Cache

Based on information from Wikipedia, Varnish is used in around a tenth of the TOP 10K sites – online newspapers, social media and media content sites; here’s short a list of the most famous: Wikipedia, Facebook, Twitter, Vimeo, The New York Times, The Guardian, The Hindu, and Tumblr.

Use Varnish in Magento

Magento is the top ecommerce platform in the world. It’s known for being flexible, but not so much for being fast. If you want to use Varnish with Magento 1, you should find the ready solution for it or if you have a lot of time: Do-It-Yourself. You just need to learn how to use Magento, understand HTTP Headers and get a hold of reverse proxy caching and Varnish Cache specifically.

In case of Magento 2 – everything is not so bad. Magento 2 officially supports Varnish versions 3.0.5 or later or any Varnish 4.x version. Just follow the tutorial in the documentation, but we’ll summarize it for the sake of completeness:

• Install Varnish and test it by accessing any Magento page to see if you’re getting an HTTP response header that indicates Varnish is working.
• Install the Magento software and use the Magento Admin to create a Varnish configuration file.
• Replace your existing Varnish configuration file with the one generated by the Admin.
• Test everything again and if there is nothing in your /var/page_cache directory, you’ve successfully configured Varnish with Magento!

Finally, remember that your (or your customers’) ecommerce site is the last possible place on the web where you should be testing random VCL snippets and unknown PHP code.