As you know, Varnish is a powerful caching system and it is a must-have on every Magento setup, even at the development workstation. But there is a little trick that very few people know about. It is the Transient cache, which I explain below.
The master process of the Varnish controls the execution of the child process, which contains the entire cache. The operating system running the Varnish has an oom-killer that makes sure the system has enough memory to run and calculates which processes can be killed. When a child process of a Varnish grows in size, it becomes the first candidate for termination. After the child process terminates, the Varnish master process spawns a new child process and the accumulated cache is completely lost.
At the same time, the memory consumption graph shows us:
In this case, the following messages can be seen in the host’s log:
kernel: Memory cgroup out of memory: Killed process 21363 (varnishd) total-vm:5264564kB, anon-rss:5099268kB, file-rss:0kB, shmem-rss:0kB, UID:1000720000 pgtables:10344kB oom_score_adj:682
kernel: oom_reaper: reaped process 21363 (varnishd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
To understand whether the child process rebooted or not, just look at varnishstat, the very top lines will show the lifetime of the master and child processes:
If the lifetime at the top is the same, then the child process was not killed. If different, then he was previously killed.
The Transient cache is a special type of Varnish storage for objects with very low ttl. By default it is 10 seconds. This ttl is controlled by “shortlived” parameter:
This is a special storage for short-lived objects, which is always active, on any installation, even if you did not specifically configure it. This is where an unexpected surprise can await us: this storage has no size limit by default! That is, you will simply use up all the memory over time and notice something only when the varnish loses the cache.
To avoid an uncontrolled increase in memory consumption, you need to set a memory limit for the Varnish. Usually, only one limit is set, but as mentioned above, there is one more additional storage (transient), so you need to set two values.
Each cache store in a Varnish can be named. This is convenient if you want to put different objects in different storages, for example, some in RAM, and others on disk. It is easy to set a limit for the transient storage, it is enough just to specify storage named Transient at startup. Another storage, where all other objects will be stored, can be called Cache, or by any other name.
This is what the parameters for launching a Varnish with the specified value of memory limits look like:
varnishd -s Cache=malloc,${CACHE_MEMORY} -s Transient=malloc,${TRANSIENT_MEMORY}
Let’s say you have 8 gigabytes of memory on the instance, how much memory should we allocate for each storage? First, we agree that we cannot use all 8 gigabytes of memory, it is recommended to use 75% of the host’s available memory. That is, we have 6 gigabytes left. This memory can be divided 80 / 20: 4.8 gigabytes for the main storage, and 1.2 gigabytes for the Transient storage.
Here are the values of the memory parameters:
CACHE_MEMORY = 4800M
TRANSIENT_MEMORY = 1200M
To see and track how memory is actually being used, you can look at varnishstat again:
Once you’ve set a Transient cache limit, you may notice that Varnish no longer suddenly loses cache.