Redis vs APCu 2018

Redis VS APCu Object Cache performance test 2018: is it relevant anyway? Redis is fast, but APCu is faster. Unfortunately, APCu is being pushed on the side of the road as an Object cache, and here is why.

There are 3 categories of caching systems on the server side: Memory code caching, Memory object caching, and Disk file caching. Objects can be anything though, including files. By file I mean any file: the generated HTML code that makes a page, a CSS, an image…

They are in the 3^rd layer on the map below:

More details about other Caching systems here.

APCu is an in-memory key-value store for PHP. Keys are of type string and values can be any PHP variables. APCu only supports userland caching of variables. APCu is APC stripped of OpCode caching.

Without even testing, without even starting reading blogs and Stackexchanges about it, I knew APCu would perform better. Indeed, they are both memory caches, but APCu should definitely be faster for two reasons:

1. APCu shares the same memory segments as PHP, for it’s an extension: apcu.so
2. Redis has more overhead because it’s external, so you access it either via TCP, http(s) or unix socket.

Either way, Redis cannot be faster than APCu at delivering content because of that.

WordPress Plugins Available

Welcome to the grey zone. You cannot test APCu or Redis object cache without a third party plugin, and as of today May 2018, there are only 3 plugins left that implement both APCu and Redis:

There are more plugins (I counted 4) that offer APCu cache but they are outdated and not maintained for one year or more.

Since their implementation of object caching will vary from developer to developer, what you are really testing is their ability to code properly. And since they are developing for free (W3 has a premium version though), you don’t know how good is their code until you read it. This is why you cannot compare different plugins implementing different cache systems, because the implementation would be different.

Therefore, to test Redis VS APCu, only 2 plugins can be used, for which I must load test 3 times: control (no cache), then Redis , then APCu, with a cache flush before each test. That’s a total of 6 load tests. Is it worth it? I say no, because of the results I got from my tests with Redis Object Cache. APCu may or may not be better than Redis but it’s not relevant, for either one cannot compete against a good old static full page cache.

What we are certain of, is that both these plugins are third party cache implementations. Therefore, they should cache the same number of objects since WordPress core and its plugins haven’t changed between the tests.

Finally, as I said in the conclusions of the Redis Object Cache tests, there is a high CPU overhead induced by this type of caching system (from PHP), which would overcome any benefits. I will test load Redis VS APCu anyway but I know this is useless for a single instance without lots of database access.

WordPress Object Cache Load Tests

Server Configuration

• Nginx, PHP-fpm, OS, APCu version (AWS EC2 t2.micro instance):

• OS (AWS EC2 t2.micro instance): Ubuntu 16.04.4 LTS 4.4.0-1050-aws x86_64
• vCPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (1 core only)
• File system: SSD disks (314MB/s read speed for small files)

System monitoring: Netdata 1.9.1, collects every 2 seconds 1,792 metrics.

These tests will be run by loader.io, with 500 clients/mn, for 1mn, in this order:

1. Cache disabled (control)
2. Redis Cache enabled
3. APCu cache enabled

That’s 500/60 = 8.3 new connections per seconds, for 60 seconds. I chose 500 connections/mn because my previous tests crashed with Object Cache at 1000 connections/mn, and I think 250/mn is not enough. Goal: having a server load between 50% and 80%.

For every tests,

1. fastCGI is disabled (BYPASS)
2. OPcache is enabled and not purged before test
3. Redis/APCu is purged before test, and only the Object Cache method is checked
4. CloudFlare is disabled

Powered Cache Load Test

Plugin Configuration

Okay… Powered Cache is not really serious:

1. First problem: you cannot chose which Redis database to use, it’s setup for #0 by default.
2. Second, Redis is set up for TCP access. Impossible to configure unix socket or anything else.

Redis VS APCu Load Test Results with loader.io

Control Test Results:

Average response time = 640ms, 16 clients/s.

Redis Test Results:

Average response time = 660ms, 16 clients/s. It’s worse than without any cache.

APCu Test Results:

Continuous increase in connections count until it crashed at 95 clients or so. Response time caped at 10s/page. Thus, loader.io decided to interrupt the test. Not good.

Redis VS APCu Netdata Analysis

It’s not obvious, but Redis brings a slight CPU overhead, and APCu implementation is completely wrong as the CPU reached 100% usage, interrupting the test with timeouts ratio > 1:

Without cache or With Redis, network traffic is quite the same, to account for PHP-MySQL database communication, then for PHP-Redis communication. Much less network traffic with APCu as expected since it’s integrated to PHP, and also because the test didn’t work well:

IPv4 packets exchanged are higher with Redis than without cache, proof that Redis induces more network operations/s because it caches a lot more things than the database requests needed to serve the pages:

Overall, Redis clearly induces more CPU usage, for itself and because PHP needs to communicate with it:

Redis operations: ~1200/s

Redis bandwidth: ~37MB/s

PHP-fpm pool responded well with and without Redis, having the same number of PHP requests processed (~8.5/s). It completely stopped responding during APCu test:

Missing data for PHP request duration as well, with a peak at 3.25 seconds/request:

Nginx performed identically with/without Redis, with ~12 active connections and ~9 requests/s. For APCu test, since PHP stopped responding, connections started piling up to 130 and these timeouts were detected by loader.io which interrupted the test:

SQL requests were increased by a factor 3 for the test without cache, with around 700 queries/s. Not the same number of requests for Redis and APCu, proof that both cache methods are developed differently and do not cache the same objects:

This translated to an increased cache hit ratio of ~570 queries/s, proof that MySQL is correctly configured for this system. Few hits for Redis, and even less for APCu:

And finally, here is the reason of the crash: too many threads opened to the SQL database during APCu test. The cap is 16:

That’s because of PHP. Since the PHP requests didn’t end, more and more connections were opened until the cap was reached.

This plugin clearly has not implemented APCu cache correctly.

OPcache status after test with op-gui.php:

[row]

[/row]

APCu cache stats with apc.php:

770,000 hits for APCu cache, and the test didn’t finished. That’s already more than 10 times the number of Redis operations: 1,200 * 60s = 72,000, give or take. Both object cache implementations are NOT the same, they do not cache the same number of objects. This is why PHP stopped responding, too many transactions per second for my small EC2 instance.

This plugin is rigged, at least for APCu implementation.

W3 Total Cache Load Test

Plugin Configuration

Good… W3 Total Cache seems to be more serious.

1. You can chose which Redis database to use.
2. Problem: You cannot configure Unix socket or anything else than TCP.

Interestingly enough, they still cover Xcache and eAccelerator, which are completely deprecated.

Redis VS APCu Load Test Results with loader.io

Control Test Results:

Average response time = 670ms, 16 clients/s. Roughly the same than with the other plugin. This is expected.

Redis Test Results:

Average response time = 640ms, 16 clients/s. Slightly better than the control.

APCu Test Results:

Average response time = 640ms, 16 clients/s. This is clearly better than Redis (by 100ms), as expected.

Redis VS APCu Netdata Analysis

Now, you can see what I’m talking about when I mentioned the difference in code implementation. This plugin clearly takes advantage of APCu and do the same caching job as with Redis. We can confirm the slight advantage for APCu over Redis with CPU usage: 80% (APCu) < 90% (Redis/no cache)

That’s because APCu is integrated in PHP as an extension and the difference in CPU usage is certainly due to the increased load in TCP networking for Redis as seen below:

Don’t be fooled by the fact that the same bandwidth is used during the first test without cache and with Redis, below is the proof that the 3 tests are different, with the number of IPv4 packets exchanged:

Overall, using Redis slightly increases CPU usage, for itself and for PHP that needs to communicate with it:

Redis operations: ~1200/s

Redis bandwidth: ~38MB/s

The PHP-fpm pool responded well, with the same number of requests processed: ~8.5/s

No problems of over duration for PHP requests:

Nginx performed identically in the 3 scenarios, with ~12 active connections and ~9 requests/s:

SQL requests were increased by a factor 3 for the test without cache, with around 700 queries/s. Same number of requests for Redis and APCu, proof that both cache methods do cache the same objects:

This translated to an increased cache hit ratio of ~570 queries/s (81% hit ratio), proof that MySQL is correctly configured for this system:

Roughly the same number of connections to MySQL database for all 3 tests:

OPcache status after test with op-gui.php:

[row]

[/row]

APCu cache stats with apc.php:

64,000 hits for APCu cache. That’s roughly the number of Redis operations: 1,200 * 60s = 72,000, give or take. Both object cache implementations are the same, they cache the same number of objects. Perfect score for W3 Total Cache.

Redis VS APCu Winner: It Depends

If we look only at CPU performance for Redis VS APCu, technically, the clear winner is indeed a local APCu server.

Just because one plugin didn’t implement APCu correctly doesn’t mean it’s more complex to develop. W3 Total Cache makes a perfect example of best practice in this matter. The load test proved that APCu is slightly faster at accessing cached objects than Redis, as expected.

Unfortunately, this plugin is the only one available to prove it. Therefore, having only one plugin available within a list of deprecated ones indicates a shift in PHP user focus: APCu is being slowly pushed down the side of the road, and therefore discarded as a standard object cache solution. That’s unfortunate but there is a good reason for that: it’s a local server cache. Its content cannot be replicated across instances or clusterized as Redis or Memcached.

Next best winner between Redis VS APCu, all things considered: Redis.

Redis is the current technology to install and maintain. It’s not meant to be a local cache server for static pages though. The slight CPU and network overhead induced for a classic blog overcomes the benefits, as shown in the load tests. It’s clearly a cache solution for heavy loaded, heavy connected application servers.