EZproxy + Squid: Bolting on a caching layer (revisited)

In a previous writeup, I detailed early results using a caching layer with EZproxy.  Now that we have quite a bit of experience with the configuration, it's time to update with a long-term view of the results and some analysis of how effective it is overall.

To understand the benefits, first a discussion of the architecture in place is necessary.  We run in a clustered configuration using the HAPeer support built into EZproxy, with each proxy server running its own local Squid installation with a sibling configuration as a cache peer of its partner's Squid instance.


    Vendor                       Vendor
      /\                           /\
      ||                           ||
+============+               +============+
|   Squid    | <- sibling -> |   Squid    |
+------------+               +------------+
      /\                           /\
      ||                           ||
Proxy/ProxySSL                Proxy/ProxySSL
      ||                           ||
      ||                           ||
+------------+               +------------+
|  EZproxy   | <-  HAPeer -> |  EZproxy   |
+============+       /\      +============+
                     ||
                     ||

                   Patron

The patron accesses the proxy cluster by the HAPeer name, which is DNS mapped to each of the proxy servers.  EZproxy makes an internal decision to either service the request, or to send it to a peer machine to service the request, and issues a redirect to the patron's browser.  From that point forward,  the request uses the regular flow as a stand-alone proxy installation through the EZproxy server.

In our implementation, we put Squid on the local proxy machine with mostly default settings for RHEL/CentOS: no disk caching (i.e. no cache_dir setting) and the defaults for cache_mem (currently 256MB).  The defaults have worked well in our workloads, as our maximum memory usage is around 128MB after the cache is fully primed.

The biggest change needed was to enable the sibling support between the Squid instances:

digest_generation on
icp_port 3130
icp_access allow localnet
icp_access deny all
cache_peer ezproxy-01.example.edu sibling   3128  3130  proxy-only

This allows the proxy server to ask its peer:  "Hey, I don't have anything for this URL, do you?" and if the sibling has the content, it can request it from the partner cache, slightly increasing the overall effectiveness of the clustered proxy setup.  In larger setups, multicast support would make sense, but ours is small enough that it was not worth the extra configuration to get that working.

One tool that we use to measure the effectiveness of the cache setup is a reporting program called Calamaris.  This gives us insight into what kind of requests are being made, how many of those requests are serviced locally or from a sibling, and how many of them are pulled from vendor content directly.

In a recent sample, the report showed that almost 48% of the requests were found in the cache, while 52% of the requests had to go all the way to the vendor to be serviced.  The sibling setup was used for 3.5% of the requests, and were successfully served from the sibling's copy about 33% of the time.  I suspect that the more diverse the vendor resources, the higher this sibling number may go, though studies suggest that the number of successful sibling requests will probably never exceed 15% overall.

Looking at the report by network usage shows a slightly different story, where 30% of the traffic going through Squid was served locally (and at maximum network speed since in our configuration all objects are stored in RAM), while the remaining 70% of the network traffic was not found in cache and had to go all the way to the vendor.

What were some of the web assets that were successfully cached?

• CSS files (90% hit ratio)
• PNG images (90% hit ratio)
• JavaScript files (89% hit ratio)
• GIF images (87% hit ratio)
• ICO images (70% hit ratio)
• JPEG images (51% hit ratio)

I suspect the lower hit ratio of the JPEG files is due to the fact that those files are more likely to be licensed photography content rather than user interface elements on the web pages, where GIF and PNG files are more commonly used.  The net result of this is that the files that typically block web page layout and rendering are served at the highest speed possible, which makes for a better overall user experience.  This was validated by actual inquiries we received from our members asking "What changed? Everything feels faster now!" after this was rolled out.

What requests typically bypass the caches?

• JSON requests for AJAX calls (3% hit ratio)
• PDF files (3.5% hit ratio)
• Dynamic content (6.5% hit ratio)

Which falls into expectations.  The JSON requests are used for functions like autocomplete of search terms, search limiters on some platforms, pagination of search results, and analytics tracking.  The PDF files are mostly licensed content, and the "Dynamic content" category largely catches searches and search results screens.  

Thankfully, we have found that the developers at the vendors make correct use of cache-control headers to keep non-cacheable content from being served, and we have had ZERO instances reported of any issues that were cache related over the past 2 years that this configuration has been in production use.  I credit this largely to the practice of ISPs implementing transparent cache servers on their home user networks to manage bandwidth usage.

Prior to implementing caching support on our proxy servers, our network usage in:out ratio was very close to 1:1.  After implementing this architecture, it is not uncommon to see a 1:2 ratio where half of our proxy-to-vendor traffic has been eliminated thanks to a shared caching configuration.  If a version of EZproxy is released that supports compression as well, this ratio may go as high as 1:10 between the combination of compression and caching.

EZproxy + Squid: Bolting on a caching layer

In an earlier wish list post for native caching support in EZproxy, I stated that the user could easily save 10-20% of their requests to vendor databases if EZproxy natively supported web caching.

I was wrong.

The actual number is closer to double that estimate.

I recently setup a Squid cache confederation upstream from EZproxy, did some testing against Gale and ProQuest databases, and found that the real world number is between 30-40% savings by adding a caching layer.

This re-validates that studies done in the late 90's on HTTP caching appear to still hold true today:

Journal of the Brazilian Computer Society
Performance Analysis of WWW Cache Proxy HierarchiesPrint version ISSN 0104-6500
J. Braz. Comp. Soc. vol. 5 n. 2 Campinas Nov. 1998
http://dx.doi.org/10.1590/S0104-65001998000300003

A Performance Study of the Squid Proxy on HTTP/1.0Alex Rousskov / National Laboratory for Applied Network Research
Valery Soloviev / Inktomi Corporation

Enhancement and Validation of Squid’s Cache Replacement Policy John Dilley, Martin Arlitt, Stéphane Perret
Internet Systems and Applications Laboratory
HP Laboratories Palo Alto

It was very interesting that in my limited testing that my results were largely inline with those studies from over a decade ago:

• 30-40% cache hit rates with a Squid memory-only cache configuration
• 5-10% improvement in cache hit ratio by just adding one peer cache

This, despite all of the technology changes that have become commonplace thanks to Web 2.0 that did not exist back when these studies were originally made.

I opted to not configure disk-based storage for the cache for this test, but I may re-visit that at some point in the future, given that Rousskov and Soloviev were reporting nearly 70% hit ratios in their study.

Disk based storage for the cache deserves a look, but  my initial expectation is that in an academic library search setting, one is unlikely to achieve a greater than 40% hit ratio, simply due to the nature of the web sites being used.  Some things that are going to prevent a higher ratio include:

• Search term auto completion using AJAX calls
• The search results themselves
• Search filtering and refinement

In a general purpose library setting, a proxy may be able to achieve higher ratios as patrons go to the same sets of web sites for news, job postings, social networks, etc.  In an academic setting, though, with patrons executing individual searches, I am not convinced that achieving the higher cache hit ratios is a reasonable expectation.

The working set of cached objects between Gale and ProQuest was approximately 90MB, so it was well within the default 256MB memory cache size Squid uses by default.  With that workload, the only thing that a disk cache could be expected to do is to re-populate the in-memory cache copy when the server is restarted.  The cache will be quickly primed after only a few requests, though, so it's not the same as a busy cache that may have gigabytes of data stored on disk.

Another interesting behavior that I observed was that even though the working set could be fully held in either cache's memory, what I saw develop over time was one of the peer caches would hold a subset of objects until they expired, and then the other cache would pick up the baton, refresh the objects, and serve the newly refreshed objects to the cache cluster.  Wash, rinse, repeat, and you start seeing a pendulum pattern as the fresh content moves between the cache peers, with ICP requests fulfilling requests from the peer before doing the long haul to the origin server.

Even a 30-40% cache hit rate is nothing to downplay, though.  That is a significant bandwidth (and to a certain extent time) savings, and given that EZproxy does not support HTTP compression, this may be the best that can be hoped for in the short term.

EZproxy wish list: caching

This is the second in a series of thoughts on how EZproxy could be made better.  For a background, please read my original post.

For all of EZproxy's features, one glaring omission is an option to enable a caching feature.

For smaller sites, this probably would not mean much.  Their patrons may be on-site at the library anyway, or geographically close to the library, so they are probably using the same ISP as the library.  In this case, how much benefit that a caching solution might offer is debatable.

But for larger sites, and sites that support a geographically diverse user population, caching can be a significant performance enhancement.

Here's a very simple example of why:

A user sitting in California needs to access a resource in Denver from a proxy in Virginia.

In this scenario, the physical path that the packets are likely to take through wires in the ground very nearly matches the mental line that you might draw between these three locations.  You'll have to very nearly cross the continent twice to get the request from the desktop to the server, and then turn around and do it again for the reply from the server to the desktop.

So what's the big deal?

One big problem: Users are impatient.  Each trip across the continent is going to cost precious milliseconds and give the perception (warranted or not) of slowness.  Waiting for every single search, every search limiter, every single page render, every single article retrieval is going to make users irritated.  The more they wait, they more frustrated they become, especially if they are new to research and do not have much patience to start with.

Another problem: Someone pays for the bandwidth.  Somewhere in your IT department's budget is a line item or three for your internet access.  The more capacity (bandwidth) you have to access the internet, the more it costs.  Web caching has been used for almost two decades now to address this.  Businesses use web caches for performance, cost savings, and security; countries use them for performance and in some cases censorship; chances are the ISP you are using right now uses a transparent web cache to save on their own bandwidth costs.

If EZproxy were to support a web caching feature the user in California would only need to contact the proxy server in Virginia for 80-90% of their web requests.   Vendors re-use many of the same CSS stylesheets, graphics, and JavaScript files on their web pages, so the only content that is unique are the search results and the article retrievals.

Depending on the vendor, even the articles might be cacheable, so if you were teaching a lab, only the first student to retrieve the article would have to wait for the full round trip, while the second and subsequent students may get the article from the cache.

If you know that you have a caching server on your campus, or at your ISP, you could setup EZproxy to participate in a web cache hierarchy.  Depending on your network, you might be able to piggy-back on other user's activities and your cache would not have to work as hard.  You can generally setup hierarchy caches such that your cache will not store that content locally, since retrieving from that upstream cache is not considered as expensive.  This frees up your cache to focus on the content that is not shared with the upstream server.

If you have multiple EZproxy servers, you could setup a cache confederation when this is combined with a clustered proxy setup, thus amplifying the benefits of having a cluster.  Instead of each cache keeping duplicate content, you could have each proxy query its peer before requesting content upstream.  This would require EZproxy to support the ICP protocol, which might make for interesting architecture possibilities as well.

Even without the advanced caching support, though, just a basic web cache implementation would be useful to many sites, both directly in performance improvements, and indirectly via bandwidth savings.