Basic kickstart file for EZproxy instances

A discussion thread about EZproxy server sizing for VMs has been underway on the EZproxy mailing list this week, and some have asked for details on the setup that we run for our hosted proxy servers.

We use VMs with 512MB RAM, 1 processor, and 8GB of disk space, with log files stored on a network share.  This provides adequate resources for both EZproxy and Squid to run side-by-side:

             total       used       free     shared    buffers     cached

Mem:        502112     272564     229548         16       4500      78088

-/+ buffers/cache:     189976     312136

This is achieved by running a minimal installation with all unnecessary daemon processes disabled.

Here is the kickstart that we use for our proxy servers:

lang en_US.UTF-8
selinux --enforcing
keyboard us
authconfig --enableshadow --enablemkhomedir --enablecache --passalgo=sha512
timezone --utc America/New_York
firewall --enabled --ssh --port=53:tcp,53:udp,80:tcp,443:tcp,3128:tcp,3130:udp
rootpw --iscrypted <hashed password>
firstboot --disabled
services --disabled anacron,atd,autofs,avahi-daemon,bluetooth,cups,firstboot,gpm,hidd,mdmonitor,netfs,pcscd,readahead_early,rpc
gssd,rpcidmapd,yum-updatesd,microcode_ctl

text
skipx
reboot
install
bootloader --location=mbr --driveorder=sda
network --bootproto=static --device=eth0 --ipv6=auto --ip=<ipaddr> --netmask=255.255.255.0 --gateway <gwipaddr> --nameserver=<dns1ip>,<dns2ip> --hostname <proxy host name>
url --url=http://<install server>/centos/6.6/os/x86_64
repo --name=epel --baseurl=http://<install server>/epel/6/x86_64

zerombr yes
clearpart --all --drives=sda
part swap  --fstype=swap --ondisk sda --size=2048
part /boot --fstype=ext4 --ondisk sda --size=256
part /     --fstype=ext4 --ondisk sda --size=1 --grow 

# Packages
%packages --nobase
epel-release
yum
yum-utils
sudo
strace
telnet
tcpdump
rpcbind
nfs-utils
autofs
openssh-server
openssh-clients
puppet
ipa-client
squid
calamaris
awstats

 %post --interpreter /bin/sh --log /root/post_install.log
chvt 3
exec < /dev/tty3 > /dev/tty3
echo "Running %post script"
echo "Running puppet agent"
puppet agent --test --waitforcert 60 --logdest /root/puppet_install.log
echo "Removing 32-bit runtime"
# We do not need 32-bit compatability by default
yum -y erase glibc.i686
echo "Performing update"
# Update to latest
yum -y update
echo "Fixing plymouth"
# Turn off the pretty end-user boot screen, and show the useful boot messages
plymouth-set-default-theme details
/usr/libexec/plymouth/plymouth-update-initrd
exec < /dev/tty1 > /dev/tty1
%end

After the install, puppet re-installs only the base 32-bit runtime libraries needed for EZproxy, copies the EZproxy binary, configures it, and starts it up.  The only step that currently still need to be done manually is the EZproxy SSL setup, which certmonger may be able to help address.

EZproxy + Squid: Bolting on a caching layer (revisited)

In a previous writeup, I detailed early results using a caching layer with EZproxy.  Now that we have quite a bit of experience with the configuration, it's time to update with a long-term view of the results and some analysis of how effective it is overall.

To understand the benefits, first a discussion of the architecture in place is necessary.  We run in a clustered configuration using the HAPeer support built into EZproxy, with each proxy server running its own local Squid installation with a sibling configuration as a cache peer of its partner's Squid instance.


    Vendor                       Vendor
      /\                           /\
      ||                           ||
+============+               +============+
|   Squid    | <- sibling -> |   Squid    |
+------------+               +------------+
      /\                           /\
      ||                           ||
Proxy/ProxySSL                Proxy/ProxySSL
      ||                           ||
      ||                           ||
+------------+               +------------+
|  EZproxy   | <-  HAPeer -> |  EZproxy   |
+============+       /\      +============+
                     ||
                     ||

                   Patron

The patron accesses the proxy cluster by the HAPeer name, which is DNS mapped to each of the proxy servers.  EZproxy makes an internal decision to either service the request, or to send it to a peer machine to service the request, and issues a redirect to the patron's browser.  From that point forward,  the request uses the regular flow as a stand-alone proxy installation through the EZproxy server.

In our implementation, we put Squid on the local proxy machine with mostly default settings for RHEL/CentOS: no disk caching (i.e. no cache_dir setting) and the defaults for cache_mem (currently 256MB).  The defaults have worked well in our workloads, as our maximum memory usage is around 128MB after the cache is fully primed.

The biggest change needed was to enable the sibling support between the Squid instances:

digest_generation on
icp_port 3130
icp_access allow localnet
icp_access deny all
cache_peer ezproxy-01.example.edu sibling   3128  3130  proxy-only

This allows the proxy server to ask its peer:  "Hey, I don't have anything for this URL, do you?" and if the sibling has the content, it can request it from the partner cache, slightly increasing the overall effectiveness of the clustered proxy setup.  In larger setups, multicast support would make sense, but ours is small enough that it was not worth the extra configuration to get that working.

One tool that we use to measure the effectiveness of the cache setup is a reporting program called Calamaris.  This gives us insight into what kind of requests are being made, how many of those requests are serviced locally or from a sibling, and how many of them are pulled from vendor content directly.

In a recent sample, the report showed that almost 48% of the requests were found in the cache, while 52% of the requests had to go all the way to the vendor to be serviced.  The sibling setup was used for 3.5% of the requests, and were successfully served from the sibling's copy about 33% of the time.  I suspect that the more diverse the vendor resources, the higher this sibling number may go, though studies suggest that the number of successful sibling requests will probably never exceed 15% overall.

Looking at the report by network usage shows a slightly different story, where 30% of the traffic going through Squid was served locally (and at maximum network speed since in our configuration all objects are stored in RAM), while the remaining 70% of the network traffic was not found in cache and had to go all the way to the vendor.

What were some of the web assets that were successfully cached?

• CSS files (90% hit ratio)
• PNG images (90% hit ratio)
• JavaScript files (89% hit ratio)
• GIF images (87% hit ratio)
• ICO images (70% hit ratio)
• JPEG images (51% hit ratio)

I suspect the lower hit ratio of the JPEG files is due to the fact that those files are more likely to be licensed photography content rather than user interface elements on the web pages, where GIF and PNG files are more commonly used.  The net result of this is that the files that typically block web page layout and rendering are served at the highest speed possible, which makes for a better overall user experience.  This was validated by actual inquiries we received from our members asking "What changed? Everything feels faster now!" after this was rolled out.

What requests typically bypass the caches?

• JSON requests for AJAX calls (3% hit ratio)
• PDF files (3.5% hit ratio)
• Dynamic content (6.5% hit ratio)

Which falls into expectations.  The JSON requests are used for functions like autocomplete of search terms, search limiters on some platforms, pagination of search results, and analytics tracking.  The PDF files are mostly licensed content, and the "Dynamic content" category largely catches searches and search results screens.  

Thankfully, we have found that the developers at the vendors make correct use of cache-control headers to keep non-cacheable content from being served, and we have had ZERO instances reported of any issues that were cache related over the past 2 years that this configuration has been in production use.  I credit this largely to the practice of ISPs implementing transparent cache servers on their home user networks to manage bandwidth usage.

Prior to implementing caching support on our proxy servers, our network usage in:out ratio was very close to 1:1.  After implementing this architecture, it is not uncommon to see a 1:2 ratio where half of our proxy-to-vendor traffic has been eliminated thanks to a shared caching configuration.  If a version of EZproxy is released that supports compression as well, this ratio may go as high as 1:10 between the combination of compression and caching.

RHEL 6.4 SELinux user mapping

Normally, Red Hat does a decent job on their release notes about explaining what a new feature or change brings with each new RHEL minor release, but this time the release notes did not do the full scope of the change justice:

SSSD Fully Supported Features
A number of features introduced in Red Hat Enterprise Linux 6.3 are now fully supported in Red Hat Enterprise Linux 6.4. Specifically:

• support for central management of SSH keys,
• SELinux user mapping,
• and support for automount map caching.

The key line here was "SELinux user mapping".  This played in with IPA to generate some undesirable results, since the installation of IPA that was serving these systems did not have any SELinux user mapping defined, other than the default role.

Prior to the upgrade, mappings were not enforced, so the SELinux context was unconfined:

$ id -Z
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Post upgrade, the IPA default user was enforced.  This was not obvious at first, though.  The symptoms were that things that normally worked, didn't: sudo, su, dmesg, looking at log files.  Standard sysadmin diagnostics stuff suddenly stopped working completely.  

Trying to dig into this lead to very strange results, like not being able to run ls -l /bin/su directly and getting "?" for the su record when running ls -l /bin.  The same kind of strange behavior happened to /var, /root, and a few other key areas.  It behaved a lot like filesystem damage at first, but the fsck came back clean.  Logging in as root on the system console did work though, so that was my first clue.

Because this behavior was so bizarre, I tried a setenforce 0 to rule out SELinux, and all of a sudden things started working for non-root users logged in remotely.  I have been used to beating SELinux into submission a daemon context, but this was the first time I'd run into it as a user.

$id -Z
guest_u:guest_r:guest_t:s0-s0:c0.c1023

Up until now, there were no SELinux User Maps in IPA, so default mapping to the guest role was being used.  Thus, the first step was to define some non-guest roles to use.

There was not a lot of information about this on the IPA wiki, but I did find a decent writeup about configuring SELinux on Gentoo's site.  This gave me the key pieces that I needed to setup a map for the user_u:user_r:user_t and the staff_u:staff_r:staff_t mappings for mere mortals and sysadmins.

Amazing what happens when you have permissions to actually run things!

The next step required was to cleanly handle the staff_t to sysadm_t transition for sudo without requiring a separate newrole command to be run.  Thankfully, sudo is SELinux aware, so adding "role=sysadm_r" to the Sudo Rule in IPA that allows sysadmin commands took care of that part.

I go, you go, we all go for SPNEGO

While working through a web SSO Kerberos authentication issue (SPNEGO), I tried testing Safari and Chrome as well as Firefox to make sure that what I was running into was not a bug in Firefox.

The experience left a lot to be desired.

To be fair, I have been working with FreeIPA, so Firefox was already mostly configured for SPNEGO,  since it already had network.negotiate-auth.delegation-uris and network.negotiate-auth.trusted-uris set for my domain.  But that's about the only trick to getting Firefox to work with SPNEGO, and when I went to use it on another server in the same REALM, it appears to be properly sending the correct authentication negotiation headers to the server.

Safari has no such settings, since it relies on the Kerberos setup at the OS level. I have used it with other Apache's mod_auth_kerb module with other servers in the same REALM, so I know it basically just works.  For some reason though, the server was not sending back a 401 authentication challenge, so Safari just may not be supported by this application.  Que sera sera.

On to Chrome.

Oh my!  Chrome requires command line arguments to enable SPNEGO support.  There are no preferences in the UI that you can set.  There is no .plist or .ini or any other kind of file you can edit to cleanly enable it in a persistent manner.  You have to type in this abomination of a command line in a terminal window to run Chrome on a Mac with SPNEGO support:

open '/Applications/Google Chrome.app' --args --auth-server-whitelist="<server>" --auth-negotiate-delegate-whitelist="<server>" --auth-schemes="digest,ntlm,negotiate" https://<server>/

I don't object to using a terminal window; in fact I spend most of my time working in one.  But one would think that Google could come up with a more graceful way to handle that.  And that's not the only time I've had to resort to that for Chrome -- certain developer options require command line switches to enable as well, but I can forgive them -- a little -- in that case.

(This also implies that it will be a cold day in the Valley before Android tablets will have reasonable SPNEGO support.  You can't exactly pass command line options to browsers on tablets without jumping through hoops.  After I get the desktop browsers sorted out, I'll have to see just how bad the situation is on the tablet front.)

Moral of this story: out of the 3 major browsers for the Mac, Firefox seems to have the most widely supported and least troublesome Kerberos/SPNEGO support of them all.

Crontab and percent signs

It's funny how long you can work with a piece of software, and never run into certain features.  Even the most basic software is not immune to this.

I was recently setting up a cron job to do some processing on the previous day's log file:

/path/to/command --logfile=/path/to/logfile-$(date +'%Y%m%d' -d 'yesterday').log

For the non-UNIX literate readers, the $() construct says to run the command inside the parenthesis and use the output.  In this case, I wanted the date for yesterday formatted as YYYYMMDD.

Tested and worked just find from the command line, but when I created a cron job for it, I found this in my inbox the next day:

/bin/sh: -c: line 0: unexpected EOF while looking for matching ``'
/bin/sh: -c: line 1: syntax error: unexpected end of file

First thing I thought is that I had missed a "'" character somewhere, but I hadn't.  How odd.

What does cron have to say for itself?

Feb 21 02:00:01 server CROND[17834]: (root) CMD (/path/to/command --logfile=/path/to/logfile-$(date +')

 Hmm.  Truncated at the first "%" sign, now why would that happen?  Well, according to the manual page for crontab, the "%" character has special meaning:

Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.

I shudder to think how many years I've been using cron, and have managed to side-step this particular feature.  I guess I've always put date functions like that into scripts, and had cron call the script, so I never had to escape the "%" in the actual crontab before.

So now the cron command looks like this:

/path/to/command --logfile=/path/to/logfile-$(date +'\%Y\%m\%d' -d 'yesterday').log

and problem solved.  I had to chuckle to myself, though, because that feature has been around for at least 20 years, and somehow this is the first time I've run into it.

EZproxy + Squid: Bolting on a caching layer

In an earlier wish list post for native caching support in EZproxy, I stated that the user could easily save 10-20% of their requests to vendor databases if EZproxy natively supported web caching.

I was wrong.

The actual number is closer to double that estimate.

I recently setup a Squid cache confederation upstream from EZproxy, did some testing against Gale and ProQuest databases, and found that the real world number is between 30-40% savings by adding a caching layer.

This re-validates that studies done in the late 90's on HTTP caching appear to still hold true today:

Journal of the Brazilian Computer Society
Performance Analysis of WWW Cache Proxy HierarchiesPrint version ISSN 0104-6500
J. Braz. Comp. Soc. vol. 5 n. 2 Campinas Nov. 1998
http://dx.doi.org/10.1590/S0104-65001998000300003

A Performance Study of the Squid Proxy on HTTP/1.0Alex Rousskov / National Laboratory for Applied Network Research
Valery Soloviev / Inktomi Corporation

Enhancement and Validation of Squid’s Cache Replacement Policy John Dilley, Martin Arlitt, Stéphane Perret
Internet Systems and Applications Laboratory
HP Laboratories Palo Alto

It was very interesting that in my limited testing that my results were largely inline with those studies from over a decade ago:

• 30-40% cache hit rates with a Squid memory-only cache configuration
• 5-10% improvement in cache hit ratio by just adding one peer cache

This, despite all of the technology changes that have become commonplace thanks to Web 2.0 that did not exist back when these studies were originally made.

I opted to not configure disk-based storage for the cache for this test, but I may re-visit that at some point in the future, given that Rousskov and Soloviev were reporting nearly 70% hit ratios in their study.

Disk based storage for the cache deserves a look, but  my initial expectation is that in an academic library search setting, one is unlikely to achieve a greater than 40% hit ratio, simply due to the nature of the web sites being used.  Some things that are going to prevent a higher ratio include:

• Search term auto completion using AJAX calls
• The search results themselves
• Search filtering and refinement

In a general purpose library setting, a proxy may be able to achieve higher ratios as patrons go to the same sets of web sites for news, job postings, social networks, etc.  In an academic setting, though, with patrons executing individual searches, I am not convinced that achieving the higher cache hit ratios is a reasonable expectation.

The working set of cached objects between Gale and ProQuest was approximately 90MB, so it was well within the default 256MB memory cache size Squid uses by default.  With that workload, the only thing that a disk cache could be expected to do is to re-populate the in-memory cache copy when the server is restarted.  The cache will be quickly primed after only a few requests, though, so it's not the same as a busy cache that may have gigabytes of data stored on disk.

Another interesting behavior that I observed was that even though the working set could be fully held in either cache's memory, what I saw develop over time was one of the peer caches would hold a subset of objects until they expired, and then the other cache would pick up the baton, refresh the objects, and serve the newly refreshed objects to the cache cluster.  Wash, rinse, repeat, and you start seeing a pendulum pattern as the fresh content moves between the cache peers, with ICP requests fulfilling requests from the peer before doing the long haul to the origin server.

Even a 30-40% cache hit rate is nothing to downplay, though.  That is a significant bandwidth (and to a certain extent time) savings, and given that EZproxy does not support HTTP compression, this may be the best that can be hoped for in the short term.

EZproxy Wish List: HTTP Compression Support

While looking at ways to make our EZproxy servers more efficient, I re-discovered something that I already knew, but had been ignoring:

EZproxy strips out the Accept-Encoding header from requests, and requests uncompressed content from the upstream servers and sends uncompressed content to the downstream clients.

One might think that simply adding

HTTPHeader Accept-Encoding

to the proxy configuration would be enough to handle this, and it does fix part of the problem.  This allows the browser's Accept-Encoding header to be passed through to the upstream server, but it is not a complete solution (and can beak in certain corner cases):

Client => EZproxy

GET / HTTP/1.1
Host: www.example.com

Accept-Encoding: gzip,deflate,sdch

EZproxy => Server

GET / HTTP/1.1

Host: www.example.com

Accept-Encoding: gzip,deflate,sdch

Server => EZproxy

HTTP/1.1 200 OK

Content-Encoding: gzip

Content-Length: 6202

EZproxy => Client

HTTP/1.1 200 OK
Content-Encoding: none

When EZproxy receives the reply from the upstream server, it decompresses the content so that it can rewrite the content as necessary to keep users from breaking out of the proxy.  The missing step is that EZproxy does not then re-compress the content before sending it back to the user's browser.

Just how big of a deal is this?  Well, on just that one request, the uncompressed content was 26.5KiB vs. 6KiB, so the proxy transferred 4.4 times as much data from the server and to the client.  For fun, ask your IT department what they would do with ~75% more bandwidth...

So why not just add the HTTPHeader line globally, and at least benefit from the Server => EZproxy compression?  Well, some vendors have tried to be smart and dynamically compress or minify JavaScript on the fly, depending on the client browser's capabilities.  In the cited example, the minify handling was broken, and served out corrupted JavaScript files.

It is not a stretch to think that there may be other issues lurking out there when the server is told that the client can handle something that it will not be given.  Look closely at that Accept-Encoding line from Chrome.  Notice "sdch"?  Yeah, I had to look it up too:  Shared Dictionary Compression over HTTP.  There are a few posts that give an overview of what SDCH is about, but in short, it's a technique for sending a delta between a web page that you have and the web page that the server is getting ready to send.  Think of it like a diff function for HTTP content.

Now, what if the upstream Server supports SDCH and sends back a reply that EZproxy has no idea how to cope with properly?  You're going to get sporadic reports of problems, and it may take a while to narrow down that it's isolated to Chrome users, and maybe even longer to figure out it's SDCH at play.

That's just one example of how blindly passing through Accept-Encoding can go wrong, so I'm not opposed to EZproxy manipulating that header.  All of the mainstream browsers handle gzip encoding, and it's easy enough to support.

There is no good reason that I can think of that EZproxy could not simply filter the Accept-Encoding header to only contain gzip (and maybe even deflate), then decompress the server reply on the fly, apply any content changes to keep the users on the proxy, re-compress the content, and send it on to the client.  Once upon a time, someone might have piped up "CPU Cycles!", but I think the days that argument is pretty much dead these days thanks to Moore's Law.

With compression support, seeing a decrease in non-graphics content (HTML, JavaScript, CSS, JSON, XML, etc) of 80% is not an unreasonable expectation.  Add in caching support to handle the graphics, and EZproxy could be significantly more bandwidth friendly.