The importance of caching

Having a great bandwidth connection is really useful when you need to retrieve different pieces from the Internet, but having a local mirror can really improve your build times.

As an example, building and deploying a full TripleO stack (seed, undercloud and overcloud vms) without any local mirrors and proxy takes (in my server) about 73 minutes, while using a local pypy mirror, apt mirror and a squid proxy takes only 41 minutes. That is roughly a 40% improvement! (Although a lot of this time is spent in booting and configuring the OpenStack vm’s ad services.

In a smaller case, just to build the seed vm (no vm ever booted), the times are the following:

1.- No cache/proxy = 12m 31s
2.- Squid proxy = 10.54
3.- Pypi mirror = 9m 56 s
4.- Pypi mirror + Squid Proxy = 8m 31 s
5.- Pypi mirror + Squid Proxy + Apt mirror = 6m 57s

Obviously, the worst your bandwidth if, the most time you will save!

You can also use the pypi mirror for other tasks, like updating your venv of your project for testing. Let’s take for example, ironic . Keeping in mind that the test suite takes ~ 25s, the times are:

– Without pypi mirror # tox -r -epy27 = 7m 58s
– With pypi mirror # tox -r -e epy27 = 7m 08s

So, hurray for local mirrors and proxys!


http://apt-mirror.github.io/
https://github.com/openstack-infra/pypi-mirror
http://www.squid-cache.org/
https://pypi.python.org/pypi/bandersnatch

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s