Alternatives to layer-based image distribution: using CERN filesystem for images

The container land has been heavily influenced by Docker. After all, the huge ecosystem building Docker images and tools is hard to ignore. Docker, defines images as lists of layers. This helps container engines cache reused layers and avoid re-downloading the whole root filesystem when minor changes get pushed. The, fairly new, OCI image spec is adopting the same approach in standardising the image format. However, layers can be chanky in size and hard to manage. Additionally, it would be better to increase the caching granularity to single files (say /bin/bash) instead of large layers. Physicists in the LHC computing GRID had a similar problem. They solced it with CernVM-FS. This talk explores the idea of using it for container image distribution. George Lestaris is a software engineer working at Pivotal in the Garden team, the container runtime of Cloud Foundry (CF). Before Pivotal, he spent time with high throughput computing on the LHC computing grid and cloud computing research in CERN. He has given talks before in CHEP 2013 regarding virtual clusters and their use in high-energy physics, in PyCon UK 2015 on interactive cloud experimentation and in the CF summit Santa Clara 2016 on containers in CF.