SONiC NOS production builds : a better way
_{Redefining a SONiC NOS build for a production environment}

SONiC stands for ‘Software for Open Networking in the Cloud’, which was initially developed by Microsoft for their Azure cloud. Microsoft released SONiC as open source software in 2016, and today it its development a co-ordinated effort involving contributions from various companies and organizations, including Microsoft, LinkedIn, Dell EMC, Broadcom, Mellanox, and others.

This collaborative approach ensures that, as SONiC develops, it continues to benefit from a wide range of expertise and perspectives across the industry. However, despite its versatility and power as network operating system (NOS), we have observed that many organizations struggle to reliably implement it in a production environment. 

Having previously solved this issue, we will now outline a step by step approach to reliably implementing a SONiC build at your organization.

Currently, the SONiC community practice is to make two releases in each year. So, there are two new branches published every year. Besides these, there is the main/master branch, which is the one into which features and other improvements are continuously added. Since each release goes through an elaborate validation of all SONiC’s features, with established benchmarks, it is a good idea to start any new project using the latest release.

Let’s say we  make a copy of the latest SONiC release and we don’t modify it. Now, we establish a practice of periodic builds (eg. daily or weekly) to create an image for a platform of our choice. We expect that the build will always be successful, since we are not making any modifications to the source. But surprise – this is not the case.

Why do builds so often fail?

As it turns out, the SONiC build procedure depends on various external components, like other open-source software, Debian packages, external libraries, utilities, etc. SONiC’s build procedure downloads these external components every time. However, these external components could be undergoing changes , and one or more of these changes can create errors while attempting to build the SONiC image.

This leads to an unpredictable build procedure which is unacceptable for any production environment. To eliminate this, we created a local mirror server for these external packages/utilities, identified these external components in the SONiC build procedure, and copied them into our local mirror setup. The mirror, however, does not keep updating itself with changes into the original packages; instead it is updated only when we specifically decide to upgrade to new versions of the packages. Then, we update the build scripts to use the mirror service instead ofthe original external site, so that the SONiC image build uses what we have stored in our environment (instead of downloading potentially different and unstable versions from various other sites).

The critical external dependencies for the SONiC build which cause this unpredictability are:

• Debian OS utility packages for SONiC supported releases, i.e. buster, bullseye & bookworm.
• Docker Debian infrastructure packages.
• Python library packages (PIP).
• External 3^rd party open-source source/Debian packages/utilities etc. (viz., Linux kernel, vendor specific SAI debs, ISC-DHCP debs, SNMP libraries, sFlow dependencies and many more).

Having discussed some of the causes for the failure of SONiC builds, we’ll now provide you with a build approach that works every time.

A step-by-step summary of our build approach is:

1. Configure a Local Mirror Server using Linux services.

2. Create a replica of the above identified external dependencies and host them using this local mirror setup.

• For Debian OS packages & utilities: Create a replica of Debian OS packages via controlled-sync using apt-mirror solution. Update SONiC NOS scripts to point to local server.

build_mirror_config.sh

…

DEFAULT_MIRROR_SECURITY_URLS=http://<local-mirror-server-url>/debian-security/

DEFAULT_MIRROR_SECURITY_URLS=http://<local-mirror-server-url>/debian-security/

• For Python packages: PIP utility supports the installation of Python packages without the use of external servers. Using this facility, create a local replica of required SONiC NOS python dependencies and install them using the pip option ‘—no-index –find-links’.

pip install package scripts of SONiC Source

Example:

…

— pip install flask

++ pip install –no-index –find-links=/pip-packages-path/ flask

• Open-source 3^rd party hosted artifacts: Pre-download the required dependencies and host them from local server via HTTP.

3. Update Community SONiC NOS scripts, Makefiles & configuration files to access the required packages from your locally configured mirror server, instead of external entities/servers.

Not only more reliable, but faster too

With the variability in SONiC’s external dependencies removed (which is responsible for so many problems), we can now generate our SONiC NOS build predictably.

As a bonus, we found that this solution approach further boosted reduced our overall SONiC build generation time by ~15-20%. In a typical 16 CPU core system, the overall build time now takes around 3 hours (down from 7 hours), so you are also likely to see time savings with this approach.

Capgemini Engineering helps clients to best use SONiC in their projects. Contact us today to see how we can help you leverage the benefits of open networking.