satis egitimisatis


Discussion on the state of cloud computing and open source software that helps build, manage, and deliver everything-as-a-service.

  • Home
    Home This is where you can find all the blog posts throughout the site.
  • Categories
    Categories Displays a list of categories from this blog.
  • Tags
    Tags Displays a list of tags that has been used in the blog.
  • Bloggers
    Bloggers Search for your favorite blogger from this site.
  • Login
Posted by on in Open Source
  • Font size: Larger Smaller
  • Hits: 6270
  • Print
  • Report this post

How the Xen Project Hypervisor team manages releases

With the Xen Project Hypervisor 4.4 having been released a few weeks ago, the project is starting the planning cycle for version 4.5 of the Hypervisor. So I thought it is worth walking you through how we manage releases.

Welcoming Oracle's Konrad Rzeszutek Wilk as Release Manager

But first I wanted to thank George Dunlap from Citrix for successfully managing the 4.3 and 4.4 releases of the Xen Project Hypervisor. The Release Manager role is a volunteer role open to Xen Project maintainers. George, has stepped down from his role recently to find more time for coding and help bootstrap the CentOS virtualization SIG.

Konrad Rzeszutek Wilk has volunteered to take on the role for the 4.5 release. A big welcome and Thank You!

Konrad is Software Development Manager at Oracle. His group's mission is to make Linux and Xen Project virtualization better and faster. As part of this work Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.

How The Xen Project manages releases

As is the case for many open source projects, the Xen Project community does not maintain a committed roadmap as proprietary software vendors do. Instead, we strive to accurately track development for new releases, with a predictable release cadence for major releases and maintenance releases. We aim to release the Xen Project Hypervisor every 6-7 months: historically our release cadences ranged from 9 to 18 months. Introducing the Release Manager role was instrumental to getting us to shorter and a predictable release cadence.

How Xen Project contributors operate

Our best practice recommends that contributors announce larger new features on the xen-devel mailing list. Otherwise there is a risk that several contributors may start implementing a feature in parallel, wasting time and creating friction. The Release Manager will usually follow up and ask whether the feature should be included into the releases' backlog. There is also a monthly community call to raise issues and discuss bigger features. We run a Hackathon once a year (typically in spring) and a 1/2 day face-2-face meeting before or after our Developer Summit at which all of our core developers are present. These face-2-face meetings are usually used to bootstrap the planning process for a release.

How the Xen Project creates a Roadmap

During planning and development of a release, the Release Manager sends out monthly development update e-mails on xen-devel. These mails are labelled Xen x.y Development Update: you can see examples by following the 4.4 mails. In these mails, we ask developers

  • whether they have items to be tracked on the roadmap and if so add them to a backlog
  • asks for status updates, when they are on the backlog
  • asks how likely it is that a feature will be in our codeline (including reviews) by the next update

Towards the end of the development cycle we start to treat blocker and critical bugs in the same way: aka they are tracked by the Release Manager on he backlog. Public announcements on release status are made on our blog and the announce list at critical points in the release cycle.

Stages within a Release Cycle

The following diagram shows the key stages, gates and git branches in our release cycle.  


Feature Development / Feature Freeze: Feature development starts as soon as the release branch for the previous release has been created. It ends at the Feature Freeze point, when we typically won't allow new features into the master branch. Patches that were submitted before Feature Freeze and are being reviewed may still be let in based on a risk analysis. This phase is typically 3-4 months long. The timing is adapted based on major holidays (XMas and New Years) or major conferences, such as Xen Project Developer Summits.

Hardening / Code Freeze: During the hardening phase, which typically lasts 3-4 weeks, we complete any remaining reviews of patches that were posted during development and focus on bug fixes. This phase ends with the Code Freeze point, after which each bug will be considered on a case-by-case basis and risk to the quality of the release. Major vendors will start running their complete test suites during this phase of development. This phase ends with the first release candidate (RC1).

Release Candidates / Release: We close the release cycle making a number of release candidates (RCs). Historically, we needed 4-6 RCs and create one every 1-2 weeks. During the Release Candidate phase, we run Xen Project Test Days and major vendors in the community run their test suites against RCs. Only bugs of blocker severity are fixed. This phase typically lasts 6 weeks, but may be extended if a blocker bugs are found that need to be addressed. We end this phase by creating a stable release branch.

Release Announcement: Usually we make an official Release Announcement shortly after the stable release branch has been created. But sometimes, we delay by a few days to coordinate press activities.

Improving the Process

We generally have discussions on how to improve the release process, at face-2-face meetings such as the Hackathons and Developer meetings. At previous developer meetings, we decided to focus on improving our test infrastructure and started using Coverity Scan regularly on our codebase and formed the test framework working group. In the upcoming May Hackathon, we will for example have a discussion how to handle the ever increasing traffic on our development list (which regularly hits more than 4000 messages per month).

Further Information

Rate this blog entry:
Trackback URL for this blog entry.
Lars Kurth had his first contact with the open source community in 1997 when he worked on various parts of the ARM toolchain. This experience led Lars to become a passionate open source enthusiast who worked with and for many open source communities over the past 15 years. Lars contributed to projects such as GCC, Eclipse, Symbian and Xen and became the open source community manager for the Xen Project in 2011 and later the chairman of the Xen Project Advisory Board. Lars is an IT generalist with a wide range of skills in software development and methodology. He is experienced in leading and building engineering teams and communities, as well as constructing marketing, product management, and change programs impacting 1000s of users. He has 17 years of industry experience in the infrastructure, tools and mobile sectors working at ARM, Citrix, Nokia and the Symbian Foundation.
  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest Wednesday, 10 September 2014


Citrix supports the open source community via developer support and evangeslism. We have a number of developers and evangelists that participate actively in the open source community in Apache Cloudstack, OpenDaylight, Xen Project and XenServer. We also conduct educational activities via the Build A Cloud events held all over the world.