CloudBolt's engineering team constantly provides new features and fixes bugs in the product. Our agile philosophy leads to rapid releases and rapid delivery of small, incremental features. We ship an Alpha nearly every week (which is tightly coupled to our 1-week sprint schedules) until we've hit the features we want in a release, usually 1–3 months. Then we begin shipping RCs (Release Candidates) for that release, usually for about a week. Finally, we ship a GA (General Availability) that is ready for production use.
A question we often get is: Can I run pre-release versions of CloudBolt in production? The answer isn't straightforward. We can't promise a pre-release version will work 100% correctly since it contains features that are still in development and bugs that haven't been found or fixed yet. However, we love feedback from our customers, which is why we provide pre-release versions of CloudBolt to our customers.
What follows are some details about our development process that might help you decide whether to run a pre-release version of the product.
CloudBolt's Development and Testing Process
As our engineers develop fixes and features for CloudBolt, they check them into our source code repository—instantly, a suite of automated tests begins testing the change, and within minutes we have a base understanding of whether the change broke anything obvious. These tests automatically build a Frequent pre-release version of CloudBolt, upgrade a server with that version, and run a suite of unit tests that cover about 33% of our codebase. (A number that we are always pushing higher.) We also use the upgraded CloudBolt server to deploy and tear down a server in Amazon, just to make sure CloudBolt can perform its most basic task.
Later that night, after our engineers have gone home, a Nightly pre-release version of CloudBolt is built. Everything in the Frequent testing repeats, integrating all of the day's changes. We also go many steps further and use the Continuous Infrastructure Testing (CIT) feature built into CloudBolt to perform a wide range of functions with CloudBolt, running about five hours of tests. We prove that provisioning on VMware, AWS, Azure, GCE, Softlayer, and others works correctly. We test remote scripts and actions, configuration managers like Chef and Puppet, verify the UI responds quickly, run Rules and Sync VMs, and finally test upgrades from the supported versions of CloudBolt (see our Deprecation Schedule for more details.). Our team reviews the results of these tests daily and usually resolves any issues discovered the previous night the next day.
Every weekend, we build a Weekly pre-release version of CloudBolt that repeats the above steps but includes another ten hours of automated CIT testing and which builds around 100 servers using CloudBolt with various combinations of technologies, crawls our API for errors, and runs through a large suite of edge cases. The following Monday, we review the results of the weekend and begin working to resolve any issues that were found through testing. If we discover and fix issues, we'll re-run the individual tests that failed until they are back to green.
Most Tuesdays (or typically once during each week), we manually build an Alpha from the latest green point in our codebase. Since it was automatically tested, the only test we perform is to make sure the upgrader runs successfully on our individual development servers. If that works, we publish it to our Downloads page and send an email to Alpha testers. (Want to get on this list? Submit a support ticket to firstname.lastname@example.org requesting to be added to the list of Alpha testers.) Your feedback from using these Alphas results in us filing additional bugs and feature requests which we often work to include in subsequent sprints, but otherwise go into our backlog for future consideration with similar stories in a sprint. We likely won't provide a fix right away, since these releases aren't intended to be run in production, but will still accept support tickets so you can report any issues you find. Our advice will usually be to upgrade to the latest Alpha version if you haven't yet, and we may not begin researching the ticket until you have done so.
Near the end of a release cycle, we will cut our first Release Candidate (RC). Functionally, these are the same as an Alpha, but the cutting of a Release Candidate means that our engineering team has transitioned into a new phase of the development cycle: now we're regression testing the product. Throughout the development cycle, as developers worked on different areas of the product, they were noting areas of the product that they touched as an area that needs regression testing. In a larger release, we may go through upwards of 200–400 manual regression tests in a week's QA cycle and may file upwards of 20 bugs that weren't caught through automated testing. These defects are fixed that same week, and we'll re-test the bugs that were discovered. Since these early Release Candidates may still have untested bugs, we likely won't provide fixes right away (as with Alphas), and don't recommend running RCs in production, but you are still welcome to submit support tickets for anything you find. We will likely require you to upgrade to the latest RC if you find an issue in an older RC.
At the end of the QA cycle, we'll cut a Final Release Candidate (RC). This final RC should look and feel identical to a GA release, and often contains the exact same bits, but occasionally a few additional release stoppers will be discovered by customers or our developers before we cut the GA, so a handful of additional bugfixes might make it into the product. We call the ~1-week period after QA before we cut the GA "Cooldown Week" and usually work on smaller improvements. Later Release Candidates are usually fairly stable. Depending on your tolerance for risk, you may want to run an RC in production, but be aware that any last-minute bugs that are discovered may get fixed before a GA.
With the end of Cooldown Week and following one final suite of (green) weekly tests, we cut the General Availability (GA) release and celebrate publishing it by kicking off the next sprint of awesome features.
So can I run pre-release versions of CloudBolt in production?
Alphas may contain features that are in the process of being developed and are likely to contain untested features and undiscovered bugs. If you are comfortable using CloudBolt under such a scenario, if you are comfortable performing frequent upgrades on your CloudBolt appliance, and if you are comfortable running with such bugs in production, running an Alpha in production might be right for you.
Release Candidates, though more stable, fall into the same category as Alphas.
In both categories of pre-releases, CloudBolt will not provide patches to resolve your issue, and will provide fixes in the form of a future upgrader.
GA releases will be supported according to our Deprecation Schedule, but you may be required to upgrade to a later version of CloudBolt in order to receive support on a particular issue, especially if it has already been fixed in a later version.