While working with other companies I have seen multiple builds and releases, plus also reading books like ‘Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation’. Through this I have learnt more and more about what should really be in the builds and releases of code applications. I would like to describe how I think they should both be used to create a scalable, reliable and repeatable process to bring confidence to your projects.
In showing these I will using Visual Studio Team Services(VSTS) and C#.NET as examples. These are the day to day parts I use and know how to represent what I would like to demo.
Continuous Integration Build
A Continuous Integration Build, or also known as CI Build, is the first build that your code should see. In the normal process I follow with code, you create a feature branch of your code which is where you write the new feature. Once you are happy with the product, it can be checked in to the development branch, where the current developing code is held before releasing. The CI build will sit in between this process to review what you are about to check in.
The goal of the CI Build is to protect the development branch and in turn all other developers that want to clone the code. Therefore we want to process the code in every way that it will be done in the release, but don’t move it. The general actions would be to get all assets, compile the code and then do unit tests.
By getting all the assets, I mean not just getting the code from the repository, but also other required assets like for ASP.NET projects, Nuget packages. This could also be things like Node Package Manager(NPM) packages and processing tasks like Grunt which manipulates the code before compiling. This is basically the setting up process for the code to be built.
We then compile the code into the state it will be used in, then run the unit tests. These unit tests are for checking if there are any errors in the code and should be testing your new change with the current state of the code, but a balance for this build is for speed as well and reliability. You and others will be checking into this branch multiple times during the day, so you don’t want to be waiting all day to find out if your code is ok. If your tests are taking a long time, then it might be an idea to do enough unit tests for you to be confident with the merge and then run all the longer and indepth tests overnight.
This build is optional to how well your daily CI build behaves. If you feel the CI is taking too long and you want to run some extensive tests on the project, then this is the build you will need. However not all projects are as large and detailed so might not need this.
The Nightly Build is the same process as the CI, as with Continuous Integration it should be a repeatable process, so it will get the resources and compile the code if required in the exact same method as the CI Build. Then at this point you can run all the same CI Unit Tests just as a confidence test that they still pass. You wouldn’t want to run the whole build and find out something failed in the small amount of tests you missed.
You can now run any lengthy Unit Tests, but what would also be good is to run any integration tests. These tests will stop using the stubbed versions of services and databases, to then use the real thing. The purpose of these tests are to make sure that when working with the real end points everything still works. When you use stubs for Unit Tests, you are practically configuring the end points to work as you would like. Even though you should be configuring them to be the same as the real deal, you can never 100% know they are working the same unless you just use the real thing. However just a point to be clear, when we say the real end points we do not mean the production ones, but the development versions instead.
After the build is complete, you should be confident that the code compiles fine, it works correctly by itself and works fine with the real systems as well. With this confidence there should be no hesitation to be happy to merge this into the next stage of the branching.
At this point you have compiled the code, tested the code, tested the integration and had human testers check the system. There is now 100% confidence that the project will work when it gets to its destination, so we move to packaging up the project and moving it to its destination.
However we don’t want to jus trust what was check a few days ago will be ok. What we do want is to trust that what we are packaging up at this point will be the working, tested and complete code. Therefore we do the repeatable process by getting the resources, compile the code and testing as much as what gives you confidence but as minimal as the Unit Tests. This now gives you the product that you should be happy to put on a server. It is also the same product you was happy with at the CI stage and the Nightly Build stage, so it is what you have practiced with throughout the process.
With the resulting product you can package it as required for the language and/or framework, which will be placed on the build server with a version number ready for the release. It is important that the package is accessible by the release for obvious reasons to pick the package up, but also the version number is very important. When the release picks up the package, we want to make sure it is the exact one we happily built, configured and tested. Most build tools like Visual Studio Team Services will automatically add this build id to the package and manage the collection of it.
We now have a confident deployable package to release, so there is no more building required, but there is still some configuration. When building an application that will be going to multiple location, you don’t want to use the same credentials for things like databases. This would be unsecure as if one of the servers was compromised then all of them are. There are also things like the database location as this would be different for each environment. There shouldn’t be one central system for all the environment, as this then can cause issues with that system goes down. If it is the development environment, then all systems should be applicable for just development. Nothing worst then testers bugging you because your development took down their testing.
What we will need to do is update the code to use the specific environment variables. This should be stored in the code base, so if the same application was deployed to multiple development environment there is minimal to no set up. Another example is a load balanced system where you want to deploy the same configuration to all servers. The way to do this will depend on the language, framework and system you are deploying to, but for an .NET Core project the best ways to have an ‘appsettings.config’ file for each environment. This would then be converted on deployment to its specific environment, so settings in the ‘appsettings.development.config’ would be merged in and the settings for ‘appsettings.production.config’ would not be touched until required.
Now the code is ready for the environment, but is the environment ready for the code. Part of the DevOps movement is Infrastructure As Code, where you not only configure the code for each environment, but also configure the environment. In a perfect cloud environment you would have the servers image with all the setting up instructions saved in the code base to keep all required assets in the same location. With this image you can target a server, install the image, then configure anything required for the environment, for example an environment variable, and finally deploy the code. This method would mean we could create and deploy any of the environments at will, for instance if the development server went down or was corrupted, you would point then fire to result is a perfect set up. An example of this would be using Azure with the JSON configuration details.
However we don’t all live in perfect world and our infrastructure is not always perfect, but we can still make it as good as we can. For instance I have worked on a managed OnPremise server where it has been created to a basic specification including Windows Operating System, user accounts and other basic details. This gives me a base to start with and an certain level of confidence that if I asked for another server to be created, it will be in the same format. Now I need to make sure it is fit for what I require it for, so we can use PowerShell that will run on the target machine to install things like IIS. This can be a script stored in the code base and then the environment variables pulled in from another file or the release configuration. This would give a level of Infrastructure As Code, by the requirements of the project being installed at each environment. This process could also check everything is in working order, so before you put your project on the server you are happy it is ready for it.
We should be all set to put the code and the server together now with a deployment, but once we have done that we have lost some confidence. Like the Integration Tests, we know the package is ok on its own and we know the server is ok on its own, but how do we know they are going to work together? At this point there should be some small, as to not increase the release time, but required tests to make sure that it has been installed correctly. This can depend on the project type and the environment etc, but should give you a certain level of confidence that everything will be ok. For an example you could have a URL endpoint that once called responds with the new codes version number. If the correct version is installed and set up on IIS, then it should be able to do this. There is now confidence it is in the correct place on the server, with the correct build version and working correctly with the environments set up. Of course this doesn’t test every endpoint of the project is working with no errors, but you would need to take some of that confidence from all the previous builds and testing.
With the CI Build every commit, the Nightly Build every night and the Release Build before all releases, then the configuration at each environment for both the server and the code, we end with a secure, resilient and well established product. This should result in you and your team being happy to fire a build or release off and not worrying about if it will work. An example of this confidence is once a developers code base was showing errors after a merge and didn’t know where the issue was. However because we had confidence in our CI build, we knew it would not be the base version but something on their machine, which closed the range of where the problem could be. In this instance it removed the question of is the base version stable and so sped up the process of finding the error.
I strongly suggest following this process or one relevant to your project as although it might take some time to set up and get developer comfortable with it, the time and assurance gain is much better.
Feel free to share any extra processes you do in your projects to create the safest process.