The problem
A common development paradigm is as follows :
- Joe developer commits code into a feature branch
- Joe developer deploys that code on his own machine to see if it works
- Joe developer either continues committing and deploying or he finds code that he likes and merges it into a dev branch
- At the same time Alice developer is committing code in her feature branch, deploying on her machine and eventually merging into a dev branch
- The current live production code lives in yet another branch and is deployed on different machines than the one's Alice and Joe deploy to
Alice and Joe's code is segregated from each other with the VCS branch model. Their deployed instances of their code are segregated from each other because they're deploying to different machines. The same is true for production.
If your organization runs a single Chef server for all environments (production, staging, development) and not one Chef server for each environment, there exists a namespacing issue with cookbooks. Whereas above in traditional development each developer has their own deployment environment, here there is a common shared Chef server. The result is that as Joe and Alice develop their code, in order to see it work on their respective machines, they need to upload it to the same Chef server. To retain the fully asynchronous development model traditionally available to developers using VCS and that have their own development machine, a separate Chef server would need to be run for each feature branch and environment. A promotion model would need to be created to enable promoting cookbooks and other Chef data (databags, environments, run_lists) from on Chef server to another (e.g. from a feature branch to the dev branch, or from the staging branch to the production branch)
Chef does provide one variable to differentiate different branches of a cookbook from each other, the cookbook version. Each cookbook has a metadata file containing information on the cookbook including the version of the cookbook in the 1.2.3 form. One complexity to the way Chef handles cookbook versions is that, by default, nothing prevents you from uploading a modified cookbook with the same version as the existing cookbook on the Chef server, thereby changing the functionality of that Cookbook at the version without a change in version number.
More information
Freeze
Chef provides the ability to "freeze" cookbooks which flags them such that the Chef server won't allow you to overwrite an existing version of that cookbook (unless you pass it a "force" option)
Environment Based Cookbook Version Constraints
Chef also allows you to specify what version or versions of a given cookbook are allowed to be used in a given environment.
Solutions
Multi chef server
You could make the Chef server provisioning method very streamlined and enable anyone to spine up their own ephemeral Chef server easily, synchronize it's datastore with a parent version, develop and deploy to your own Chef server, and destroy it when the feature is complete and merged.
This would be complex and hard to implement.
Chef solo
You could develop on Chef Solo and not deal with the constrained cookbook namespace until you merge your feature branch into the dev branch
This prevents you from using any of the Chef Server functionality (the ability to search across your server base, databags, etc) since it wouldn't be present in Chef Solo.
Namespacing with version number : Mapping the version fields
You could dedicate each of the 3 version fields (major, minor, patch) to a different environment to avoid collision.
This constrains you to 3 environments and doesn't allow for feature branches. It also requires that, in development, every time a developer wants to test his change to see if it works, the cookbook version needs to be incremented. This results in a large proliferation of different versions of a given cookbook which doesn't fit well with the Chef model. It also violates semantic versioning standards
Namespacing with version number : Assigning version ranges
You could designate ranges of versions for each environment. For example production gets 9000.0.0 - 9999.0.0, staging gets 8000.0.0 - 8999.0.0, dev gets 7000.0.0 - 7999.0.0 and each developer gets their own range for feature branches.
This method results in foreign looking version numbers and again violates semantic versioning standards. It also bounds the maximum number of feature branches that you have.
Namespacing with version number : Protect only what you need to
You could
- modify the Chef server, removing the REST interface for the cookbook upload function
- create a redirect on the Chef server that reveals a new REST interface which points to the cookbook upload function
- create a knife plugin that extends the environment feature to add a "set cookbook version"
- The new knife plugin would
- See if the environment being affected is production or not production
- If it's not production, set the cookbook version for the environment
- If it is production, append the tuple of the cookbook and cookbook version being set to a list in a databag indicating that this cookbook at this version has ever run in production. Next, set the production environment cookbook version to the one indicated. We'll use this later to know if a given cookbook at a given version has ever run in production and if so to protect it from being changed.
- create a knife plugin that extends the cookbook upload functionality and talks to this new API interface
- The new knife plugin would
- Query the Chef server to see if it has a cookbook of the same version as the one being uploaded already
- If it doesn't, do a couple git queries of the local git repo to determine the name of the branch that it's on and the committer name of the most recent commit and add those into the cookbook metadata file. Then upload the cookbook and revert the local metadata file changes.
- If there is a conflicting version, check the databag mentioned above to see if cookbook at the give version, for which there's a copy already on the Chef server, is now running or has ever been run in production.
- If it has, refuse to make the environment cookbook version change. This prevents anyone from accidentally changing production
- If it hasn't, upload the cookbook to the Chef server replacing it's existing cookbook version with the new contents. Read the branch name and last committer from the cookbook metadata, check to see if it's different than the current users branch and last committer name and if so report to the user that they've just overwritten that persons cookbook at that version.
Upsides :
- Nobody can accidentally change production behavior by uploading a cookbook. The only way to change production behavior is to change the production cookbook version constraint which should be obvious as to it's impact.
- Rapid development iteration is possible without concern over impacting other developers or environments. The developer can iterate changes over and over without incrementing cookbook version, testing each time he commits and uploads. The upload function could be linked to the commit as a commit hook.
- The complexity of enabling this model is low (server side redirects, 2 simple knife plugin extensions).
- Cookbook versioning can conform to semantic versioning standards because the versions are chosen by humans.
Downsides :
- There is nothing to prevent any developer, QA or devops engineer from changing production behavior by changing the production cookbook version constraint. Depending on how collaborative or combative these groups are with each other, mere policy constraints saying that developers shouldn't change production behavior may not be enough.
- If a developer created their own knife plugin to interact with the new cookbook upload REST interface that didn't block overwriting an existing cookbook of a version being used in production, they could change production behavior by overwriting the contents of a cookbook running in production. Again, this comes down to the place on the collaborative combative spectrum that your organization lies.
- Developers can unintentionally change the behavior of other development or staging environments if they commit to the same cookbook version as one that somebody else is developing on. This would happen if two developers begin working on feature branches of the same cookbook around the same time and choose the same version number. The developer will be notified that this has happened but not prevented from doing it. In this case, keep in mind, no code is lost since everything is in VCS, it merely means that one developer could momentarily step on another developers feet.
Summary
These are some of the solutions that I've come up with. I don't feel that any of them are very elegant. I also could be misunderstanding the problem in which case there may be a whole other class of solutions available that I haven't imagined. What do you think?
