----
ChefDK 0.3.0 Released! Introducing Policyfiles
// Chef Blog
We're excited to announce that ChefDK 0.3.0 is now released. You can get it from our download page now.
Today's release contains a preview release of the new Policyfile feature. This is a significant milestone on our journey to improve the way you develop and distribute Chef code to manage your infrastructure.
What's this Policyfile Stuff?
Before we talk about Policyfiles in-depth, I want to be clear that the feature isn't complete and some aspects of the design haven't been finalized yet. Though one of our goals is to make Chef a lot easier to get started with, it's definitely not a good idea to start using it if you're new to Chef. If you're an experienced user, we'd love for you to use it and provide feedback. If you choose to try it, please do so in an isolated organization (or separate server, if you're using the 11.x Open Source Server). Aside from being good practice when trying pre-release features, there are some specific reasons why you need to be cautious when using the Policyfile preview release, which we'll describe more below.
Using Policyfiles Today
The rest of this article has a thorough explanation of what Policyfiles are, why we made certain design decisions, and so on, but before we get to that, let's go though a simple example. You can download the example code from GitHub if you'd like to see it all at once. We'll walk through it here:
Firstly, install ChefDK 0.3.0.
Now, create an app cookbook layout with chef's generator, and cd into that directory:
chef generate app policyfile_demo cd policyfile_demo Now create a file named Policyfile.rb with the following content:
name "jenkins" default_source :community run_list "java", "jenkins::master", "recipe[policyfile_demo]" cookbook "policyfile_demo", path: "cookbooks/policyfile_demo" Run chef install. This will solve dependency constraints, install 3rd party cookbooks to your cache, and emit a Policyfile.lock.json file. If you inspect this file, you'll see that it contains information about each cookbook used, including its name, version, source, content hash, and more.
To see the policyfile applied to a node, you'll need a Chef Server organization you can use for demo purposes. The example repo includes a knife config file you can use with knife serve to set up a temporary server with Chef Zero, or you can provide your own.
To upload your policy, run chef push demo. ChefDK uses your normal knife config file by default, use -c PATH to specify an alternate if necessary. You should see output like:
WARN: Uploading policy to policy group demo in compatibility mode WARN: Uploading cookbooks using semver compat mode Uploaded policyfile_demo 0.1.0 (f04cc40f) Uploaded java 1.28.0 (299d981f) Uploaded jenkins 2.1.2 (f5d46bcd) Uploaded apt 2.6.0 (278be58f) Uploaded runit 1.5.10 (b400659b) Uploaded build-essential 2.0.6 (88a112f9) Uploaded yum-epel 0.5.1 (631202c9) Uploaded yum 3.3.2 (0a0d0426) Policy uploaded as data bag item policyfiles/jenkins-demo You can inspect the uploaded data with knife if you're curious.
To configure chef-client for Policyfile mode, add the following directives to the client config file:
use_policyfile true deployment_group 'jenkins-demo' The use_policyfile true directive enables policyfile mode and the deployment_group 'jenkins-demo' part specifies both the Policy name ("jenkins") and the policy group ("demo") together.
There's currently a bug in chef-client policyfile mode for which the fix hasn't been released. The next minor release of Chef 11, and an upcoming Chef 12 preview release will contain the fix. To patch it, change line 160 of /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.2/lib/chef/policy_builder/policyfile.rb from
Chef::Cookbook::FileVendor.on_create { |manifest| Chef::Cookbook::RemoteFileVendor.new(manifest, api_service) } to
Chef::Cookbook::FileVendor.on_create { |manifest| Chef::Cookbook::RemoteFileVendor.new(manifest, http_api) } Now you can run chef-client to apply your policy to the node. Chef will run as normal, except: * It uses the run list from the Policyfile.lock.json * It downloads the cookbooks specified by the Policyfile.lock.json. No dependencies are computed at run time.
Ok, Tell Me More
The Policyfile is a Chef feature that allows you to specify precisely which cookbook revisions chef-client should use, and which recipes should be applied, via a single document. This document is uploaded to the Chef Server, where it is associated with a group of nodes. When these nodes run chef-client they fetch the cookbooks specified by the policyfile, and run the recipes specified by the policyfile's run list. A given revision of a policyfile can be promoted through deployment stages to safely and reliably deploy new configuration to your infrastructure.
A Policyfile.rb is a Ruby DSL where you specify a run_list and tell ChefDK where to find the cookbooks. It looks like this:
# Policyfile.rb name "jenkins" default_source :community run_list "java", "jenkins::master", "recipe[policyfile_demo]" cookbook "policyfile_demo", path: "cookbooks/policyfile_demo" When you run chef install, ChefDK caches any necessary cookbooks and emits a Policyfile.lock.json that describes the versions of cookbooks in use, a hash of the cookbooks' content, the cookbooks' sources, and other relevant data. It looks like this (content snipped for brevity):
{ "name": "jenkins", "run_list": [ "recipe", "recipe[jenkins::master]", "recipe[policyfile_demo::default]" ], "cookbook_locks": { "policyfile_demo": { "version": "0.1.0", "identifier": "f04cc40faf628253fe7d9566d66a1733fb1afbe9", "dotted_decimal_identifier": "67638399371010690.23642238397896298.25512023620585", "source": "cookbooks/policyfile_demo", "cache_key": null, "scm_info": null, "source_options": { "path": "cookbooks/policyfile_demo" } }, "java": { "version": "1.24.0", "identifier": "4c24ae46a6633e424925c24e683e0f43786236a3", "dotted_decimal_identifier": "21432429158228798.18657774985439294.16782456927907", "cache_key": "java-1.24.0-supermarket.getchef.com", "origin": "https://supermarket.getchef.com/api/v1/cookbooks/java/versions/1.24.0/download", "source_options": { "artifactserver": "https://supermarket.getchef.com/api/v1/cookbooks/java/versions/1.24.0/download", "version": "1.24.0" } You can then test this set of cookbooks in a VM or cloud instance. When you're ready to run this policy on a set of nodes attached to your server, you run the chef push command to push this revision of the policy to a specific policy group. We haven't determined the specifics of how Chef Server will implement policy groups, but at a basic level, they are containers for nodes, and you'll probably give them names like "staging" or "prod-cluster-1″–these are groups of nodes that all apply the same revision of a given policy. Your policies will have names like "jenkins-master" or "webapp" or "database"–the policy specifies a machine's functional role. Individual instances of chef-client will configure both a policy group and a policy.
The chef push command will also upload cookbooks to a new cookbooks storage API which stores each cookbook according to a SHA-1 hash of the cookbooks' content, so that chef-client is guranteed to apply a consistent set of cookbook code to your infrastructure (more on this below). For more information on the new cookbooks API, see this Chef RFC.
When chef-client runs, it reads its policy name and policy group from configuration, and requests the current policy for its name and group from the server. It then fetches the cookbooks from the new storage API, and then proceeds as normal.
Motivation and FAQ
Policyfiles are a pretty big change from the way Chef currently delivers configuration code to your machines today. Our design goals and answers to some of the more common questions about Policyfiles are below.
Focus Workflow on Configuring Machines to do Useful Work
Chef's current tooling (knife in particular) maps very closely to Chef Server's REST API and therefore is centered around manipulating individual objects and uploading them to the Chef Server. chef-client assembles these pieces at run time (more on that below) to configure a host to do some useful work for you organization. With the Policyfile feature, we want to focus the workflow on creating and configuring entire systems, rather than individual components. For example, Policyfiles describe whole systems and individual revisions of Policyfile.lock documents are uploaded with all required components as a unit to the Chef Server.
Code Visibility
In Chef currently, the exact set of cookbooks that a node will apply is defined by:
- The node's
run_listproperty; - Any roles present in the node's
run_listor recursively included by those roles; - The environment, which restricts the set of valid cookbook versions available to a particular node according to a variety of constraint operators;
- Dependencies specified in cookbook metadata;
- The dependency solver implementation, which tries to pick the best set of cookbooks that meet the environment and dependency criteria.
- Whether or not you or people in your organization have updated a particular version of a cookbook (see "Cookbook Mutability" for more about this).
These conditions are re-evaluated each time chef-client runs, so it's not always easy to tell exactly which cookbooks chef-client will run, or what the impact of updating a role or uploading a new cookbook will be.
The Policyfile feature solves this problem by computing the cookbook set on the workstation and producing a readable document of the solution. chef-client runs re-use the same precomputed solution until you explicitly update their specific policy group.
Role Mutability
Roles are currently global objects and changes to existing roles are applied immediately to all nodes that contain that role in their run_list (either directly or via another role's run_list). This means that updating existing roles can be very dangerous, so much so that many users advocate abandoning them entirely.
The Policyfile feature improves the situation in two ways. Firstly, roles are expanded at the time that the cookbook set is computed (i.e., the chef install step). Roles never appear in the Policyfile.lock.json document. As a result, roles are "baked in" to a particular revision of a policy, so that changes to a role can be tested and rolled out gradually. Secondly, Policyfiles offer an alternative means of managing the run_list for many nodes at once, since there is a one-to-many relationship between policies and nodes. Therefore users can, if desired, stop using roles without needing to use role cookbooks as a workaround for managing the run_list of their nodes.
Cookbook Mutability
The Chef Server currently allows an existing version of a cookbook to be mutated (in other words, you can change the code and re-upload without changing the version number, so that the "apache2 1.0.0″ cookbook has different code at different times). While this provides convenience for users who upload in-development cookbook revisions to a Chef Server (this is common among beginners and some Ci patterns), it presents the same problems as Role mutability. Some users account for this by following a rigorous testing process so that only fully integrated (i.e., all contributors' changes are merged) and well tested cookbooks are ever published to the Chef Server. While this process enforces good development habits, it is not appropriate for everyone, and should not be a prerequisite for getting safe behavior from the Chef Server.
The Policyfile feature solves this issue by using a new Chef Server cookbook publishing API which does not provide cookbook mutability. In order to avoid name collisions, cookbooks are stored by name and an arbitrary ID, which is computed from the content of the cookbook itself.
One particularly frustrating cause of name/version collisions is when users need to temporarily use a fork of an upstream cookbook. Even if the user contributes their change and the maintainer is very responsive, there may be a period of time where the user needs to use their fork in order to make progress. However, this presents a versioning quandry: if the user doesn't update the version, they must overwrite the existing copy of the cookbook on their server. Contrarily, if they do update the version number, they might conflict with the version number of a future release, which they could only fix by overwriting the newer version on their server. Using content-based IDs with sourcing metadata makes this use case easy.
But Opaque IDs are Confusing!
It's definitely true that such opaque IDs are less comfortable than the name, version number scheme that users are used to. "my-cookbook 1.2.3″ feels much more meaningful than "my-cookbook ddf827″, for example. In order to ameliorate the problem, ChefDK and the new server APIs do a few things:
- When working with the
Policyfile.rb, you deal with cookbooks mostly in terms of names and version numbers. For cookbooks from an artifact service like supermarket, you use names and version constraints like you're used to; for cookbooks from git, you use branch/tag/revision as you're used to; for cookbooks from disk, you specify paths. The opaque IDs are mostly behind the scenes. - Extra metadata about cookbooks is stored and included in API responses, as well as the
Policyfile.lock.json. This includes the source of the cookbook (supermarket, git, local disk, etc.) and the upstream ID of the cookbook (such as git revision); for cookbooks loaded from local disk, the Policyfile implementation detects if they are in a git repo and collects the upstream URL, current revision ID, whether the repo is dirty, and whether the commits are pushed to the remote. - Cookbooks uploaded to the new cookbook storage API can have extended SemVer version numbers with prerelease sections, like
1.0.0-dev.
Limit Expensive Computation on Chef Server
In order to determine the cookbook set for a given chef-client run, Chef Server has to load dependency data for all known versions of all cookbooks, and then run an expensive (NP expensive) computation to determine the correct set. Moving this computation to the workstation and running it less frequently makes Chef Server more efficient.
Where Does the Policyfile Live?
At the moment, we see three main ways to organize your Policyfiles:
- Store the Policyfile and related cookbooks in the same repository as the application you're deploying. If you're deploying custom applications written in-house and your software developers are comfortable working with Chef, you can put the Policyfile in the same repo as the application's source code, along with any application-specific cookbooks, and version everything together.
- Store the Policyfile with a cookbook. If you're following the single cookbook per repo workflow, you can include the Policyfile in the highest-level cookbook's (i.e., the cookbook ultimately responsible for deploying a server's primary application) repository.
- Store all of your Policyfiles in a single directory. This is likely to be the most common way to use Policyfiles with the monolithic cookbook repo workflow. There are still some details to be worked out for this case.
Am I Going to be Forced to Use This?
This change will be rolled out by adding new functionality to Chef Client, ChefDK, and Chef Server alongside the existing functionality. There are no plans to remove or even deprecate the existing APIs. The plan is to get people to switch by offering an alternative with both more safety and more freedom than the current situation.
That said, if adoption is large enough then eventually we may remove the dependency solver from Chef Server.
Does this Replace Berkshelf?
The Policyfile is definitely a replacement for the "environment cookbook" pattern in Berkshelf. It also provides a dependency solver and fetcher (thanks to some code from berks), so it may replace some other berkshelf use cases. However it is much less opinionated than Berkshelf, and may not replace Berkshelf for all use cases, so we'll have to see how things turn out.
Does this Force Me to Use the Single Cookbook per Repo Thing?
No. We're still figuring out the optimal way to support the "megarepo" workflow, but it will be supported. In particular we have to study the tradeoffs of versioning your Policyfile.rb files (we'll support other names) with your chef-repo vs. outside of it. We plan to do some dogfood testing to inform the design here.
Users who use the "megarepo" workflow may see some benefit to using single repos for third-party cookbooks, but this will be optional and users can convert from vendor branches piecemeal if they decide to do so.
Do I Have to Change My Workflow to Use This?
The answer to this depends on how you define "workflow." As noted above, you can choose to have a chef-repo or not, and you can fetch third party cookbooks using the Policyfile or an out of band mechanism (vendor branches). You and your team can decide to publish only completely integrated "release" cookbooks to the server if that works for you, but you can also safely publish development versions of cookbooks to the server without risk of mutating the production versions and without needing a versioning scheme (devodd and friends) to workaround cookbook mutability issues.
That said, the mechanics of how you get configuration code from your workstation to production will be different. In particular, when using the Policyfile feature in the recommended way, you cannot publish an updated cookbook or role and have it applied immediately to all machines. Tools that use the old APIs will need to be updated.
Are Policyfiles Versioned?
There currently isn't any detailed design for the Chef Server policy API (which will store the Policyfile.lock.json documents). One design decision we have made is that the documents will be namespaced (by policy group). This means that at minimum it will be possible to independently update the policy for different stages of your release process independently. For example, if you have policy groups for "dev," "stage," and "prod," you can iterate on new feature work in "dev" and release a critical hot fix to "prod" independently of each other.
If this is all that's implemented, then you will be able to version your policies by committing your Policyfile.rb and Policyfile.lock.json documents to revision control and using a branching policy that fits your release requirements. That said, features to support operations such as reverting or undo and/or tracking changes over time will be considered.
What About Environments?
We have not made a final decision about how environments will work with Policyfiles. In compatibility mode, you cannot use environments and Policyfiles together, but this choice could be reversed. Policyfiles do completely replace the cookbook version constraint portion of the environments feature. However, environments do offer a useful way to set environment-wide attributes, which some users rely on heavily. The main sticking point is that environments provide the same double edged sword as many other Chef features where updates to environments are propagated immediately to all nodes in an environment. When done correctly, this is very convenient, but it also allows mistakes to propagate to all nodes immediately. Contrarily, if environment attributes are rolled into the Policyfile, you can more easily test the effects of changes and control the way these updates are applied, but it's more difficult to apply changes globally.
Compatibility Mode
The Policyfile feature depends on new APIs in Chef Server that don't yet exist. In order to provide a preview of the feature, the current implementation operates in a compatibility mode that uses existing Chef Server APIs to demonstrate the Policyfile behavior. This makes it difficult to safely use Policyfiles and the current code publishing mechanisms in the same organization (or 11.x Open Source Server), so we recommend you use a separate organization when trying the policyfile features.
Cookbook Artifact Storage
In compatibility mode, ChefDK must implement content-hash-based storage of cookbooks using the existing /cookbooks endpoint. To do so, it maps hash IDs to X.Y.Z version numbers. While this works to demonstrate the Policyfile behavior, it is certainly a kludge. If you are trying the Policyfile feature in compatibility mode, beware:
- Cookbooks uploaded by the policyfile commands will have very large version numbers with no sort order. Any
chef-clientthat is not operating in policyfile mode will prefer these cookbooks to ones uploaded normally unless you are dilligent about using environment version constraints. - The
/cookbooksendpoint is not designed to be used this way, so it doesn't show you the "real" version numbers or additional metadata about these cookbooks. While we have plans to make arbitrary cookbook IDs easier to manage in the final implementation, there's little we can do about it in the exiting API.
Policyfile Storage
In compatibility mode, ChefDK uses data bag items to store Policyfile.lock.json documents. To minimize the chance of conflict with other data bag items, ChefDK stores all of these documents in the "policyfiles" data bag; individual Policyfile.lock.json revisions are given IDs of the form $policyname-policygroup.
Upcoming Features
The implementation of the Policyfile feature is still very incomplete. Here's a list of features we'll be working on as we make progress towards a production-ready release:
- Conservative Updating: ability to update a subset of the cookbooks in your
Policyfile.lock.json - Role Support: roles aren't supported in
Policyfile.rbrun lists yet, but will be soon. - Policyfile Attributes: you will be able to set role-level attributes directly in the
Policyfile.rb. - Multiple Run List Support: Policyfiles don't support override run lists, so named run lists will be allowed in the
Policyfile.rbas a replacement. - In-House Cookbook Hosting: Currently you cannot host cookbooks behind your firewall; support will be added for using Chef Server's
/cookbooksendpoint as a "mini supermarket", or you will be able to run a full supermarket instance. - Server API Support: As mentioned above, "native" support for Policyfile documents and cookbook storage will be added to the Chef Server.
Chat Us Up at the Summit(s)!
If you'd like to chat about anything Policyfile-related, the Community Summit is a great venue. If you can't make it, we're available on the mailing list and IRC if you have any questions.
----
Shared via my feedly reader
Sent from my iPhone
No comments:
Post a Comment