Policyfiles: A Guided Tour [feedly]

Policyfiles: A Guided Tour
https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/

-- via my feedly.com reader

If you watched the ChefConf keynote, attended last years' community summits, or follow our open source mailing lists, you've probably heard about Policyfiles.

If you haven't, here's the deal: Policies are a new feature of Chef that combine the very best parts of Roles, Environments, and client-side dependency resolvers like Berkshelf into a single easy to use workflow. Policies are built by defining a Policyfile, which looks similar to a Chef Role combined with a Berksfile. When a Policy is ready for upload, a workstation command included with the ChefDK compiles the Policyfile into a Policyfile.lock file. This locked Policy, along with all of the cookbooks it references, are treated as a single unit by the Chef tooling. The bundle of Policyfile.lock and cookbooks are uploaded to the server simultaneously. They are also promoted simultaneously through the deployment lifecycle, from dev to QA to production.

Policies make your chef-client runs completely repeatable, because cookbooks referenced in a Policy are identified by a unique hash based on their contents. This means that once the lock file + cookbook bundle has been generated, the code underlying it will never change.

For more information, see the Policyfile README in the ChefDK repo.

Previously we've recommended that you only use Policyfiles in specialized testing environments because using Policyfiles in compatibility mode along side existing infrastructure could cause unexpected behavior. With Chef Server 12.1 and ChefDK 0.7, Policyfile data is now stored via specialized APIs, making it safe (and a lot easier) to use Policyfiles in your existing Chef Server setup.

To help you get familiar with the workflows Policyfiles make possible, we'll walk through deploying a simple demo application using Policyfiles to manage our dependencies across the application's lifecycle.

The code here is based on a demo my colleague created for the London Chef meetup. It deploys the "Awesome Appliance Repair" Python application.

Getting Started With Local Development and Testing

To follow along with this example, you'll need:

• ChefDK 0.7+
• Hosted Chef or Chef Server 12.1+
• Vagrant (for local testing with kitchen)
• To have configured your ~/.chef/config.rb (or ~/.chef/knife.rb) to communicate with your Chef Server.

Initialize Shell

If you have previously installed ChefDK or Test Kitchen on your machine using rubygems, you might have older versions of these tools in your PATH. If that's the case, you can use chef shell-init to setup your environment variables, like so (I use zsh, be sure to update the command for your shell):

eval "$(chef shell-init zsh)"  which kitchen  # => /opt/chefdk/bin/kitchen  

Generate:

For this example, we'll structure our code as if we were developing our infrastructure code alongside the application, in the same source repo.

Note that Policyfiles don't require you to manage your code this way, you can use individual git repos per cookbook or a monolithic repo if you prefer. Just follow along for now :)

We use chef generate to create the required files and directories for us:

mkdir aar  cd aar  chef generate app .  # Policyfiles will be the default someday, 'till then:  chef generate policyfile  

Commit

We'll commit our work now so we can roll back to a fairly blank slate if we make a mistake later:

git add .  git commit -m 'initial policyfile demo commit'  

Edit Policyfile:

We describe how we want Chef to compose our cookbooks to configure a machine to run our application by editing the Policyfile.rb. This file defines a few things:

• name: This describes the kind of machine we are creating. We name this "aar", which is our abbreviation for "Awesome Appliance Repair."
• default_source: The place where we get shared cookbooks. The default source is :community, which is the Chef Supermarket site. You can also use an internal supermarket instance or a monolithic Chef Repo.
• run_list: The list of recipes, in order, that you want Chef to evaluate to configure your system. When using Policyfiles, you set the run_list in the Policyfile instead of on the node.
• cookbook: A Policyfile.rb can have multiple cookbook statements; these can configure specific cookbooks to be loaded from alternative sources or set additional version constraints on them.
• Default and override attributes: these define attributes at the 'role' precedence level. We'll look at these later.

Edit the Policyfile.rb as follows:

# Policyfile.rb - Describe how you want Chef to build your system.  #  # For more information on the Policyfile feature, visit  # https://github.com/opscode/chef-dk/blob/master/POLICYFILE_README.md    # A name that describes what the system you're building with Chef does.  name "aar"    # Where to find external cookbooks:  default_source :community    # run_list: chef-client will run these recipes in the order specified.  run_list "aar::default"    # Specify a custom source for a single cookbook:   cookbook "aar", path: "cookbooks/aar"  

Chef Install

With our basic Policyfile.rb, we run chef install to fetch dependencies and generate a Policyfile.lock.json. We haven't specified any dependencies yet, so we don't need to fetch anything, but we will need the lockfile to be generated before we can proceed to the next step.

chef install  

Let's take a peek at the Policyfile.lock.json we just created. We'll go over each part individually:

Revision ID

  "revision_id": "5f750bf464100b487cd7c276c5d532341b79fbeb5e8accd29538ae972896992b",  

Each time we create or update the lock, chef will automatically generate a revision_idbased on the content. These values are used to automatically version your policies, so that you can apply different revsions of a policy to different set of servers. We'll see this in action a little later.

Name and Run List

  "name": "aar",    "run_list": [      "recipe[aar::default]"    ],  

The lock includes the name and run list we specified previously. The run list is normalized to the least ambiguous form.

Cookbook Locks

  "cookbook_locks": {      "aar": {        "version": "0.1.0",        "identifier": "cff2d37260c04b21053ad30b68aa20e674e52e6c",        "dotted_decimal_identifier": "58532310150070347.9294424438433962.36174175743596",        "source": "cookbooks/aar",        "cache_key": null,        "scm_info": {          "scm": "git",          "remote": null,          "revision": "3455fb415d56f9a7cabbb76f2063942a6547b2eb",          "working_tree_clean": true,          "published": false,          "synchronized_remote_branches": [            ]        },        "source_options": {          "path": "cookbooks/aar"        }      }    },  

For each cookbook we use, there is a corresponding entry in the cookbook_locks section. The exact data collected about each cookbook is dependent on the cookbook's source. In this case, we have a cookbook sourced from the local disk which happens to be in a git repo. In the event we need to debug this cookbook later, ChefDK has collected information about the cookbook's git revision. If we'd setup a remote, git would tell us the cookbook's git URL and whether we'd pushed this commit to a branch on the remote.

Attributes

  "default_attributes": {      },    "override_attributes": {      },  

Policyfiles have attributes that replace role attributes. We'll see these a little later.

The Rest

  "solution_dependencies": {  

You can ignore the solution_dependencies section. It's used to keep track of dependencies in your cookbooks so ChefDK can check whether changes to your cookbooks are compatible with their dependencies without having to download the full cookbook list from supermarket every time.

Commit the lockfile

We'll want to compare it to an updated version later, to see what changed.

git add Policyfile.lock.json  git commit -a -m 'updated Policyfile and created lock'  

Edit .kitchen.yml

ChefDK ships with a policyfile_zero provisioner for Test Kitchen that allows us to test our policies in local (or cloud) VMs. Note that currently Chef Zero doesn't fully support "native" Policyfile APIs, so instead it runs in compatibility mode. This isn't a problem on an isolated server like Chef's local mode, but it's something you might notice when debugging.

---  driver:    name: vagrant    network:      - ["forwarded_port", {guest: 80, host: 8080}]    provisioner:    name: policyfile_zero    require_chef_omnibus: 12.3.0    platforms:    - name: ubuntu-14.04    suites:    - name: default      attributes:  

Run TK with Empty Cookbook

To verify our test rig works, we'll run kitchen with our empty cookbook:

kitchen converge  

If you get an error like Message: Could not load the 'policyfile_zero' provisioner from the load path then you didn't run the chef shell-init stepabove. Run that and try again.

If that worked without a problem, you can throw that VM away:

kitchen destroy  

Develop your Cookbook

This is where we'd normally run a TDD testing loop, but for the purpose of this walkthrough, we'll just import the aar cookbook fully formed:

cd cookbooks  curl -LO https://github.com/danielsdeleo/aar/releases/download/draft-1/aar-cookbook.tgz  tar zxvf aar-cookbook.tgz  rm aar-cookbook.tgz  cd ..  

If you inspect cookbooks/aar/metadata.rb, you'll notice that our cookbook now has some dependencies, but we haven't yet downloaded them. We'll do that next.

Chef Update and Commit

Now that we've added some dependencies to our cookbook, we need to run chef updateto fetch them.

chef update  

This will recompute our dependencies and cache all the cookbooks we need for our Policy.

To see what's changed since our last commit, we could just run git diff, but chef diffgives us itemized output, listing added, removed, and changed cookbooks. Let's give it a go:

chef diff --head  

Now that we're satisfied with our changes, we'll commit again.

git add .  git commit -m 'Update aar cookbook and deps'  

Run Kitchen Again

We can run test kitchen again to see the result of our changes:

kitchen converge  

Visit the Site

With the port forwarding, we can visit http://localhost:8080 and see the site. For more info on using the application, see the awesome appliance repair README on github.

Deploy

We can use the chef provision feature to create a "staging" and then a "production" node. We'll use these to see how we can deploy different revisions of a policy to different machines. Since chef provision is new and somewhat experimental, it's not yet integrated with chef generate, however, we can generate a cookbook like this (make sure it's named "provision", that name is special):

chef generate cookbook provision  

Then overwrite the generated provision/recipes/default.rb with the following. NOTE: It's vital that you set the convergence_options as shown here. Nodes currently don't have any attributes to set Policyfile options; instead you must set policy_group andpolicy_name in the config file. The provisioning convergence options will take care of that for you automatically.

If you'd like to extend this example to use something other than Vagrant, you can learn more about Chef Provisioning on docs.chef.io.

context = ChefDK::ProvisioningData.context    # Set the port dynamically via the command line:  target_port = context.opts.port    with_driver 'vagrant:~/.vagrant.d/boxes' do      options = {      vagrant_options: {        'vm.box' => 'opscode-ubuntu-14.04',        'vm.network' => ":forwarded_port, guest: 80, host: #{target_port}"      },      convergence_options: context.convergence_options    }        machine context.node_name do      machine_options(options)        # This forces a chef run every time, which is sensible for `chef provision`      # use cases.      converge(true)      action(context.action)    end  end  

Notice that we are making the forwarded port configurable via the command line. This lets us run multiple VMs on the same host without the forwarded ports colliding with each other.

We can sync the policy to the server and create our "staging" node with a single command:

chef provision staging --sync -n aar-staging-01 -o port=8000  

That will sync our local policy lock to a policy group called 'staging' (creating that policy group in the process), then run an embedded Chef Client which creates a VM (via Chef Provisioning), configures it and converges it.

We can see the site running by visiting http://localhost:8000 (notice that's port 8000 this time).

Since it works in staging, we'll create a "production" node with the same policy:

chef provision production --sync -n aar-production-01 -o port=8888  

Update the Attributes via Policyfile

Policyfiles allow us to set attributes. Since Policyfiles don't support roles, these attributes replace role attributes in the precedence hierarchy. In our Policyfile.rb, we set attributes using the same syntax we use in cookbooks. In this example, we'll change the version number that appears on the home page. Add the following line to your Policyfile.rb

default['aar']['version'] = "19.7.4"  

To apply the changes to the Policyfile.lock.json, use chef update:

chef update --attributes  

We can see the effect of our changes with chef diff:

chef diff --head  

And we can see that our local policy differs from what we've deployed to our staging group:

chef diff staging  

Now that we're satisfied with our changes, commit to git again:

git commit -a -m 'update aar version'  

Deploy it to Staging:

In a normal TDD workflow, we'd run kitchen again to see our changes, but this time we'll just deploy it to staging by running chef provision again:

chef provision staging --sync -n aar-staging-01 -o port=8000  

If we visit the site at http://localhost:8000 we see "Awesome Appliance v.19.7.4″ right under the login dialog.

Let's suppose we're not ready to apply the change in production, but we want to run chef-client on our production machine. We can do this by using the --policy-name option instead of the --sync option:

chef provision production --policy-name aar -n aar-production-01 -o port=8888  

Note that since we did not update the policy, nothing is updated. If we visit the web page at http://localhost:8888 we'll see that nothing has changed. Though we only changed the attributes, the same is true if we updated the cookbooks, since they're locked down by content in our policy.

Oh No! A Bug in Our Cookbook

To demonstrate how cookbook code is automatically versioned and sandboxed by policyfiles, let's introduce a "bug" into our cookbook. Add this line to the top of cookbooks/aar/recipes/default.rb:

raise "OH NO THIS IS A BUG"  

Since this cookbook is local, chef will automatically pull in updates when we upload (no need to run chef update). Lets upload to staging and run chef-client again. This time, we'll use chef push to upload our changes without invoking provisioning (which is probably what you'd want to do in your normal workflow).

chef push staging  

We can invoke provisioning without syncing the policy like so:

chef provision staging -p aar -n aar-staging-01 -o port=8000  

This will cause an error like this in the chef run on the VM (which will cause another error in chef provision on your workstation):

================================================================================  Recipe Compile Error in /var/chef/cache/cookbooks/aar/recipes/default.rb  ================================================================================    RuntimeError  ------------  OH NO THIS IS A BUG    Cookbook Trace:  ---------------    /var/chef/cache/cookbooks/aar/recipes/default.rb:1:in `from_file'  

But if we run Chef on our production node, everything is roses and sunshine:

chef provision production -p aar -n aar-production-01  # => Chef Client finished, 0/41 resources updated in 10.634577518 seconds  

We can also confirm that the policies applied to each group are different with the show-policy subcommand:

chef show-policy aar  

The output should be similar to:

aar  ===    * production:  8312cd89c9  * staging:     eb0fedf311  

Cleaning Up

To shut down the Vagrant VMs, you can use the -d option to chef provision to set the default action to destroy:

chef provision production --policy-name aar -n aar-production-01 -o port=8888 -d  chef provision staging -p aar -n aar-staging-01 -o port=8000 -d  

Fin

Policyfiles give us a consistent and repeatable description of how we want Chef to configure our machines, with minimal hassle. Because versioning is built-in and automatic at both the cookbook and policy level, we can make changes to our infrastructure code safely and explicitly. While we still have work to do to fill out the feature set around Policyfiles, enough of it exists for you to get started today.

Currently, the most complete documentation is in the Policyfile README in the ChefDK repo, but we'll be moving documentation to docs.chef.io as we complete work on Policyfiles.

If you give it a try and find that any missing feature is a deal breaker for you, let us know and we'll do our best to make you successful.

Happy Cheffing!