Chef and DevOps at JustGiving [feedly]

Chef and DevOps at JustGiving
https://www.chef.io/blog/2015/08/18/chef-and-devops-at-justgiving/

-- via my feedly.com reader

JustGiving is the world's largest fundraising platform. In the last twelve months, JustGiving has helped tens of millions of people raise over $700,000,000 dollars. 

This post describes how JustGiving uses Chef and DevOps to help manage its infrastructure and create a flexible, cooperative work environment. Contributors were Richard Atkinson, the Chief Information Officer, Owain Perry, Software Architect, and Mark Chamberlain, Head of Services.

JustGiving is 15 years old. Historically, it has used Windows but, over the last few years, it has also begun to use Linux. There have been other changes as well. Originally, the company used managed data centers, but they are moving towards an architectural pattern of microservices that run on Amazon Web Services. New services are immediately deployed to the cloud while the legacy codebase is gradually being transformed to follow this new pattern.

The company began using Chef at about the same time that they began to adopt the microservices approach. Currently, Chef automates a large percentage of JustGiving's cloud-based web operations. In addition, their continuous delivery pipeline for deploying microservices is configured and managed with Chef. They also use Chef for cloud orchestration—servers in the cloud are managed using Chef resources.

They use hundreds of cookbooks. Each microservice has its own cookbook, and there are others for running the infrastructure. For example, JustGiving has created a cookbook for provisioning cloud instances that uses heavyweight resource providers (HWRPs). JustGiving also uses some community cookbooks.

The company has adopted DevOps principles, both social and technological. Originally, the Operations team was separated from the Development team but, gradually, barriers between teams came down, and they began to collaborate. The team that writes the infrastructure code is now co-located with the ServiceOps team.

The ServiceOps team is responsible for running all production services. In close collaboration with the Development team, they maintain a backlog of tasks, and use a quasi sprint-planning process that produces two-week cycles. They use JIRA to manage the process.

There are a variety of other teams as well, each with specific skills such as front-end development, backend development, quality assurance, user experience design, and architecture.

Richard Atkinson says,

"The teams come together in 'swarms,' which have all the skills they need to deliver to business objectives, and include nontechnical skills such as marketing. Everyone sits on the same floor. We also have blameless post mortems. They are vital to self-improvement after a failure or an outage."

Members of the teams are free to work from home or anywhere around the office. Everyone has a high-spec laptop and there is a range of tools to help foster cohesion, even when working remotely. For example, JustGiving uses JIRA, HipChat and cloud repositories in GitHub.

JustGiving has a continuous deployment pipeline that is used for both infrastructure and application code. They use TeamCity as the build server and Go to manage the pipeline. The development team can use the pipeline to provision their own infrastructure as they need to.

Owain Perry described the workflow as code moved from a developer's laptop to production.

"When a developer has a requirement for a new service, they copy a template Git repository, fill in the blanks, which includes a Chef cookbook for the service, create (clone) the relevant pipelines in TeamCity and Go and commit their hello world application."

They then clone the infrastructure Git repository and define the infrastructure that's required in a data-driven document and a Chef recipe, and then push the change. The document and recipe will build their infrastructure all the way into production.

From this point onwards, developers can create code on their laptops and push changes. The code is compiled and unit tested in TeamCity; a versioned package is stored in Artifactory; and a versioned cookbook is stored in Chef.

The Go pipeline can deploy that version of the service through our environments into production. After each deployment into an environment, including production, post-deployment tests are executed. If the tests are happy, the environment is switched from inactive to active (blue/green) to turn the code on.

Richard Atkinson discussed the effect Chef has had on JustGiving.

"As well as speeding up our deployment cycle such that we're releasing code almost constantly, there have been situations where we have turned around hotfixes within an hour of spotting symptoms. We have seen service resilience improve as services are removed from the production environment to be replaced by new instances. The deployment process is less disruptive to end users, with blue/green deployments keeping service capacity online throughout."

Mark Chamberlain added,

"We spin up environments regularly due to demand spikes. We also use Chef to power down/up our staging and development environments when they aren't being used overnight, producing reasonable savings. We have pre-approved configurations available, and we have moved on to baking full machine images, using Chef, that enter our configuration and deployment process. We see this as an area for development. We feed the Chef configuration and change logs into our environment, which is an ELK (Elasticsearch, Kibana) stack. We have microservice and cookbook changes automated in our staging and production environments so that there is a central change JIRA ticket for every change in these environments."

Looking toward the future, Richard is excited at the prospects.

"What I see today with DevOps reminds me of my early career where I saw the rise of concurrent engineering methodologies in aerospace organizations, where production expertise was being extended upstream to ensure products were being designed for factors other than utility, such as manufacturability.

"I think there are interesting parallels from history, too. Ask anyone who invented the car and most people would mention names like Karl Benz or Henry Ford, but actually the functional principles were invented 200 years earlier by Verbiest. What Messrs. Benz and Ford did was make the car a reality for the masses by removing blockers to economic scale and performance. Where am I going with this? I believe computing has not yet had its Ford Model T moment, but history will show that now is the defining era when the conditions were created, through solutions like Chef, for IT to go mainstream. That's cloud. That's mobility. That's DevOps. Although we may think computing has been done, the best is yet to come."