Tuesday, May 7, 2013

PuppetDB 1.3

https://puppetlabs.com/blog/puppetdb-1-3/

Tuesday, May 7, 2013 12:04 PMPuppetDB 1.3Puppet LabsMike Hall

Today, we've released version 1.3 of PuppetDB.  This release adds some new features around Puppet reports, and we've got big plans for the next several months. For more details, read on!


PuppetDB is the next-generation open source storage service for Puppet-produced data. With the initial 1.0 release in September 2012, it provided a high-performance system for capturing all of the catalogs and facts for your Puppet nodes. It could be used as a drop-in replacement for the Puppet inventory service, and for the first time gave users a fast and scalable way to take advantage of Puppet storeconfigs and exported resources.

But you already know all of that, because you're already using and loving PuppetDB, right? On the off chance that you're not, take a peek at the "Learn More" section at the end of this document. There you'll find links to an introductory video, blog post, and other info about getting started with PuppetDB.

What's New In PuppetDB 1.3

Extended Report/Event Query API

In PuppetDB version 1.1, we added experimental support for storing your Puppet report data.   PuppetDB 1.1 also included a very basic retrieval API for listing your Puppet reports and collecting all of the events that occurred for an individual report.  Now, in version 1.3, we've added a much more robust query API for interacting with your report data.

Query Events by Timestamp

Often you're only going to be interested in events that occurred during a certain time frame.  Now you can query for them by doing something like this:

 [">", "timestamp", "2013-05-02T12:00:00-03:00"]

This query will return all of the events (across all of the Puppet reports stored in your PuppetDB database) with a timestamp later than noon on May 2.  The results might look something like this:

[       {         "certname": "foo.localhost",         "old-value": "absent",         "property": "ensure",         "timestamp": "2013-05-02T19:01:05.000Z",         "resource-type": "File",         "resource-title": "/tmp/reportingfoo",         "new-value": "file",         "message": "defined content as '{md5}49f68a5c8493ec2c0bf489821c21fc3b'",         "report": "38ff2aef3ffb7800fe85b322280ade2b867c8d27",         "status": "success"       },       {         "certname": "bar.localhost",         "old-value": "absent",         "property": "message",         "timestamp": "2013-05-02T19:01:05.000Z",         "resource-type": "Notify",         "resource-title": "notify, yo",         "new-value": "notify, yo",         "message": "defined 'message' as 'notify, yo'",         "report": "87552aef3ffb7800fe85b322280ade2b867c8d27",         "status": "success"       }     ]

Compound Queries

The timestamp query shown above is a lot more useful if you can specify both a start and and end time for your query. To accomplish this, we now support the same set of compound / boolean query operations for reports / events that we support for querying catalogs and facts.  For example:

   ["and", ["<", "timestamp", "2013-05-02T12:05:00-03:00"],     [">", "timestamp", "2013-05-02T12:00:00-03:00"]]    That would allow you to look at all of the events that occurred within a 5-minute window.

Query by Resource Type/Title/Property

What if you're only interested in events that happened for a specific resource?  You can query for events matching certain resources using something like this:

   ["and", ["=", "resource-type", "Service"],             ["=", "resource-title", "postgresql"],             ["=", "property", "ensure"]]

This will only give you results relating to the Postgresql Service, and specifically only events that relate to the ensure property of that resource.

Query by Event Status

A common scenario that we envision is that you might only be looking for events that had a certain status: e.g., only failed events, or only noop events.  To do that, all you need to do is query against the status field:

["=", "status", "noop"]

Regular Expressions

Equality queries are powerful, but sometimes it's useful to be able to match against a pattern.  This can be really handy when you're querying against host names, so we now support the same types of regular expressions for reports / events as we do for resources:

["~", "certname", "^www.*"]

That'll return events for all nodes whose certnames start with www.

Putting it All Together

We can put together some pretty powerful queries by combining all of the features above.  For example, let's say that you had a known window of time where some bad things happened on your infrastructure.  You'd like to quickly determine all of the services that failed during that time, but you've already taken care of the apache service on all of the nodes, and you've taken care of all of your nodes whose name begins with database–so you're not concerned about those.

["and", ["=", "resource-type", "Service"],         ["=", "status", "failed"],         [">", "timestamp", "2013-05-02T12:00:00-03:00"],         ["<", "timestamp", "2013-05-02T12:05:00-03:00"],         ["not", ["=", "resource-title", "apache"]],         ["not", ["~", "certname", "^database.*"]]]

Voila!  For more info, make sure to check out the complete documentationfor the report / event query API.

Import and Export Catalogs and Reports

In PuppetDB 1.2, we added two new command line tools: puppetdb-import and puppetdb-export.  These tools provide support for importing and exporting catalog data from PuppetDB.  This feature is mostly intended to help users migrate to a new PuppetDB instance or database.

In PuppetDB 1.3, these tools now support importing and exporting report data in addition to catalogs.

After writing these tools to help with data migrations, we realized that they also provide a unique opportunity for users to play a role in improving our products by sharing their real-world data with us. We're constantly looking for new ways to optimize PuppetDB and test new features, and doing so with actual user data would provide us with much greater confidence that our new products are going to work well for our users.  If you'd be willing to share some of your data with us, please contact us! (We'll keep it private. It will only be used for internal testing and performance profiling.) We'll even give you a free t-shirt!

What's Next?

We're working hard to keep cranking out new features and make sure that you consider PuppetDB to be an indispensable part of your Puppet ecosystem. Here are a few things that you can expect to see in the not-too-distant future:

  • PuppetDB bundled with Puppet Enterprise: we intend to deliver an absolutely seamless, dead-simple, out-of-the-box experience for leveraging the power of PuppetDB in Puppet Enterprise environments.
  • Capture data about Puppet modules: in an upcoming release of Puppet core, we'll be adding more information to catalogs about what module (if any) each class or resource was defined in. We'll also capture that data in PuppetDB, and extend the query API to allow you to include module information in your queries.
  • Even more report query functionality: we're looking into options for querying reports based on classes and modules, querying for aggregates (counts, etc.), and several other possibilities.

Tell Us What You Think!

We think that PuppetDB is really cool. But at the end of the day, it's not what we think that keeps the lights on over here at Puppet Labs — it's what you think. So, if you have an opinion on any of our current features or suggestions to help us shape and prioritize features for upcoming releases, we really want to hear from you! Here are some great ways to get in touch with us:

  • Sign up to be a Puppet Test Pilot! You'll get free goodies as a reward for your participation if you are able to volunteer a few minutes of your time to tell us what you think about prototypes of upcoming features. We did a lot of user testing around our query API in the past, and we hope to do a lot more in the future.
  • Ping us on IRC: we're usually online in #puppet on freenode. Just mention 'puppetdb' and you're pretty likely to get a quick response.
  • Send an e-mail to the puppet-users or puppet-dev mailing lists.

Learn More

Here are some good resources for learning about and getting started with PuppetDB:





Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)