Notes from the Lab: Community Cookbooks Survey Results [feedly]

Notes from the Lab: Community Cookbooks Survey Results
https://www.chef.io/blog/2015/08/17/notes-from-the-lab-community-cookbooks-survey-results/

-- via my feedly.com reader

In the final post in our series on the Community Cookbooks Survey, we turn it over to Nicole to report the results. To catch up on how the project began and the science behind it, see these posts here and here.

Ohai Chef community! I'm very excited to tell you all about the community cookbooks survey and what the data tells us. But before I get into an analysis of the results, I'll talk about how we asked people to take the survey, otherwise known as sampling. For this study, we used a snowball sampling method. Basically, we announced the survey broadly at ChefConf and over social media, and asked people to retweet and share the survey, so that the group of respondents grew like a snowball.

Some of you may have taken a statistics class and are now possibly shaking your head. "But what about sampling?!" We did do a form of sampling, but couldn't do randomized or stratified sampling methods for this survey, and here's why. In order to do these types of sampling, you need to know your population: who they are and how to find them. While we can define our population of users (i.e., Chef community cookbook users), we can't exactly find our population of users. Think about it: this is a two-tiered challenge. Because we have an open source software solution that offers an open source resource, we don't have a central repository or list of all Chef users (one possible population of users) or all community cookbook users (another possible population). Without access to those populations to sample from, we used snowball sampling. While this sampling method may not be ideal, it was the best method we had available to us. (As an aside, there are plenty of examples where data collected through snowball sampling is used in top-tier peer-review research.)

Demographics

Of the 278 surveys started, 210 were completed. (This is more than enough for the analyses conducted, and enough for peer-review.)

The reported age of the respondents ranged from 21 to 59, with an average age of 34. Respondents reported working in their current position an average of 5 years, ranging from 0 to 26 years. Once outliers were removed, respondents reported working with Chef an average of 3 years, with a range from 0 to 7 years. 96% of the respondents were male, 2% female, and 2% did not disclose their gender. While this appears male-centric, it is fairly consistent with sysadmin demographics reported in other studies (e.g., 2008 SAGE salary survey reported 86.6% men, 2006 SAGE reported 96.1% men).

Our respondents report using Linux OS the most, at 96%, followed by Windows at 19%, and other at 5%. Note that these are not mutually exclusive, as our users can work with multiple operating systems.

Analysis

Now, on to the analysis! If you recall from the last post, the constructs in the survey included:

• Perceived ease of use
• Perceived usefulness
• Intention to use cookbooks
• Use of cookbooks
• Use of Chef

We also included control variables, like age, gender, and years of experience with configuration management.

Following analysis and verification of the constructs (with methods like factor analysis), I used structural equation modeling (SEM) to analyze the model. The statistical software package was Stata 13.1.

First, I'll present the model in a diagram, since I think it really helps to visualize what we were trying to analyze. Note that significant paths in the model are a solid line, and non-significant paths are a dotted line. You can read each arrow as a predictive path.

The diagram shows us that:

• Usefulness of cookbooks is predictive of intention to use cookbooks and cookbook use

• Cookbooks' ease of use (EOU) is not directly predictive of intention to use cookbooks and cookbook use.

• However, EOU does indirectly contribute to use through its impact on usefulness.

• The relationship between cookbook use and Chef use is unclear – they seem to be the same thing for the users who took the survey.
• Other factors such as age, gender, or years of experience do not impact use of cookbooks.

The analysis shows that usefulness of cookbooks impacts intention to use cookbooks. But what does "useful" mean here? In this analysis, useful cookbooks are ones that help our survey respondents:

• improve their ability to make good decisions
• get their work done more quickly
• enhance their effectiveness on the job

These three things are key factors in deciding to use a community cookbook.

Now on to EoU. You'll notice that in the diagram, the arrow from "EoU" to "Intention to Use Cookbooks" is dotted. TAM (the theory we are referencing) suggests that EOU is predictive of intention to use a technology. This is true in many cases, but does not appear to be directly important in the case of community cookbooks. Why might this be? Perhaps users don't expect implementing code for infrastructure management to be easy to do, but they do expect that the effort will provide them with useful functionality. Also cookbooks EoU can impact cookbooks perceived usefulness because they can help users get their work done more quickly and/or enhance their effectiveness on the job. In this way, EOU still matters to one's intention to use cookbooks, but only indirectly through perceived usefulness.

Another point that doesn't show up directly in the graphic but did appear in the preliminary data analysis is that use of community cookbooks and use of Chef are tightly linked in users' minds. How do I know this? Use of Chef and use of community cookbooks loaded together and were highly correlated. Because of this correlation, I only included use of cookbooks in the rest of my analysis. When we think about it, this makes sense: those who took the survey are community cookbook users, and they probably don't use Chef without using cookbooks.

When I included the control variables in the model, they were not significant. This was very interesting to me, because quite often we see differences between genders or among those with differing levels of experience.

Summary

Regardless of demographics, the survey shows us that:

• Users don't consider using Chef, the platform, as being distinct from using Chef community cookbooks.
• Users care about how useful a cookbook is. If the cookbook makes them better at their jobs, it's useful and fuels their intentions to use more cookbooks.
• Users aren't directly affected in their intentions to use cookbooks by how easy cookbooks are to use. However, because EoU indirectly affects how well users perform at their jobs, it indirectly affects their intention to use cookbooks.

When we evaluate our own efforts to clean up existing cookbooks or create new ones, we should do so by making those cookbooks more "useful."