“Without data, you’re just a person with an opinion.”<\/p>\n
Those are the words of W. Edwards Deming, the champion of statistical process control, who was credited as one of the inspirations for what became known as the Japanese post-war economic miracle of 1950 to 1960. Ironically, Japanese manufacturers like Toyota were far more receptive to Deming\u2019s ideas than General Motors and Ford were.<\/p>\n
Community management is certainly an art. It\u2019s about mentoring. It\u2019s about having difficult conversations with people who are hurting the community. It\u2019s about negotiation and compromise. It\u2019s about interacting with other communities. It\u2019s about making connections. In the words of Red Hat\u2019s Diane Mueller, it\u2019s about “nurturing conversations.”<\/p>\n
However, it\u2019s also about metrics and data.<\/p>\n
Some have much in common with software development projects more broadly. Others are more specific to the management of the community itself. I think of deciding what<\/em> to measure and how<\/em> as adhering to five principles.<\/p>\n In 2008, Daniel Ariely published Predictably Irrational<\/em>, one of a number of books written around that time that introduced behavioral psychology and behavioral economics to the general public. One memorable quote from that book is the following: \u201cHuman beings adjust behavior based on the metrics they\u2019re held against. Anything you measure will impel a person to optimize his score on that metric. What you measure is what you\u2019ll get. Period.\u201d<\/p>\n This shouldn\u2019t be surprising. It\u2019s a finding that\u2019s been repeatedly confirmed by research. It should also be familiar to just about anyone with business experience. It\u2019s certainly not news to anyone in sales management, for example. Base sales reps\u2019 (or their managers\u2019) bonuses solely on revenue, and they\u2019ll try to discount whatever it takes to maximize revenue, even if it puts margin in the toilet. Conversely, want the sales force to push a new product line\u2014which will probably take extra effort\u2014but skip the spiffs? Probably not happening.<\/p>\n And lest you think I\u2019m unfairly picking on sales, this behavior is pervasive, all the way up to the CEO, as Ariely describes in a 2010 Harvard Business Review<\/em> article: \u201cCEOs care about stock value because that\u2019s how we measure them. If we want to change what they care about, we should change what we measure.\u201d<\/p>\n Developers and other community members are not immune.<\/p>\n There\u2019s a lot of folk wisdom floating around about what\u2019s relevant and important that\u2019s not necessarily true. My colleague Dave Neary offers an example from baseball<\/a>: \u201cIn the late ’90s, the key measurements that were used to measure batter skill were RBI (runs batted in) and batting average (how often a player got on base with a hit, divided by the number of at-bats). The Oakland A\u2019s were the first major league team to recruit based on a different measurement of player performance: on-base percentage. This measures how often they get to first base, regardless of how it happens.\u201d<\/p>\n Indeed, the whole revolution of sabermetrics in baseball and elsewhere, which was popularized in Michael Lewis\u2019 Moneyball,<\/em> often gets talked about in terms of introducing data in a field that historically was more about gut feel and personal experience. But it was also about taking a game that had actually always been fairly numbers-obsessed and coming up with new metrics based on mostly existing data to better measure player value. (The data revolution going on in sports today is more about collecting much more data through video and other means than was previously available.)<\/p>\n As a corollary, collecting lots of tangential but easy-to-capture data isn\u2019t better than just selecting a few measurements you\u2019ve determined are genuinely useful. In a world where online behavior can be tracked with great granularity and displayed in colorful dashboards, it\u2019s tempting to be distracted by sheer data volume, even when it doesn\u2019t deliver any great insight into community health and trajectory.<\/p>\n This may seem like an obvious point: Why measure something that isn\u2019t relevant? In practice, metrics often get chosen because they\u2019re easy to measure, not because they\u2019re particularly useful. They tend to be more about inputs than outputs: The number of developers. The number of forum posts. The number of commits. Collectively, measures like this often get called vanity metrics<\/em>. They\u2019re ubiquitous, but most people involved with community management don\u2019t think much of them.<\/p>\n Number of downloads<\/em> may be the worst of the bunch. It\u2019s true that, at some level, they\u2019re an indication of interest in a project. That\u2019s something. But it\u2019s sufficiently distant from actively using the project, much less engaging with the project deeply, that it\u2019s hard to view downloads as a very useful number.<\/p>\n Is there any harm in these vanity metrics? Yes, to the degree that you start thinking that they\u2019re something to base action on. Probably more seriously, stakeholders like company management or industry observers can come to see them as meaningful indicators of project health.<\/p>\n Neary makes this point to caution against myopia. \u201cIn one project I worked on,\u201d he says, \u201dsome people were concerned about a recent spike in the number of bug reports coming in because it seemed like the project must have serious quality issues to resolve. However, when we looked at the numbers, it turned out that many of the bugs were coming in because a large company had recently started using the project. The increase in bug reports was actually a proxy for a big influx of new users, which was a good thing.\u201d<\/p>\n In practice, you often have<\/em> to measure through proxies. This isn\u2019t an inherent problem, but the further you get between what you want to measure and what you\u2019re actually measuring, the harder it is to connect the dots. It\u2019s fine to track progress in closing bugs, writing code, and adding new features. However, those don\u2019t necessarily correlate with how happy users are or whether the project is doing a good job of working towards its long-term objectives, whatever those may be.<\/p>\n Some measurements may be non-obvious but useful for tracking the success of a project and community relative to internal goals. Others may be better suited for a press release or other external consumption. For example, as a community manager, you may really care about the number of meetups, mentoring sessions, and virtual briefings your community has held over the past three months. But it\u2019s the number of contributions and contributors that are more likely to grab the headlines. You probably care about those too. But maybe not as much, depending upon your current priorities.<\/p>\n Still, other measurements may relate to the goals of any sponsoring organizations. The measurements most relevant for projects tied to commercial products are likely to be different from pure community efforts.<\/p>\n Because communities differ and goals differ, it\u2019s not possible to simply compile a metrics checklist, but here are some ideas to think about:<\/p>\n Consider qualitative metrics in addition to quantitative ones. Conducting surveys and other studies can be time-consuming, especially if they\u2019re rigorous enough to yield better-than-anecdotal data. It also requires rigor to construct studies so that they can be used to track changes over time. In other words, it\u2019s a lot easier to measure quantitative contributor activity than it is to suss out if the community members are happier about their participation today than they were a year ago. However, given the importance of culture to the health of a community, measuring it in a systematic way can be a worthwhile exercise.<\/p>\n Breadth of community, including how many are unaffiliated with commercial entities, is important for many projects. The greater the breadth, the greater the potential leverage of the open source development process. It can also be instructive to see how companies and individuals are contributing. Projects can be explicitly designed to better accommodate casual contributors.<\/p>\n Are new contributors able to have an impact, or are they ignored? How long does it take for code contributions to get committed? How long does it take for a reported bug to be fixed or otherwise responded to? If they asked a question in a forum, did anyone answer them? In other words, are you letting contributors contribute?<\/p>\n Advancement within the project is also an important metric. Mikeal Rogers of the Node.js community<\/a> explains: \u201cThe shift that we made was to create a support system and an education system to take a user and turn them into a contributor, first at a very low level, and educate them to bring them into the committer pool and eventually into the maintainer pool. The end result of this is that we have a wide range of skill sets. Rather than trying to attract phenomenal developers, we\u2019re creating new phenomenal developers.\u201d<\/p>\n Whatever metrics you choose, don\u2019t forget why you made them metrics in the first place. I find a helpful question to ask is: \u201cWhat am I going to do with this number?\u201d If the answer is to just put it in a report or in a press release, that\u2019s not a great answer. Metrics should be measurements that tell you either that you\u2019re on the right path or that you need to take specific actions to course-correct.<\/p>\n For this reason, Stormy Peters, who handles community leads at Red Hat, argues for keeping it simple<\/a>. She writes, \u201cIt\u2019s much better to have one or two key metrics than to worry about all the possible metrics. You can capture all the possible metrics, but as a project, you should focus on moving one. It\u2019s also better to have a simple metric that correlates directly to something in the real world than a metric that is a complicated formula or ration between multiple things. As project members make decisions, you want them to be able to intuitively feel whether or not it will affect the project\u2019s key metric in the right direction.\u201d<\/p>\n1. Recognize that behaviors aren’t independent of the measurements you choose to highlight.<\/h2>\n
2. You need to choose relevant metrics.<\/h2>\n
3. Quantity may not lead to quality.<\/h2>\n
4. Understand what measurements really mean and how they relate to each other.<\/h2>\n
5. Different measurements serve different purposes.<\/h2>\n