A community of technology community managers, leaders, and builders.

CLS16 Notes: Metrics unconference session

Hi everyone,

Thanks a lot for this awesome CLS meeting :).

I’m attaching the notes from the metrics discussion we had on Saturday. Thanks a lot for the people that attended. We had some great discussion.

Kudos to Jackson from New Relic that took the notes that you can see. I’ve edited a bit them to give some format. I’ve also removed any reference to companies (not open source projects) as this may be sensitive information. If you feel this is valuable information we can add this later.

Comments and extra discussion is really welcome :).


Topic about Metrics (Initial discussion and needs and roles introduction)

  • Measuring community and what does measuring community really mean?

  • Share some experience - two or three roles around the topic of metrics - what is the purpose of measurement? “I am good at my job, and this is how I’m good at doing my job”

  • Specific needs like - people are coming, and w need to say hello and “thanks for coming.” “it is great that you provided X”

  • after a while you might have people who are leaving the community and why they are leaving. Getting them interacting after they have left

  • Daniel is focused on open source communities - so please share other things you’d like to share

  • they are interested in aggregation and measuring these items.

  • All of the standard metrics - email lists, stack overflow, attendance,

  • Engineering roles - metrics could be useful so how they are performing - how fast are we –are we talking to specific volunteers, and and are we saying hell and providing them with numbers about time for review about them vs community - which came from the wikimedia foundation that really cares about volunteers - and they are continually doing hackathons, wikimedia meetings, and academia events. They needed metrics for volunteers and.

  • Metrics for human resources - need for hiring people, but no idea where to look for them, and where is the best place(s) - aggregate the best places to hire, such as stack overflow, glassdoor, etc

Why are you measuring things? (Discussion about the reasons why the attendees were measuring things)

  • Ben Lloy - interested in seeing how they perform to the rest of the industry. How do they contribute to the rest of his company, and how they compare to the rest of that company, along with the impact of their efforts on open source and that company.

  • People mostly focused on quantitative metrics, not qualitative

  • Some discussions and ideas:

  • Understanding how many companies there are around

  • Business vs community, and to make sure everything is fair (eg code review process).

  • Measuring things that don’t matter in order to make a case for things that do.

  • working with the community and measuring things that don’t

  • Examples of metrics “how may downloads, replies to blog posts, etc”

  • Measure in numbers their community impact -

  • Manrique - Many metrics are based on the amount of people in a community - joining, leaving the community, and and how to see the flow as they flow through the community. How to make the path though a contribution - and how to make the path as short as possible.

  • Sometimes you are measuring too much, and how to remove the metrics that don’t matter, as well as how to come down to the low-hanging fruit

Some discussion

  • Amanda

  • number of lines in slack, twitter mentions, members, news letters, blogs,

  • mostly concentrated on engagement

  • Davey: community is mostly php, and some do not know about the offerings that we have. Most people that I speak to are long-term ‘[investments]’

  • Amanda - we aren’t tied to revenue either, and the second that you get associated with indirect revenue is the minute that you kill organic growth

  • Ben - we regularly publish technical articles on the items were are working on. We recently had a post about our key technologies we posted a change that is a controversial change - other websites picked it up and that was valuable - and that’s valuable about engagement.

  • Davey - Do you track negative vs positive connotations on engagement?

  • Clay - Tying metrics to revenue, especially in B2B companies - I get the impression that measurement will increasing in pressure to track revenue based on engagement

  • Davey - API calls are maybe a basic metric to evaluate how people use the product

  • Alexandra - You can use a user funnel and see how people flow through your website and downloads or trials and tracking how people

  • Jason Yee - Metric about No one is going to click a tweet and decide to blow $1500.

  • Going from a developer and decision maker.

  • Doing A/B testing could be based on those who are engagement with the community - are they spending more money that people not engaged with the community?

  • What sort of metrics? Sometimes there aren’t = correlations can be measured, but correlations cannot be tied to causation.

  • Rikky

  • we do a lot of metrics concerning the site, contribution,s page views. documentation about the event, ROI - who we talked to, takeaways from sessions, rather than what you may be able to capture with metrics

  • Stephen Walli - Conflation between community and product revenue. - trying to tie the cost of community marketing, and tying it to the business value.

  • Can community mangers articulate the difference between items in the open source arena or

  • Value must be only monetary -

  • Churn retention and acquisition - for community, not customers.

  • Upper management - “one metric that matters’

  • be able to track lower things so you can get more information.

  • Reporting on one item, which aligns most with the business goal.

  • Ale Murray - Apache cassandra

  • We have a very established community after 2.5 years of working on that.

  • the more people that use the open source will eventually fall into the pipeline of the revenue streams

  • growth metrics - how many people attended an event, downloads, etc

  • once the community has hit scale and is self-sustaining - now trying to measure the success of those who are using the product

  • ex: how many users stories - how many forks, etc

  • Jyee - metric around use - looking at top skills on linked, top 10 database skills

  • “how many interactions happen on stack overflow”

  • forking of forks become a ripple effect of engagement - which becomes harder to measure

  • what is the impact of may company on this particular project?

  • If someone is investing in cassandra that is outside of cassandra - it means that people want to add value to the project

  • Are people using qualitative vs quantitative data to evaluate their efforts?

  • foundation efforts - see how contributes to the foundation, and see who asks questions about the project on the mailing list, for example - Company X asks questions, and then joins the foundation 3 months later.

  • Why do we have this peak of questions - why do I have this peak of of pull requests -

  • Then you can dig in about KPIs and how metrics are evaluated

  • For some projects, we need to have 6 months before ”the project is dead.” in some projects, they may be 1 year vs 6 months.

  • Problems - using metrics to find people to hire

  • going to the community itself is a non-starter

  • Analyzing the community around the item to evaluate the environment and how people are using metrics at a given company - using pull requests that are good to be able to see who is accepted in the community

  • Having real numbers is different from what you might be able to find form an interview

  • Having a look at the community and how they use for instance the API allow us to improve the documentation.

Some challenges

Challenge - how to get metrics about how people improve their skills around community

  • how the community is improving he collaborative support if they don’t work around repositories
  • value of the community, which is hard to track because they don’t have common infrastructure.
  • it is difficult to connect to some activities with the training of the community, with the usage of the community, and how they support each other in a collaborative way.

Challenge - measuring impact on a very extended timeframe. How do you measure impact of in person interactions and an eventual move to buy?

Some tooling

Thanks to for being the kind sponsor for this forum!