A community of technology community managers, leaders, and builders.

Engagement metrics, ROI and data science (CLS15 session)


Yesterday afternoon at CSL there was a great discussion around engagement, metrics, value, and tools. The session was actually a merger of three separately proposed sessions that had mutually compatible/overlapping contexts. Probably 40-50 people in the room.

For me one of the best takeaways was the idea of looking at engagement as more or less a marketing funnel with metrics around the different levels and activities to move people between the different levels. As with most things you want to measure, you should know what you are trying to achieve. If you don’t you won’t get the value out of the metrics. Metrics drive behavior so you run the risk of driving the wrong behavior (e.g., gamification) if you measure the wrong thing. Of course, if you can use this to your advantage if you know what you are trying to achieve.

The discussion around on-ramps was interesting. Measuring time to first contribution is instructive as it talks about how inviting the community and technology is, how well it is documented, how easy it actually is, … Relating this back to the funnel, it talks about moving people from level 1 (early level) to level 2. Measuring how many people move, how long it takes and defining the activities that we can do to make it easier/faster/more likely for folks to transition.

A number of tools for gathering insights were suggested

  • Google analytics
  • Metrics inherent in the mechanism/system being used (e.g., github metrics, forum metrics, …)
  • Support interaction measurements (number of questions, time to first response, number of interations, time to resolution, number solved, …)
  • Adobe Analytics
    ** Feedback loop to the people answering (let them know that they are helping)
  • Stack overflow/stack exchange
  • New relic insights
  • MetricsGrimoire – http://metricsgrimoire.github.io/ measure oss involvement
  • Discourse for forums gives good metrics, gamification
  • Sentiment analysis

General sentiment in the room was that this is by no means a solved problem and most people/communities are still trying to figure out how to measure, manage, value engagement. There is lots of different kinds of data, lots of actual data, communities are very different with different goals, … http://communitybuilding.stackexchange.com/ is a forum for community building.


I’m adding my notes from the session:

What defines value of the community?

  • The sum of its engagements
  • List item
    • Purpose: achieve $GOAL from sum total of all engagements.
    • What are the outputs of engagement?
    • What kinds of engagement?
    • How do you measure these?

Outputs of engagement
Goal of measuring: Creating active, engaged, proficient contributors/users

  • social media
    • tagging members/friends in posts
  • transitive engagement
  • aggregating things in reports
    • in some FLOSS communities this does not matter
  • How do you want communities/users to engage?
    • Basic marketing concepts
      • facilitating community engagement, behaviors you want to see
    • Measuring non-contributor engagement
      • cognitive engagement, emotional engagement, use

Goal: Don’t be creepy

  • Effectiveness metrics
    • How long between people posting to responses
    • How long between ticket creation and ticket closure
    • PR to merge
    • TIme to get started, how easy is it to get started
      • e.x. “17 page document, very detailed”
      • “Time to ‘Hello World’”
      • Are there diverse onramps?
    • tools used
      • google analytics
      • cobbling together social media interactions
      • data sources
      • Grimoire
      • Discourse
      • Insights
      • open source forums
      • Adobe Analytics

For Support Based Communities

  • Askers/Answerers
    • people who ask usually only ask
    • people who answer usually only answer
    • rarely people move categories
  • Useful metrics
    • number of questions asked
    • number of questions solved
    • time between these
    • provide personal interactions on impact, positive response time, etc
    • Documentation
      • How many questions turn up that are not in the documentation?
      • Use filter that shows if link to documentation in thread

Our Impact

  • Can we affect metrics?
    • Should we?
    • Do meetups or hackathons have more impact?
      • do either have any impact?

Take Aways

  • People want to know more about tools
  • This is a popular subtopic for community people
  • We need places to talk about these problems
    • Create a CLF subforum?
    • Use the slack channel for community managers
  • Common KPIs
    • time to user engagement
  • You need a goal
    • Why do you want a particular kind of engagement
  • A lot of people are figuring these things out, continually, constantly
  • What matters to your project? Is there one thing? Do you need to keep changing this?
  • “Take care of what you measure, or you will become what you are measuring.”
  • Metrics should come from what you are trying to achieve
  • Start from your goals
    • Think about the goals of individuals within your community

Thanks to for being the kind sponsor for this forum!