Address feedback about the overview board

[jon-betts]

We have had the following feedback about the board:

About CAGR

• The CAGR numbers were expected to be a comparison from the numbers in one month to the same month from a year before
  • I believe the CAGR / CMGR calculations are correct, but they are not what was expected
  • We can also deliver a "this month last year" report and see which we like the best
  • More descriptions on reports might help with this

I've added a this month last year comparison and added more descriptions on reports explaining them. I think we might want to revisit the CAGR / CMGR calculation as it's based on whole years, so I suspect it will go massively negative in January. Should this be based on a trailing two year period?

Annotating users in H looks too small compared with active LMS users (solved?)

• The number of "active" LMS users this month, and the number of "annotating" H users looks out of whack
• We do count activity in LMS and annotating in H differently - This could be correct?
• We could be counting different months?
• It could just be very bugged?

So far this just looks correct - You can't directly compare annotating H users with and active LMS users

Incorrect last month (solved?)

• The "last whole active month" doesn't look quite right. It should have changed to November here but doesn't seem to have
  • Could this be the cause of some of the discrepancies above?

Our date ranges weren't quite correct. I've set these to both the "last two months", but I have a feeling Metabase might mess this up throughout the month. We'll need to keep an eye and see what happens.

[jon-betts]

About: "Annotating users in H looks too small compared with active LMS users"

According to:

SELECT 
    DATE_TRUNC('month', created_week) as month,
    COUNT(DISTINCT(user_id))
FROM report.user_activity
WHERE annotation_count > 0
GROUP BY DATE_TRUNC('month', created_week)
ORDER BY month DESC;

LMS US annotating users:

• October: 60,039
• November: 38,623

LMS CA annotating users:

• October: 402
• November: 331

Total:

• October: 60441
• November: 38954

According to the authority activity report we have (for LMS):

• October: 60039 (US)
• November: 38623 (CA)

If we get the raw user launch counts:

SELECT 
    DATE_TRUNC('month', timestamp_week) AS month,
    COUNT(DISTINCT(user_id))
FROM report.events
WHERE 
    event_type = 'configured_launch'
GROUP BY DATE_TRUNC('month', timestamp_week) 
ORDER BY month DESC

We see:

• October: 78,782
• November: 54,793

According to the organization activity report we have:

• October: 79,573
• November: 55,310

It doesn't surprise me that these are a bit higher, as the same user can be in different orgs and get double counted.

Whats the main story here then?

I think this is just because we're counting different things:

• About 70% of our annotating users in the last month were from LMS
• An LMS user is "active" if they login, regardless of whether they annotate
• It just so happens only about 70% of our LMS users that login annotate in a given month
• The end result is the two numbers are similar, because we are counting different things

If we want to see the number of annotating users (rather than active) from LMS you can see that in the authority graph.

What actions to take:

• We could base the active LMS users in the last month on the direct events, instead of the org activity - This would reduce double counting

[jon-betts]

Nup... this was all a bad query I wrote for debugging. The numbers line up

[jon-betts]

I think we're done here for now barring getting some confirmation, which Daryl has agreed to do.