Most engagement dashboards report three numbers that make a publisher look better than they are. They count sessions instead of outcomes, they average away the users who left, and they mistake rewarded activity for engagement. None of those is a lie. Each of them is just a measure of something other than what the name on the chart suggests.
The three distortions
The first is counting the session rather than the outcome. A person who opens the app, completes nothing and leaves is a session. It counts in every rate built on top of it, and it tells you nothing about whether anything happened.
The second is averaging away the cohort that churns. A change that lifts the median session while the bottom half of your users leave faster is not an improvement, and the aggregate will not show it, because the aggregate is dominated by the users who stayed.
The third is mistaking rewarded activity for engagement. When the task wall is the product, the completion rate measures how good the offers are, not whether the app is worth opening. This is the distortion with the biggest commercial consequences, because it is the one that hides a product problem behind a working feature.
Why the third one is dangerous
It is easy to raise a completion rate. Make the tasks easier, pay more per task, put fewer of them on the wall. Every one of those changes improves the number on the chart. Two of them also make the product worse.
The tasks get easier and the app gets less valuable, because a person who completes a trivial task has learned nothing and has nothing to come back for. The wall gets fuller and the session gets shorter, because the app now competes with its own inventory for attention. In both cases, completion goes up and retention goes down, and the dashboard shows the first while the revenue shows the second.
The diagnostic is simple: if a completion rate moves and nothing else does, ask which lever moved it. If the answer is that the tasks got easier, the metric went up because the product went down. If the answer is that the offers got better, it went up because the product got better. Those two cases look identical on the chart.
Measure a chain, not a rate
The fix is to stop reporting rates and start reporting a chain: did the person do something, did they get something, and did they come back and do something more demanding. Each link is checkable, and each link tells you where the problem is.
Link one is the action - did a session end in a completed task, or in an abandoned wall. Link two is the reward - did the credit land, how long after the action, and how often was it reversed. Link three is the return - did they come back, and was the thing they came back for harder than the thing they did last time.
This structure also catches the fraud, because the middle link is where an unverified completion shows up: the reward side of the chain grows while the revenue side never arrives. A dashboard that reports only completions cannot see that. A dashboard that reports the chain sees it immediately.
What to instrument instead
Then instrument the return, by cohort, with the shape of the curve rather than the average of it. Day-one, day-seven and day-thirty retention for a cohort tells you how fast it decays. The ratio between those numbers tells you how long it will last, which is the number that determines whether the acquisition cost was ever going to be recovered.
The cohort discipline is what makes this useful rather than merely correct. Averages across all users move for reasons that have nothing to do with what you changed. Cohorts do not.
The honest version of a dashboard
An honest engagement dashboard has fewer numbers than the one most publishers have, and each of them is a behaviour. Sessions that ended in something. Rewards that landed on time. Reversals. Return rates by cohort. Completion rates of attempted tasks. Session length as a secondary figure rather than a headline one.
That list is shorter and it is harder to move in the wrong direction without a product change, which is the point. A metric that can be improved by making the product worse is not a metric. It is a lever wearing a label, and the label is what gets it into a board deck.
The final test of any engagement number is simple: if this number went up, would the business be better? If the honest answer is that it might not be, the number is measuring activity, and activity is not engagement.
Related reading
Run rewards in your app
Tell us the platform and roughly how many daily active users you have, and we will tell you whether rewarded games, surveys or both are the better fit.
Questions
What is the most reliable engagement metric for a reward app?
The share of sessions that end in a completed action, tracked by cohort. It describes a behaviour, it cannot be improved by making the product worse, and it moves before the revenue does. Session length is the least reliable, because it measures how long the phone was in the foreground rather than what the person did.
Why does my completion rate go up when the app gets worse?
Because completion rate measures how good the offers are, not whether the app is worth opening. Making the tasks easier or the wall fuller raises the number while lowering the value of the product. The diagnostic: if a completion rate moves and nothing else does, ask which lever moved it - easier tasks mean the metric went up because the product went down.
What should I measure in a reward app instead of session length?
The completion rate of attempted tasks, the share of sessions ending in a completed action, reward latency between the action and the credit, the ratio of reversed credits to total credits, and return rates by cohort. Each of those describes a behaviour and can only be improved by changing the product.
How do I know my engagement numbers are not flattering me?
Measure a chain rather than a rate: did the person act, did they get rewarded, did they come back and do something more demanding. That structure exposes both distortions - counting sessions that end in nothing, and averaging away the cohort that churns - and it surfaces unverified completions, which inflate the reward side of the chain while the revenue never arrives.