The metrics that social sector technology reports most reliably are the ones easiest to collect: time-on-device, completion rates, downloads, active users. These are real numbers. They are not meaningless. But they measure engagement, not outcomes, and in a sector where the goal is learning, health, or economic change rather than engagement, they can actively mislead.
An organization optimizing for time-on-device can produce a product where learners stay on the screen longer without learning more. One optimizing for completion rates can produce a product where completing a module is easier than skipping it. One optimizing for active users can show growth while reaching shallower engagement with each new cohort.
The organizations doing this well have separated engagement metrics from outcome metrics, built data architecture that can capture the difference, and designed data collection in ways that serve the frontline workers holding the devices.
The difference between engagement and outcomes
Engagement metrics capture behavior: did the user show up, for how long, how often. They are useful for understanding reach and retention. They are not useful for understanding impact.
Outcome metrics capture change: did the learner understand the concept, did the patient follow the protocol correctly, did the farmer apply the advisory. Measuring these requires a different kind of data, collected at a different point in the interaction, requiring interpretation rather than simple logging.
You can log time-on-device passively. You cannot log productive struggle passively. Outcome data requires designed data collection: interactions intentionally constructed to generate evidence of understanding rather than evidence of presence.
The distinction matters architecturally because outcomes require designed data collection. You log time-on-device by recording session start and end. You cannot log whether a learner understood a concept the same way. You have to design an interaction that generates evidence of understanding, collect that evidence in structured form, and process it in ways that distinguish genuine comprehension from correct guessing.
The productive struggle principle
In learning science, "productive struggle" describes the cognitive state where a learner is challenged enough to be working hard but not so challenged as to disengage. It's the zone where learning actually happens, and it is the zone hardest to measure.
Products that optimize for engagement tend to minimize struggle. Content is calibrated to be completable rather than challenging. Learners click through because clicking through is easy, not because they're learning. The product's metrics look strong. The learning outcomes may not be.
Products that optimize for learning outcomes look different: they present problems before explaining solutions, use spaced repetition to surface content at moments of near-forgetting, and use diagnostic tools to identify specific gaps rather than aggregate performance scores.
Designing for productive struggle changes what you collect. You're not logging whether a learner finished the module. You're logging where errors occur, how many attempts a concept requires, which material needed repeated exposure before understanding showed up. This data is richer and more complex to analyze, and it's the data that tells you whether the product is actually working.
Avoid vanity metrics. Focus on whether learners are encountering the right level of challenge and whether that challenge is producing measurable change. A learner who struggles and succeeds has learned something. A learner who completes easily may not have.
Systemic trust: data collection that serves frontline workers
There's a principle that determines whether field data is trustworthy: the people collecting it need to benefit from collecting it.
Community health workers who fill in forms to satisfy donor reporting requirements and never see the output of those forms will fill them in incorrectly, incompletely, or not at all. Educators who log student assessments for program monitoring and never receive feedback from those assessments will treat logging as compliance, not as useful work. The data that results reflects the conditions under which it was collected, which is to say it reflects obligation rather than accuracy.
Dimagi's CommCare platform, deployed with frontline health workers across 70-plus countries, succeeds in part because it was designed around this principle. The platform gives workers structured decision support in real time: the data they enter immediately informs what the system shows them next. Data collection serves them first. Program monitoring and donor reporting are downstream outputs of the same data, not a separate parallel reporting burden.
Qure.ai's TB screening tool in tribal communities in Chhattisgarh operates on the same logic. A community health worker without radiology training enters a chest X-ray into the system and receives a probability score with an explainability overlay. The output serves her decision (refer or not refer) immediately. The same data feeds surveillance systems and program reporting. The worker benefits from the system. The system benefits from her accuracy.
Designing data flows this way requires a deliberate choice early in the architecture: who is the primary consumer of this data? If the answer is the program team and the funder, you will get data that is unreliable and incomplete. If the answer is the person holding the device, you will get data that is accurate and rich, and the program team and funder will benefit from that accuracy downstream.
Building outcome-oriented data architecture
Outcome-oriented data architecture has a few structural characteristics worth designing in from the start.
Collect at the point of interaction, not after the fact. Retrospective reporting is less accurate, less complete, and less useful than real-time collection. If data entry happens hours or days after the interaction it's meant to capture, accuracy degrades and completeness suffers. The architecture should make real-time collection the path of least resistance.
Keep collection lightweight for the collector. Every additional field in a form is a potential point of error or omission. The architecture should collect the minimum data that enables the desired outcome insight, in ways that feel natural within the workflow. If entering data takes longer than doing the work it's meant to document, the data will not be entered accurately.
Build a feedback loop from data to collector. This is the mechanism that builds trust and maintains data quality over time. When frontline workers see that the data they enter produces useful guidance for them, data quality improves without requiring external monitoring. The feedback loop is not a reporting feature; it's a data quality infrastructure choice.
Separate outcome data from engagement data in the reporting layer. This allows different audiences to see what's relevant to them: program coordinators see learning or health outcomes, funders see aggregate impact, frontline workers see their own caseload or classroom data. The underlying data can be the same; the reporting layer presents it differently. This separation prevents the engagement metrics from drowning out the outcome metrics in funder reports.
Three questions for product builders and funders
1. What would a frontline worker in your program say they get out of the data they collect? If the honest answer is "nothing, it goes to the program team," that's the data quality problem before the data quality problem. Fix the feedback loop before worrying about the dashboard.
2. Can your current system distinguish between a learner who completed a module quickly because they understood it and one who completed it quickly because they guessed? If not, your completion data is measuring something other than learning, and optimizing for it may be making the learning outcomes worse.
3. If you removed all engagement metrics from your next funder report and replaced them with outcome metrics, what would you need to build to make that possible? The gap between what you can currently report and what you'd need to report to actually demonstrate impact is the data architecture gap worth closing.