Photo by Patrick Fore on Unsplash
Collect information in the easiest way possible
The goal of a measurement or evaluation data collection system is to get complete and valid information that addresses information goals while minimizing the reporting and program burden. Even when you have a set of data questions that perfectly addresses your information needs in principle, you may fall short of your information goals if the system makes it hard to get complete and valid data. While it’s important to seek the right data—the data that can provide the information you need to make a decision or take action—it’s equally important to be realistic about the level of difficulty you are likely to face in getting it. How hard will it be for data reporters to enter the information into the system? And how hard will it be for the data management side to develop reporting system and work with the data in the form they need? Minimizing difficulty for both sides is critically important to ensure quality data.
How data are affected by difficult or cumbersome collection systems
When data systems are hard for reporters to use, data quality suffers in two dimensions. First, data may be incomplete, as reporters may simply not use the systems to report any information. If they do use it, they may skip questions or simply stop when they reach a question they can’t interpret or that requires too much effort.
Second, if the system is complex or difficult to navigate, the validity of the data—the alignment of the intended information to what is understood by the reporters—is also likely to be impacted. Reporters may interpret and answer a question in a variety of ways. They may be uncertain about exactly what is wanted, or what qualifies to be counted or included, but simply guess and move on.
In addition to these data quality challenges resulting from cumbersome data collection systems, there can also be spillover effects on the work environment or the relationship between the reporter and the program or data manager. The more work a system imposes on reporters, the more they will feel it is unreasonable, which may translate into resentment and resistance.
In addition, any reporting that is difficult for reporters will turn out to be difficult on the data management side, requiring substantial ongoing resources to train, provide guidance, nudge reporters, and quality check the data. Reviewing and cleaning the data will likely involve guiding and nudging once again as part of the follow up process. These “care and feeding” activities are not impossible, but they can be difficult to carry out and can represent a big drain on time and staff resources that is both hard to estimate and hard to carry out.
How to make systems easier to use
In order to obtain complete and valid data in the most sustainable way (minimizing drain on program management resources), there are two main strategies.
- Use the easiest applicable type of data source to get valid and complete data while minimizing start up and ongoing time needs. Existing systems are always easier both for reporting and for data management and use.
- For new and modified reporting systems, choose data collection attributes to minimize reporter burden to get the most valid and complete data and minimize ongoing time needs.
