When developing a data system to help track and evaluate public engagement, it’s important to make sure the information or data characteristics align with the measurement needs.  Depending on the purpose of the measurement, there will likely be a clear preference for quantitative or qualitative detail in the data.  The importance of other information or measurement characteristics such as comprehensiveness, consistency across programs or over time, and validation/verification may be less apparent.  Here, I briefly describe each of these feature of measurement systems and present their relevance for each of the measurement and evaluation purposes outlined in the “why measure engagement” discussions found here and here.

Information Characteristics

Quantitative information.  Quantitative information includes data that can be expressed as a number or quantity.   These are often referred to as metrics.

Qualitative information.  Qualitative information is descriptive.  It can be structured or unstructured; closed ended or open ended; captured through discussions (e.g. interviews or focus groups), observations, or submitted reports.

Consistent across programs, program components, or units within a portfolio. Information within a system is consistent across programs when the same types of information are collected on all programs.

It’s relatively easy to collect consistent information across programs and portfolios with a standardized model.  For those with more varied components, for some needs it may be a priority to find some unit of information that can be standardized widely and applicable to all for purposes of aggregation or comparison.

Consistent over time.  There are three primary dimensions to information consistency over time:  that the same information types are collected; that they are collected through the same channels or methods using the same definitions, and that the same reporting sources (e.g. units, programs) are included in the collection.

Comprehensive.   Information within a system is comprehensive when it covers all aspects of a program or of an organization’s portfolio of programs.

Comprehensive data coverage can be an issue for programs and portfolios with many components, since some of these may be more difficult to track and measure.  For most purposes, making sure to have data from all programs is useful or important but not essential, but for some, complete data coverage is needed.

Validated/verified.  Information is validated or verified when it has checks to make sure it is complete and correct.  Typically, information can be validated through rigorous analysis, triangulation from different sources, external review, or documentation.

Choosing the applicable level of validation helps to use resources effectively.

Mapping your M&E purposes to the information characteristics you need

• An essential characteristic is one that is needed to achieve the purpose—it can’t be done without it.
• An important characteristic is one that is needed, but the purpose is achievable even if the data don’t fully meet the characteristic.
• Useful characteristics are ones that are helpful to have, but the purpose is achievable even if they aren’t met.