March 12, 2018

Dr. Nancy Potok

Office of Management and Budget

Executive Office of the President

725 17th Street Northwest

Washington, D.C. 20503

Re: OMB Request for Information on Data Linkage Activities

Dear Dr. Potok:

On behalf of the American Evaluation Association, we applaud the Office of Management and Budget's plans to advance the analysis of agency records in order to enhance understanding of government programs and policies. We also appreciate the opportunity to provide feedback on OMB's Request for Information on issues pertaining to combining data from multiple sources (Federal Register, 83(9): 1634).

As the American Evaluation Association said in a November 1, 2017, statement about the Commission on Evidence-Based Policymaking's recommendations: "The Commission's recommendations represent a nonpartisan approach to making greater use of data the government already collects, improving data privacy protections, and strengthening the government's capacity to generate the evidence needed to support policy decisions." We noted that implementation of those recommendations would require additional attention to details, and appreciate OMB's request for additional information about improving data linkage activities.

Combining government's existing data on program participation with data on outcomes, such as population health or income, can provide valuable information that addresses a range of research and evaluation questions. Combining data sources can also save time and resources compared to collecting new data on such variables. However, because data access and linkage are often time consuming and resource intensive, we expect that the opportunities for such analyses will initially be limited, and thus require careful planning to ensure their value and usefulness.

As OMB continues to pursue increased uses of administrative data for statistical purposes, we recommend four planning activities that could enhance the success and usefulness of these efforts in response to OMB's request in the RFI for "suggestions for additional research in those or related areas:"

1. Solicit and consider a wide range of agency, congressional, and researcher input on

the most pressing government program and policy questions to address through the use of existing data.

To provide credible, useful insight into program operations and effectiveness, data and

analyses should address identified information needs. While beginning the data linkage

project with the most readily accessible data may be cost efficient, new linkage activities

may delay investment in the data preparation needed to answer the most pressing policy

questions. Both OMB, in the fiscal year 2019 budget proposal, and the Commission on

Evidence-Based Policymaking (Recommendation 5-1) have suggested that "learning

agendas" and the creation of chief evaluation offices (Recommendation 5-2) may be

productive vehicles for supporting and articulating such agency needs and encouraging

longer-term planning that may identify potential priorities for linkages.

2. Develop guidelines and provide incentives for thorough agency documentation of

agency administrative records that are being considered for future data linkage.

The Commission on Evidence-Based Policymaking recommended that OMB improve its

efforts to make data documentation and metadata available (Recommendation 4-5); we

agree. Poor existing documentation and metadata of some agency databases at the level required by researchers and evaluators is a frequent and serious barrier to the generation and subsequent use of high-quality evidence in policymaking. Efforts to improve the documentation and reliability of agency data will require additional resources and staff expertise and should be weighed against the expected value of the analyses they promise to support.

OMB's 2014 memorandum to federal agencies entitled "Guidance for Providing and

Using Administrative Data for Statistical Purposes" (M-14-06) provided a promising first

step from OMB in encouraging greater use of data already collected by government,

including guidance to update Statement of Record Notices (SORNs) when needed to

encourage linking or combining data. Additional guidance from OMB to agencies,

including about the use of existing authorities available under the Paperwork Reduction

Act or other relevant laws, will be critical for continued progress in improving

documentation and metadata standards.

3. Engage agencies with less evaluation experience to explore realistic opportunities for

them to engage in these efforts and provide them with planning and analytical assistance. Evaluation capacity varies widely among federal agencies, and so, too, does the technical capability to combine administrative data for research and evaluation. Today there are many important information gaps about the relevance, efficiency, and effectiveness of federal programs and policies. Current data linkage efforts have sometimes improved evaluation cost-efficiency for agencies that are already experienced with conducting such studies; expanding technical capabilities in other agencies may provide a valuable opportunity for data linkage to expand analyses in less frequently explored policy areas.

4. Study the cost-effectiveness of data linkage approaches.

Because of the intensive resource implications for combining data across agencies,

including the potential for substantial cost savings in research and evaluation, we

encourage OMB to support efforts to specifically examine the cost-effectiveness of such

approaches.

The American Evaluation Association is a professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The American Evaluation Association has approximately 7,400 members representing all 50 states in the United States and the District of Columbia, as well as over 80 countries.

The American Evaluation Association appreciates the opportunity to provide suggestions to OMB as it undertakes this important set of activities. We look forward to being a resource for OMB staff in the future as OMB undertakes the task of implementing the vision and recommendations of the Commission on Evidence-Based Policymaking.

If we can be of assistance, or if you need more information on our organization, please do not hesitate to contact me ( President2018@eval.org) or Dr. Cheryl Oros, senior advisor for evaluation policy ( EvaluationPolicy@eval.org).

Sincerely,

Leslie Goodyear, PhD., President

Kathryn Newcomer, PhD., Past President

Tessie Catsambas, MPP, President-Elect