Lessons Learned in Evaluation

Working across a wide range of clients and evaluation types offers a unique advantage—the opportunity to apply lessons learned in one context to another. Although each grant program and project presents its own distinct goals and challenges, successful evaluations often share common characteristics. Over time, certain principles and practices consistently emerge as key to meaningful, effective evaluation. Here are some of the most valuable lessons we've learned along the way:

1. Engage Stakeholders Early—Especially Those Involved in Implementation and Data Collection

One of the most valuable steps in any evaluation is collaborating with stakeholders who are directly involved in program implementation or data collection. While project directors often oversee planning, they may not be the ones executing the work or gathering data. By involving them early, project directors and evaluators gain insight into what’s feasible and what may need adjustment. These leaders, embedded in the daily operations of their schools or departments, can identify who has the capacity to support implementation and data efforts—insights that are often missed without their input.

2. Be Strategic and Selective with Data Collection

More data isn’t always better. One of the most important lessons in evaluation is the value of being intentional about what data to collect—and why. Project directors should focus on gathering data that directly supports understanding of implementation progress, while also being mindful of the burden placed on both data collectors and respondents (such as survey participants).

A helpful starting point is to review what data are already being collected. Leveraging existing sources can reduce duplication, streamline efforts, and ensure that new data collection is purposeful and manageable.

3. Prioritize Ongoing Communication Throughout the Project

Consistent communication is essential for a successful evaluation. Project directors should regularly connect with those responsible for implementation to understand how things are progressing, identify emerging needs, and ensure staff have the time and resources to carry out their roles effectively. These check-ins—whether through recurring meetings or informal in-person visits—build trust and foster a collaborative environment.

Equally important is maintaining open lines of communication between the evaluator and the project director. Regular meetings not only provide valuable context about implementation but also create space for reflection, adaptation, and strategic decision-making. These conversations often lead to insights about additional data needs or necessary pivots in the evaluation approach.

Evaluation is as much about relationships and strategy as it is about data. Across diverse projects and grant programs, three lessons consistently stand out: the importance of stakeholder collaboration, the need for intentional and manageable data collection, and the value of ongoing communication. When these elements are prioritized, evaluations become more responsive, grounded, and impactful—ultimately supporting programs in achieving their goals more effectively.

Interested in working with Shaffer Evaluation Group? Contact us today for a free 30-minute consultation: seg@shafferevaluation.com.

Sustainability Planning for Federal Grants: From Closeout to Continuity

Sustainability planning for federal grants is a disciplined process. It ensures a project’s goals, principles, and activities endure after the award period ends. Federal funders increasingly prioritize long-term impact. Proposals and performance reports should demonstrate a credible plan for lasting success. In this article, I will share guidance on sustainability planning to support your federal grant.

Start Early

Treat sustainability as a design constraint from day one. Don’t wait until the final quarter to address it. Build time, roles, and budget for sustainability work into your management plan. Revisit this plan at every reporting cycle to ensure you stay on track.

Use a Sustainability Assessment

At Shaffer Evaluation Group, we guide clients using the Program Sustainability Assessment Tool (PSAT). The PSAT helps teams score and prioritize capacity across eight domains: Environmental Support, Funding Stability, Partnerships, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, and Strategic Planning. Conducting a quick PSAT baseline early in Year 1, a midpoint check, and a final assessment creates a clear trajectory of capacity-building for your organization. This also provides evidence of progress for funders.

Build a Sustainability Plan

Your sustainability plan should be developed early in your grant term. Here are the core components a sustainability plan for a federally funded project should include, distilled from cross-agency best practices:

• Purpose & Outcomes: A concise statement of your project's value proposition, intended outcomes, and beneficiaries, aligned with a logic model you will continue to monitor and adjust post-award.

• Core Program Model & Operating Protocols: A description of your core program model and what must be maintained (including workflows) so the project can persist with quality after grant closeout.

• Governance & Ownership: Named post-award owner(s) and ongoing “champions” to anchor the work.

• Funding Strategy (Braided Financing): A targeted mix of reallocated (internal) funding and external sources, with milestones, responsible leads, and cultivation plans (e.g., philanthropy, fee-for-service).

• Partnerships & MOUs: Roles, cost-shares, and data-sharing agreements with agencies, nonprofits, and community partners that are essential to sustain activities.

• Workforce Continuity & Knowledge Transfer: Staffing map, cross-training to avoid single points of failure, onboarding materials, and a schedule for refresher trainings.

• Communications & Stakeholder Engagement: A plan to keep sponsors, end-users, and other stakeholders informed and invested.

• Data, Evaluation & Learning: Post-grant metrics, feedback loops, and routines for using evidence in decisions.

• Policy & Institutionalization: The specific policies, procedures, and standards that will embed the work into normal operations.

• Risk & Scenario Planning: Anticipated risks (funding gaps, turnover, vendor change), trigger points, contingencies, and “minimum viable operations” to protect continuity.

• Technology & Infrastructure: Post-award requirements for platforms, equipment, licensing, accessibility/privacy, and support so services can run at a steady state.

  
  

Operationalize It

Translate the plan into a 12–24 month roadmap. Include owners, milestones, and a lightweight dashboard. Budget explicitly for sustainability tasks, such as grant writing time, partnership stewardship, and training refreshers.

How We Help

Shaffer Evaluation Group uses the PSAT to run a rapid diagnostic. We co-facilitate strategy sessions with leadership and partners. Together, we co-develop a sustainability roadmap that balances quick wins (policy changes, SOPs, cross-training) with longer-horizon plays (braided funding, institutionalization). A thoughtful sustainability plan signals stewardship, strengthens proposals, and, most importantly, keeps effective work alive for the communities it was designed to serve.

For more information, contact us today at seg@shafferevaluation.com.

Key Considerations for Evaluating Career and Technical Education (CTE) Programs

Shaffer Evaluation Group partners with a wide range of Career and Technical Education (CTE) programs supported by various grant initiatives, including the Department of Defense Education Activity (DoDEA), the Native American Career and Technical Education Program (NACTEP), and the National Science Foundation (NSF).

When designing an evaluation strategy for CTE programs, consider the following two key factors:

Capture the Full Range of Program Impact

Outcome measures should reflect both short-term achievements and long-term impacts. While traditional metrics like completion rates, graduation rates, and post-program plans are essential, they only tell part of the story. To gain a deeper understanding of program effectiveness, collect data throughout the program’s duration. This might include:

• Surveys that track changes in participants’ beliefs, attitudes, and dispositions

• Measures of academic performance and engagement

• Insights into future educational or career plans

• Enrollment in additional training, educational programs, or courses

These broader indicators provide a more complete picture of how the program shapes participants’ trajectories.

Account for Attrition in Your Evaluation Design

Attrition is a reality in many CTE programs, so it’s crucial to build evaluation measures that reflect progress, even when participants don’t complete the full program. Consider tracking the following:

• Milestones toward program completion

• Course-level completion rates (e.g., percentage of students who finish individual courses)

• Endorsement or certification attainment

These metrics can demonstrate early wins and incremental progress, offering valuable insights into program effectiveness even when full completion isn’t achieved.

Final Thoughts about Evaluating Career and Technical Education Programs

Effective evaluation of CTE programs requires a thoughtful and flexible approach that captures both the breadth and depth of program impact. By incorporating diverse outcome measures and planning for participant attrition, evaluators can build a more accurate and resilient picture of program success. Whether you're working with DoDEA, NACTEP, NSF, or other grant-funded initiatives, these strategies help ensure your evaluation reflects the real-world progress and potential of your CTE efforts.

---

Shaffer Evaluation Group is committed to helping programs tell their story through meaningful data. With the right tools and insights, your evaluation can do more than meet reporting requirements—it can illuminate the transformative power of CTE. Contact us today for a free 30-minute consultation.