Tools for Program Design

By Dr. Patricia Moore Shaffer

Theory of change and logic modeling are tools that program managers and evaluators often use to describe a program’s design or, simply put, how an educational program is supposed to work. A theory of change allows us to understand the underlying hypotheses of a program by illustrating the program’s outcomes pathway – the activities understood to produce a series of results that contribute to achieving the desired change. A logic model, meanwhile, gives more clarity about the components (i.e., inputs) that need to be in place for the program to work; it also includes visual depictions of how the planned activities will result in specific outputs or products, and outcomes.

At Shaffer Evaluation Group, we often do theory of change and logic modeling exercises in the early phases of an evaluation or capacity-building project to help program stakeholders articulate how their program works. When we worked with Valencia College evaluating its Art of Tomorrow Scholars program, we engaged a broad cross-section of stakeholders, including staff, beneficiaries, partners, and funders, in the development of these models. In addition to producing valuable tools to guide program design, development, and evaluation, stakeholders participating in this process increase their knowledge and understanding of the program. Very often this process leads stakeholders to a fresh perspective on their programs that may lead to changes in program design.

When we engage stakeholders in developing a theory of change or logic model for a program, we keep in mind principles of adult learning to make the experience meaningful and engaging for everyone involved:

• Adults want or need to be involved in how our learning experiences are planned and delivered. We consult with stakeholders about how to organize program model development work. Do they want to organize a day-long retreat or a series of short meetings with interspersed independent group work?

• Adults draw upon our own knowledge and experience when learning. Stakeholders have rich knowledge of how their program works in their community – we facilitate discussion and use other methods of engagement to tap into this knowledge during model development work.

• Adults are active learners, preferring to use reasoning and collaboration to find solutions. We actively engage stakeholders in model development – assigning backwards mapping exercises or small group projects to identify program inputs, outputs, or outcomes.

• Adults want learning activities to be immediately applicable to our lives or work. We take the time to demonstrate how theories and models can be applied to improve programs’ effectiveness.

One resource we use is ActKnowledge’s Facilitator Source Book for Leading Theory of Change Development Sessions, which provides guidance on leading collaborative sessions to craft long-term outcomes and use “backwards mapping” to identify the earliest changes that need to occur. The Logic Model Workshop Toolkit, prepared by the REL Northeast & Islands, provides guidance for facilitating a logic model workshop; resources include a facilitator workbook, a participant workbook, and a slide deck.

The Importance of Education Policy Research

By Dr. Marquita Hockaday

Consider that omnipresent, yet pertinent quote from historians: “It’s important to study history so we don’t repeat the mistakes of the past?” The saying is close to that (Caveat: I did major in history as an undergraduate AND I taught history in grades 6-12 for over eight years, so I really should know this line well…) Whether those are the exact words or not, the sentiment behind the quote is what matters, and that sentiment remains when it comes to studying and understanding education policy research.

First, what is education policy? According to Ballotpedia (2021), education policies are the plans and principles put in place to educate students. Since education has existed in our nation, the goals of education policies have included citizenship, literacy, preparation for college and the workforce, preparation for the global market, diversity, helping students to become critical thinkers, and creating common standards for students across the nation. When reviewing education policy, it is important that some questions are considered--such as what are we teaching and how is that instruction being evaluated? What standards are we using to measure academic success?

Shaffer Evaluation Group (SEG) provides an evaluation service where we assist organizations with answering the question of how instruction (or other outputs, such as assessments and teacher training programs) is being evaluated. Additionally, SEG assists organizations with answering the question of what standards of measurement might be used to measure success. For instance, an organization wants to determine how well they are doing with implementing a new education policy initiative related to COVID-19. The school or district may be working on integrating engaging virtual lessons and reach out to SEG regarding a non-experimental study to gauge the fidelity of their existing programs. SEG’s data collection would include interviews, focus groups, and possible quantitative data (e.g., surveys).

Educators should take the time to research, review, and understand existing policy and its impact on the current state of educational practice. Additionally, it’s important to evaluate the fidelity to which these policies are put into practice. More information about how SEG can assist with program evaluation can be found here.

Standing with the AAPI Community: Supporting Communities of Color Against Hate

By Dr. Marquita Hockaday

For several months, numerous instances of racism, discrimination, and overall violence against the Asian-American and Pacific Islander (AAPI) community has continued in America. These hate crimes include name-calling, shunning (such as refusal of service), workplace discrimination, and random assaults. Hate crimes against the AAPI community increased early in 2020 at the first signs of the coronavirus pandemic in America. Some politicians and activists have stated that hate crimes were exacerbated by former president Donald J. Trump and his strong language toward Chinese individuals and their role in spreading COVID-19.

AAPI individuals who have reported being victims of hate crimes have stated that they were spat on, coughed on, told to go back to their country, and personally blamed for spreading the pandemic. These individual violent crimes against the AAPI community came to head on March 16th, 2021 in Atlanta, GA. A gunman shot and killed eight victims at three different massage parlors, six of those victims were of AAPI descent. While some media reports have claimed this was hate crime and others have stated “it is too soon to tell” whether this is a hate crime or not (based on the shooter having a “fetish for AAPI women”), the sexualization of the six AAPI victims at the massage parlor is a form of racism. This is a demonstration of how society needs to educate themselves on implicit and explicit biases.

Unfortunately, this past year has seen a rise in xenophobic expressions (such as “China virus” for COVID-19), and violence towards the AAPI community, so much so that President Joe Biden signed an executive order right after taking office in January denouncing anti-Asian discrimination.

Shaffer Evaluation Group (SEG) follows the American Evaluation Association’s guiding principles, which addresses competence, integrity, respect for people, common good, and equity. SEG intends to uphold these principles, including honoring dignity, well-being, and self-worth of individuals, as well as acknowledging the influence of culture within and across groups. It is our goal as a company to hold ourselves and others accountable by revealing systems and structures of bias (both implicit and explicit) to stop xenophobia and racism and stand with the AAPI community. We are working toward a future where all people of color can feel safe in public spaces and are not prejudged based on their outward appearance.