Creating Results-Oriented Programs (CROP)

Syllabus for Course 94-824 at The H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University taught by instructor Matt Hannigan

Course Overview

Course Description

Successful programs require more than just great ideas. From a program’s initial planning and design through its execution and evaluation, program managers must deftly deploy disparate skills and work closely with colleagues, board members, funders, and constituents to measure, understand, and discuss programmatic results.

In Creating Results-Oriented Programs, graduate students are introduced to tools and techniques to assist them in managing highly effective programs at small nonprofit and arts organizations. Students explore the connections between program theory, project budgets, requests for support, and reports on outcomes. Students develop skills needed to build measures for success into program activities and use quantitative and qualitative assessments to make strategic decisions before, during, and after implementation.

Learning Objectives

The learning objectives of this course are to enable students to:

1. Articulate programmatic goals through a compelling theory of change;
2. Apply the basic techniques of process and outcome evaluations;
3. Gather and interpret relevant and useful program data;
4. Provide tangible and specific information about program results to internal and external stakeholders; and,
5. Bring an analytic and pragmatic approach to each stage in the lifecycle of a typical program at a small nonprofit or arts organization.

Prerequisites

None

Course Framework

Meeting Format

This graduate course will progress over a period of 6 or 7 sessions and will feature a mixture of lecture content, discussion of assigned reading materials, group exercises, and student presentations.

Each class session will be divided into 80-minute segments with a 10-minute break in between. Slides to accompany each lecture will be made available prior to the beginning of each class.

Readings

There are no required textbooks for this course. For each class session, students will be required to read a selection of chapters, articles, reports, or other materials as PDF files to download or web links to visit.

Assigned readings during 6-session editions of the course average 85 pages of material per session (low: 60 pages, high: 116 pages) and will require, on average, 2.5 hours of preparation prior to class. Assigned readings during 7-session editions of the course average 68 pages of material per session (low: 60 pages, high: 102 pages) and will require, on average, 2 hours of preparation prior to class.

Requirements

In-class Group Exercises

Student will complete 3 in-class group exercises during the course. Students will be randomly assigned to groups and asked to work together through a set of questions or activities related to that class session’s assigned readings or lecture content. Each student group will report out the key points of discussion to the entire class. Each group will also submit worksheets that record responses to questions and summarize the group’s dialogue with the names of each participating student. The group exercises will be graded as pass/fail. Students with unexcused absences for a given class session will automatically receive a failing grade on that group exercise.

Take-home Individual Assignments

Students will complete 2 take-home assignments during the course. Students must complete each assignment on their own. The assignments are structured so as to contribute toward the development of the student’s final class project (see below). The take-home assignments will be graded as pass/fail. Students failing to turn in an assignment by its due date without gaining prior permission from the Instructor will automatically receive a failing grade on the assignment.

Final Project

Students will complete a summative final project as a key aspect of the course. Through the project, students will demonstrate their proficient use and understanding of concepts, ideas, and techniques developed during the course.

The final project requires students to design, plan, and prepare for evaluation a new project or program of their own choosing based on areas in their own interests or professional expertise.

It includes 2 separately-evaluated outputs:

• Electronic deliverable, including executive summary, resubmissions of take-home assignments, and other components
• In-class presentation with supporting slides

Additional details regarding the final class project will be discussed during Week 2 of the course. The final project will be graded using numeric scores.

Assessment & Evaluation

The Instructor will evaluate each student’s progress toward achieving the course’s learning objectives through completion of the course requirements, according to the following weights:

• 33.3% In-Class Group Exercises & Take-home Individual Assignments
• 66.7% Final Project Electronic Deliverable & Presentation

Aggregate performance will determine the overall final grade assigned at the conclusion of the course using the following grading scale:

The grade of A+ is reserved for truly exceptional performance. The median grade in the course is typically a B+ or A-.

At the Instructor’s discretion, a student’s final overall score may be rounded up 0.5% because of a student’s high level of engagement in the course as indicated by frequent in-class participation and demonstrated comprehension of assigned reading materials. For example, an engaged student with a final course score of 93.5% (A-) may have their grade rounded up to an A.

Instructor Information

Matt Hannigan was a co-founder of The Sprout Fund and served as its Deputy Director & Vice President until its sunset in 2018. The Sprout Fund was Pittsburgh’s leading agency supporting innovative ideas, catalyzing community change, and making our region a better place to live, work, play, and raise a family. Matt helped start Sprout in 2001 while receiving his MSPPM degree from Heinz College. Matt was involved in the creation and implementation of all of Sprout’s major programs, which provided catalytic support for more than 1,500 community-based people, projects, and ideas through nearly $8 million of investments.

Matt provided oversight for Sprout’s principal program areas, actively participated in fundraising, program development, and planning, and maintained primary responsibility for overseeing the finances and business operations of the organization. Through his work at Sprout, Matt developed a keen interest in decisionmaking processes and fostering civic innovation through grantmaking.

From 2005 to 2011, Matt was an adjunct teaching assistant for the program evaluation course (90-823) at Heinz College. Creating Result-Oriented Programs was created to provide the key concepts of program evaluation and other insights from a decade of work in Pittsburgh’s nonprofit community in a half-semester setting. It has been offered continuously, twice each year, since the 2011–12 academic year.

Course Schedule

Summary Course Calendar

Week 1: Introductions / Course Overview / Program Entrepreneurs

As an introduction to the course, the first session details the course framework, requirements, expectations, and learning objectives. Students, viewed as future program entrepreneurs at small nonprofit and arts organizations, are introduced to the program design, strategy, and evaluation as adaptable techniques in the service, delivery, and reporting of charitable activities in today’s nonprofit environment.

This session’s optional readings are:

• J. Gregory Dees, “Mastering the Art of Innovation” in Enterprising Nonprofits: A Toolkit for Social Entrepreneurs, edited by J. Gregory Dees, Jed Emerson, & Peter Economy (New York: John Wiley & Sons, 2001), pp. 161–197. [37 pages]
• Mike Hudson, “Managing Performance” in Managing at the Leading Edge: New Challenges in Non-profit Management, 1st edition (San Francisco, CA: Jossey-Bass, 2005), pp. 41–80. [40 pages]

Week 2: Developing Program Theory & Logic Models

The second class session introduces theory of change and its attendant tool, the program logic model, as essential component of a program entrepreneur’s toolkit for testing and developing new approaches to community challenges. With a clear articulation of program theory, preparing for implementation and planning for evaluation are made easier.

This sessions assigned readings are:

• Carol H. Weiss, “Understanding the Program” in Evaluation: Methods for Studying Programs and Policies, 2nd edition (Upper Saddle River, NJ: Prentice Hall, 1996), pp. 46–71. [26 pages]
• W.K. Kellogg Foundation, Chapters 1–3 in Logic Model Development Guide (Battle Creek, MI: W.K. Kellogg Foundation, 2004), pp. 1–34. [34 pages]

During the first half of class, the Instructor will introduce the Final Project. During the second half of the class session, students will work together in small groups on Exercise #1 (Logic Model). At the end of class, the Instructor will discuss Take-home Assignment 1 (Problem/Opportunity Definition), which students will complete individually for the next class.

Week 3: Establishing Measures of Success

Measures of success and the importance of having defensible metrics in the current funding climate is the principal topic of the third class session. Distinguishing between outputs, which are used to monitor a program’s process and implementation, and outcomes, which are used to indicate a program’s effect and impact, establishes realistic expectations for program participants and supporters.

This sessions assigned readings are:

• Carol H. Weiss, “Developing Measures” in Evaluation: Methods for Studying Programs and Policies, 2nd edition (Upper Saddle River, NJ: Prentice Hall, 1996), pp. 114–151. [38 pages]
• Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman, “Assessing and Monitoring Program Process” in Evaluation, 7th edition (Thousand Oaks, CA: SAGE Publications, 2004), pp. 169–202. [34 pages]
• Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman, “Measuring and Monitoring Program Outcomes” in Evaluation, 7th edition (Thousand Oaks, CA: SAGE Publications, 2004), pp. 203–232. [30 pages]

Students will digitally submit Assignment 1 (Problem/Opportunity Definition) prior to the start of class. During the second half of the class session, students will work together in small groups on Exercise 2 (Outputs vs. Outcomes). At the end of class, the Instructor will discuss Assignment 2 (Measures of Success), which students will complete individually for the next class.

Week 4: Designing Experiments & Understanding Threats to Validity

Designing experiments to gauge program impact and test hypotheses requires an understanding of the core threats to validity, randomization techniques, and the strengths and weaknesses of various design approaches. After selecting an experimental design strategy, surveys, interview questions, and existing datasets are effective, low-threshold mechanisms for program managers and evaluators to gather relevant data on program implementation and outcomes.

This session’s assigned readings are:

• William R. Shadish, Thomas D. Cook, Donald T. Campbell, “Experiments and Generalized Causal Inference” in Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Boston, MA: Houghton Mifflin Company, 2002), pp. 1-32. [32 pages]
• William R. Shadish, Thomas D. Cook, Donald T. Campbell, “Randomized Experiments: Rationale, Designs, and Conditions Conducive to Doing Them” in Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Boston, MA: Houghton Mifflin Company, 2002), pp. 246-278. [33 pages]

Students will digitally submit Assignment 2 Measures of Success prior to the start of class. During the second half of the class session, students will work together in small groups on Exercise 3 (Experimental Design).

Week 5 of 6 / Weeks 5 of 7 & 6 of 7: Gathering Data, Communicating Results & Securing Resources to Lead Change

After selecting an experimental design strategy, surveys, interview questions, and existing datasets are effective, low-threshold mechanisms for program managers and evaluators to gather relevant data on program implementation and outcomes. To provide timely information to decisionmakers, program managers use data visualization tools such as dashboards, diagrams, and infographics to distill details into actionable steps to continue, change, or modify a program.

After a program is complete, new techniques in documentation and reporting, such as multimedia storytelling, are among the most resonant ways to demonstrate impact to supporters and the public. Topics and ideas previously discussed in the course will be revisited within the specific context of securing organizational support to advance innovative ideas, seeking resources to design and implement new projects and activities, and managing relationships with major funders and early backers.

These sessions’ assigned readings are:

• Marc Bolan, Kimberly Francis, and Jane Reisman, “Data Management Planning and Preparation” in How to Manage and Analyze Data for Outcome-Based Evaluation (Seattle, WA: The Evaluation Forum, 2000), pp. 7-32. [26 pages]
• Art Feinglass, “Setting Goals, Clarifying the Message, and Creating the PR Plan” in The Public Relations Handbook for Nonprofits (San Francisco: Jossey-Bass, 2005), pp. 21–45. [25 pages]
• Harold S. Williams, Arthur Y. Webb, and William J. Phillips, Part III, Chapters 12–15 “Targets and Accomplishments” in Outcome Funding: A New Approach to Targeted Grantmaking, 4th edition (Rensselaerville, NY: Rensselaerville Institute, 1996), pp. 113–178. [65 pages]

Students will digitally submit Assignment 3 Evaluation Approach prior to the start of class. At the beginning of week 5, the Instructor will review key concepts from Sessions 2–4 of the course and students may work together in teams on an additional group exercise. During 7-sessions editions of the course, during the first half of the class session in Week 6, guest(s) from Pittsburgh’s nonprofit and arts community will discuss their successes in creating results-oriented programs.

Last Week: Final Project Deadlines

Students must submit their Final Project Deliverable by the deadline listed on the course Canvas webpage. The document(s) must be submitted as Microsoft Word or Adobe PDF files.

During the final class session, students will provide oral summaries of their Final Project Deliverable for the Instructor and their classmates. Students must submit the slides for their Final Project Presentations digitally by 5pm ET on the day of class.

Course Policies & Procedures

Expectations

Students should expect that the Instructor will

• Plan each class session with care so that it is worthwhile to attend;
• Encourage all students to participate in class discussions;
• Start and end class session on time;
• Make lecture outlines available no later than 1 hour before the beginning of a class session;
• Answer student questions directly or make reference to resources where answers may be found (and be forthright about lack of knowledge for those questions that the Instructor cannot answer);
• Reply to emails or other messages within 24 hours on weekdays and within 48 hours on weekends;
• Assign readings that adequately cover the materials and meet the learning objectives of the course while adhering to the time expectations for a 6-unit graduate-level course; and,
• Fairly assess individual and group work products against standards common to the professional fields of nonprofit and arts management.

The Instructor expects that Students will

• Arrive on-time for class and remain for the entire session;
• Be attentive and engaged while in class;
• Spend an adequate amount of time on the readings each week to be prepared for lectures and to meaningfully contribute to class discussion;
• Seek help from the Instructor, classmates, or university resources when appropriate;
• Act with decorum and civility even when in serious disagreement with another student or the Instructor;
• Prioritize this course’s requirements appropriately in consideration of other academic, professional, and personal obligations;
• Provide honest responses to Heinz College course evaluation questions and thoughtful feedback on the course materials and the Instructor through the course evaluation process; and,
• Maintain or exceed Carnegie Mellon University’s high standards for academic integrity (see next).

Academic Integrity

The Instructor views plagiarism and other forms of academic misrepresentation are viewed as extremely serious matters. Misrepresentation of another’s work as one’s own is widely recognized as among the most serious violations. The violation is clearly flagrant when it occurs as plagiarism on a course requirement, including take-home assignments and the final project. The punishment for such offenses may result in a failing grade in the course and disciplinary action by Heinz College.This course abides by the Ethics and Discipline policies and procedures as outlined in section 7 of the Heinz College’s College Handbook as well as Carnegie Mellon University’s Policy on Academic Integrity.

Plagiarism

Plagiarism is the failure to indicate the source of work—either with quotation marks or footnotes. The source can be a phrase, a graphic element, a proof, specific language, or an idea derived from the work of another person. (Note that material on the web is another person’s work and is therefore equally subject to the rules on plagiarism and cheating as any other source material.)

Cheating

Cheating includes, but is not limited to:

• Plagiarism (explained above);
• Submission of work that is not the student’s own;
• Submission or use of falsified data;
• Unauthorized access to an exam or assignment;
• Use of a stand-in for an exam;
• Use of unauthorized material in the preparation of an assignment or during an examination;
• Supplying or communicating unauthorized information to another student for use in an assignment or exam;
• Unauthorized collaboration on a course requirement (collaboration must be explicitly permitted by the Instructor to be considered authorized); and
• Submission of the same work for credit in more than one course.

Other Policies

Late Work

Unless otherwise specified by the advanced express consent of the Instructor:

• In-class group exercises must be turned in at the end of class.
• Take-home assignments will be accepted up to 24 hours after the assignment was due, with a 1% penalty of assessed against the final course score.
• The Final Project Deliverable will be accepted 24 hours after it was due, with a 2% penalty assessed against the student’s final course score.
• The Final Project Presentation must be made during the final regularly-scheduled class session unless other arrangements are made.

Extra Credit

No extra credit will be given to individual students. Students should plan to do well on all course requirements throughout the mini.

Incomplete Coursework

If, for any reason, a student is unable to complete Final Project by the deadlines specified herein, the student may request that an incomplete grade be assigned. The Instructor reserves the right to issue an incomplete grade in an instance where a student’s submitted Final Project so evidently fails to meet course expectations that, if scored, it would result in the student receiving a failing grade in the course.

In either circumstance, the student and the Instructor, in consultation with the student’s Program Director, will establish a contract detailing the work to be performed and the timeframe in which the work must occur to achieve a passing grade.

Licensing

Unless otherwise noted, all classroom materials created by the Instructor, Matt Hannigan, are provided for student use with a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Learn more at creativecommons.org.

Recording of Classroom Activities

Students may make audio of classroom activities for their personal use only. No copies may be distributed without the express consent of the Instructor. The Instructor may record audio for future instructional use. No video recordings are permitted.

Electronic Devices

Please disable and/or mute the noise-making function of all cell phones and computers before the start of class. Please use laptop computers and other electronic devices for course-appropriate purposes, for example taking notes and referencing reading materials. When this is not the case, please sit in the back row so that other students are not distracted by non-class screen content.

Preferred Name & Gender Pronoun

If a student prefers to be called a different name than what is on the class roster or has a preferred gender pronoun, please let the instructor know at the start of the course. If the instructor incorrectly pronounces a student name or uses an incorrect preferred gender pronoun, please do not hesitate to correct him during class.

Personal Health & Safety

Take care of yourself. Do your best to maintain a healthy lifestyle this mini by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. This will help you achieve your goals and cope with stress.

All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of your Heinz experience is learning how to ask for help. Asking for support sooner rather than later is often helpful.

If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, seek support from Counseling and Psychological Services (call 412-268-2922; web www.cmu.edu/counseling/) or consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.

If you or someone you know is feeling suicidal or in danger of self-harm, call Counseling and Psychological Services immediately, day or night at 412-268-2922 or the Re:solve Crisis Network at 888-796-8226.

If the situation is life threatening, call campus police (412-268-2323) or 911.

Americans with Disabilities Act

In compliance with the Americans with Disabilities Act (ADA), all qualified students enrolled in this course are entitled to “reasonable accommodations.” If a student has a documented disability, please contact the Office of Disability Resources to request an appropriate academic accommodation. Students must inform the Instructor of any special needs prior to the start of the second class session.

Grievances

Students are encouraged to discuss concerns regarding the course or the Instructor with their Program Director or the Heinz College Associate Dean. Students who wish to file a formal grievance should submit their concerns in writing to the Heinz College Dean.

Course Changes

The Instructor may change or alter the course syllabus, requirements, grading, readings, or any other matter whenever appropriate, provided that students receive at least one week’s notice of the change going into effect.