Formative And Summative Outcomes Assessment:
What Do We Mean By Doing It With Meaning?
Brooke Hallowell, Ph.D.
With the recent enhanced focus on educational outcomes in accreditation
standards of ASHA's Council on Academic Accreditation (CAA) and with all
regional accrediting agencies in the Unites States now requiring extensive
outcomes assessment plans for all academic units, it is increasingly important
that we share assessment ideas and methods among academic programs.
It is also important that we ensure that our assessment efforts are truly
meaningful, relevant, and useful to our students and faculty. I will
provide a brief discussion of "meaningful" assessment practices to serve
as an introduction to the next two presentations, which will focus on some
hands-on means of outcomes assessment in audiology and in speech-language
pathology. I will also review and clarify a few of the key terms
we use to discuss and analyze outcomes measures, and then provide some
materials to help member programs evaluate their own assessment programs
(Appendix A) and strategize for improved assessment practices (Appendix
Members of the CAPCSD's Working Group on Educational Outcomes have spent
a great deal of time over the past two years discussing and clarifying
what it means to engage in meaningful outcomes assessment. In our
1998 address to the Council (Hallowell & Lund, 1998), Nan Lund
and I summarized that meaningful assessment practices:
Meaningful outcomes assessment programs are advantageous in that they highlight
our accountability for student learning, exploit the phenomenon that students
learn according to how they are assessed, allow for strategic program planning,
lead to more bottom-up than top-down emphases in handling bureaucratic
assessment demands, and encourage academic faculty and clinical staff members
to communicate and collaborate with one another.
Enhance our education and training missions in specific, practical, measurable
Are oriented toward improving the effectiveness of training and education;
Involve all of a program's faculty and students; and
Involve the use of assessment results to shape real training program modifications.
Agreeing on Terminology
Reference to previous presentations to the CAPCSD (Hallowell, 1996;
Hallowell & Lund, 1998) may help to clarify details pertaining to the
constructs that are most helpful educational outcomes practices. The specific
terms used to discuss educational outcomes are variable across contexts.
The way we develop and use assessments methods matters much more than our
agreement on the definitions of each of the terms we might use to discuss
assessment. Still, a few key terms are presented here just for the
sake of establishing common ground before our further discussion.
We use the term "outcome" to refer to the status of the output of our
programs. We might assess the outcome of student's experiences in
a component of a course, in a whole course, clinical practicum or research
experience, or in a degree program, as well as in a department, college,
institution, or region. With outcomes measures, we address the question
of how students are changed by their experience in one of our training
programs. It is important to differentiate between (1) outcomes,
which involve the assessment of groups of students to make statements about
the effectiveness of one or a set of teaching/learning experiences, and
(2) student evaluation, which involves grading or estimation of individual
students' accomplishments. The distinction between outcomes assessment
and student evaluation is especially critical when we discuss the use of
student evaluations grouped together across classes of students (e.g.,
graduates from a particular program for a specific year) as indicators
of educational outcome. Some of the specific methods of evaluating
individual students to be presented by Karen Richardson and Cynthia Bartlett
are mentioned in today's discussion of educational outcomes because, when
used across groups of students, they may yield important information about
aspects of program effectiveness. Let us clarify, however, that indices
of individual student gains do not constitute outcome measures per se.
There is often some confusion of the term "outcome" in the discussion
of accreditation standards as compared to the discussion of certification
standards in speech-language pathology and audiology. Accreditation
standards, overseen by ASHA's CAA, involve a focus on outcomes assessment.
Candidates for accreditation are expected to demonstrate that they effectively
assess programmatic learning outcomes of students and graduates and that
they use their assessments to shape program modifications. By contrast,
certification standards, overseen by ASHA's Council on Professional Standards
in Speech-Language Pathology and Audiology, involve a focus on individual
accomplishment. Individuals who wish to be certified are expected
to demonstrate specific knowledge and abilities.
Formative and Summative Outcomes
Summative outcome assessments are indices of the final product or the
end result of an educational program. They are used to characterize
programs, college divisions, or even whole institutions. By contrast,
formative outcome assessments are those that may be used to shape the experiences
and learning opportunities of the very students who are being assessed.
A balance between formative and summative assessments is essential to strategic,
effective outcomes assessment practice.
A basic taxonomy of different types of learning outcomes helps us to
verify that we are addressing an appropriate blend of different type and
areas of learning in our assessments. Cognitive outcomes are those
that relate to intellectual mastery, or mastery of knowledge in specific
topic areas. Performative outcomes relate to a students' accomplishment
of behavioral tasks. Affective outcomes relate to personal qualities
and values that students ideally gain.
Assessing Your Own Assessment Practices
Appendix A is presented as a tool for directors and faculty members
to evaluate their own assessment practices. Nan Lund and I originally
developed it for the ASHA CAA to help plan for site visitor training in
light of new outcomes assessment foci in the accreditation standards that
became effective in January of 1999.
Action Planning for Improved Outcomes Assessment
Appendix B is offered in an effort to stimulate specific actions for
improved assessment practices among member programs. Nan Lund and
I have used and modified the content of both appendices over the past three
years in our consultations with representatives from numerous programs
in communication sciences and disorders, as well as with CAA board members
and site visitors.
Hallowell, B. (1996). Innovative models of curriculum/instruction: Measuring
educational outcomes. In Council of Graduate Programs in Communication
Sciences and Disorders, Proceedings of the seventeenth annual conference
on graduate education, 37-44.
Hallowell, B. & Lund, N. (1998). Fostering program improvements
through a focus on educational outcomes. In Council of Graduate Programs
in Communication Sciences and Disorders, Proceedings of the nineteenth
annual conference on graduate education, 32-56.
EVALUATING OUTCOMES ASSESSMENT PRACTICES WITHIN
AN EDUCATIONAL/TRAINING PROGRAM
Buffalo State College
Is there a well-defined process for articulating and modifying targeted
educational outcomes for the program?
Is it clear what the program's targeted educational outcomes are?
Are student and alumni perspectives on outcomes assessment practices incorporated?
Are students aware of general targeted educational outcomes?
Is there a balance of focus on both summative and formative educational
Do stated targeted educational outcomes represent a blend of cognitive,
performative, and affective outcomes?
Is there a focus on outcomes across, as well as within, the curriculum?
Are there clearly defined assessment procedures that may be used to guide
student learning, validate learning experiences, and guide program
Is there a plan and schedule for sampling student performance or products
to assess all goals within a reasonable period of time?
Are all faculty members involved in current outcomes assessment practices
within the program?
Are they all familiar with program's stated target educational outcomes?
If there is a person or committee responsible for overseeing assessment
efforts, is that person or committee including all of the faculty in goal-setting,
assessment, and report-writing efforts, and in discussion of how the program
might respond to strengths and weaknesses identified through assessment
If there are separate clinical and academic faculty members, do they collaborate
in establishing targeted educational outcomes, in outcomes assessment,
and in discussion of how to make use of assessment results?
Are they all involved in the development of comprehensive outcomes assessment
practices for the program?
Does the program have an action plan for addressing any faculty concerns
regarding the administrative demands of outcomes assessment?
Have effective pedagogical methods been identified to foster targeted outcomes?
Are faculty members using effective pedagogical methods to enhance targeted
USE OF ASSESSMENT DATA
Is literature on educational outcomes, teaching methodologies, and assessment
practices available for use by faculty members wishing to develop or implement
additional teaching practices and assessment techniques?
Does the program have access to consultants who can assist in faculty training
and motivation regarding outcomes assessment issues?
Is there sufficient administrative assistance for data collection and analysis?
Are there institutional offices available to assist with summative data
collection? If so, are their services being used?
Are assessment data used to demonstrate accomplishment of goals for student
Are assessment data used to evaluate specific learning experiences within
Are assessment data used to guide program modifications?
Has the process of reviewing assessment data included action planning for
additional assessment methods and instruments that would improve the assessment
Have additional assessment instruments been developed, adapted, and/or
refined to match targeted educational outcomes?
If so, have they been used?
Are formative assessment results being used to improve the experiences
of individual students and of the program?
Are summative assessment results being used to improve the program?
Is there clear evidence of program improvements based on outcome assessment
RECOMMENDED STEPS IN MEANINGFUL OUTCOMES ASSESSMENT
Buffalo State College
Cognitive outcomes were defined as those relating to intellectual mastery,
or mastery of knowledge in specific topic areas (e.g., ability to describe
the basic anatomy of the human hearing mechanism).
Develop a faculty awareness of the goal of improving the focus on educational
outcomes within your programs, and engage faculty in the development of
comprehensive outcomes assessment practices for all programs.
Address faculty concerns regarding administrative load.
Develop a faculty resource base, including literature on educational outcomes,
teaching methodologies, and assessment practices, to be used by faculty
members wishing to develop or implement additional teaching practices
and assessment techniques. Make this resource base available
to all faculty members, and continuously expand upon it.
Decide whether to include students' perspectives on outcomes assessment
Define your general goals for student learning, the general targeted educational
Define your goals for the process of assessing educational outcomes.
Balance your focus on both summative and formative educational outcomes,
in order to ensure the use of assessment activities that will: (1) lead
to programmatic improvements that will influence the outcomes of students
in the future; and (2) provide more immediate feedback to faculty and administrators
concerning students' learning such that the experiences of individual students
and classes of students can be modified to enhance the learning of students
currently enrolled in each program.
Focus on outcomes across, as well as within, the curriculum.
Enlist all clinical and academic faculty to collaborate in defining specific
targeted formative outcomes grouped according to three outcome categories:
Cognitive, Performance, and Affective.
Performance outcomes are defined as those relating to the functioning
of a student, or of a program graduate, in a professional setting (e.g.,
ability to select appropriate formal assessment instruments for effective
diagnostic problem solving for a variety of
Affective outcomes are defined as personal qualities and values that
students ideally gain from their experiences during a particular educational
and training program (e.g., demonstration of sensitivity to and appreciation
of multicultural issues in a variety of contexts).
Analyze and edit for redundancy the collaborative list of targeted outcomes.
Your new targeted educational outcomes list may be used as the basis
for the next phase of outcomes assessment planning activities.
Develop a plan to sample students' performance or products that demonstrate
accomplishment of specific goals. It is not necessary for every goal
to be assessed every year or for all students. A subset of goals
may be targeted in a given year, with other goals in subsequent years.
The plan should address the sampling procedure and the schedule for assessing
Use assessment data to demonstrate accomplishment of goals for student
learning, as defined by the faculty.
Develop and implement an analytical system for tracking: (1) the articulation
of specific targeted outcomes to students; (2) the refinement of stated
targeted outcomes; (3) the means of assessment that are used to guide student
learning, validate learning experiences, and guide program modifications;
(4) needs for enhancement of learning opportunities; and (5) the specific
experiences intended to further develop targeted educational outcomes.
Develop, refine, share, and implement additional assessment instruments
to match targeted educational outcomes for all programs.
Make use of formative assessment information to improve the experiences
of individual students.
Make use of summative assessments to improve your program(s).
Document evidence of program improvements based on outcome assessment
Review and expand upon each program's general and specific goals
Address concerns regarding administrative load
Monitor the consistency and clarity with which you articulate targeted
outcomes to students at all program levels
Analyze needs for improved assessment practice
Stimulate use of effective pedagogical methods to enhance targeted outcomes