With the recent enhanced focus on educational outcomes in accreditation standards of ASHA's Council on Academic Accreditation (CAA) and with all regional accrediting agencies in the Unites States now requiring extensive outcomes assessment plans for all academic units, it is increasingly important that we share assessment ideas and methods among academic programs.  It is also important that we ensure that our assessment efforts are truly meaningful, relevant, and useful to our students and faculty.  I will provide a brief discussion of "meaningful" assessment practices to serve as an introduction to the next two presentations, which will focus on some hands-on means of outcomes assessment in audiology and in speech-language pathology.  I will also review and clarify a few of the key terms we use to discuss and analyze outcomes measures, and then provide some materials to help member programs evaluate their own assessment programs (Appendix A) and strategize for improved assessment practices (Appendix B).

Members of the CAPCSD's Working Group on Educational Outcomes have spent a great deal of time over the past two years discussing and clarifying what it means to engage in meaningful outcomes assessment.  In our 1998 address to the Council (Hallowell & Lund, 1998),  Nan Lund and I summarized that meaningful assessment practices:

• Enhance our education and training missions in specific, practical, measurable ways;
• Are oriented toward improving the effectiveness of training and education;
• Involve all of a program's faculty and students; and
• Involve the use of assessment results to shape real training program modifications.

Agreeing on Terminology

Reference to previous presentations to the CAPCSD (Hallowell, 1996; Hallowell & Lund, 1998) may help to clarify details pertaining to the constructs that are most helpful educational outcomes practices. The specific terms used to discuss educational outcomes are variable across contexts.  The way we develop and use assessments methods matters much more than our agreement on the definitions of each of the terms we might use to discuss assessment.  Still, a few key terms are presented here just for the sake of establishing common ground before our further discussion.

Outcomes

We use the term "outcome" to refer to the status of the output of our programs.  We might assess the outcome of student's experiences in a component of a course, in a whole course, clinical practicum or research experience, or in a degree program, as well as in a department, college, institution, or region. With outcomes measures, we address the question of how students are changed by their experience in one of our training programs.  It is important to differentiate between (1) outcomes, which involve the assessment of groups of students to make statements about the effectiveness of one or a set of teaching/learning experiences, and (2) student evaluation, which involves grading or estimation of individual students' accomplishments.  The distinction between outcomes assessment and student evaluation is especially critical when we discuss the use of student evaluations grouped together across classes of students (e.g., graduates from a particular program for a specific year) as indicators of educational outcome.  Some of the specific methods of evaluating individual students to be presented by Karen Richardson and Cynthia Bartlett are mentioned in today's discussion of educational outcomes because, when used across groups of students, they may yield important information about aspects of program effectiveness.  Let us clarify, however, that indices of individual student gains do not constitute outcome measures per se.

There is often some confusion of the term "outcome" in the discussion of accreditation standards as compared to the discussion of certification standards in speech-language pathology and audiology.  Accreditation standards, overseen by ASHA's CAA, involve a focus on outcomes assessment.  Candidates for accreditation are expected to demonstrate that they effectively assess programmatic learning outcomes of students and graduates and that they use their assessments to shape program modifications.  By contrast, certification standards, overseen by ASHA's Council on Professional Standards in Speech-Language Pathology and Audiology, involve a focus on individual accomplishment.  Individuals who wish to be certified are expected to demonstrate specific knowledge and abilities.
 
Formative and Summative Outcomes

Summative outcome assessments are indices of the final product or the end result of an educational program.  They are used to characterize programs, college divisions, or even whole institutions.  By contrast, formative outcome assessments are those that may be used to shape the experiences and learning opportunities of the very students who are being assessed.  A balance between formative and summative assessments is essential to strategic, effective outcomes assessment practice.

Cognitive/Affective/Performative Outcomes

A basic taxonomy of different types of learning outcomes helps us to verify that we are addressing an appropriate blend of different type and areas of learning in our assessments.  Cognitive outcomes are those that relate to intellectual mastery, or mastery of knowledge in specific topic areas.  Performative outcomes relate to a students' accomplishment of behavioral tasks.  Affective outcomes relate to personal qualities and values that students ideally gain.

Assessing Your Own Assessment Practices

Appendix A is presented as a tool for directors and faculty members to evaluate their own assessment practices.  Nan Lund and I originally developed it for the ASHA CAA to help plan for site visitor training in light of new outcomes assessment foci in the accreditation standards that became effective in January of 1999.

Action Planning for Improved Outcomes Assessment

Appendix B is offered in an effort to stimulate specific actions for improved assessment practices among member programs.  Nan Lund and I have used and modified the content of both appendices over the past three years in our consultations with representatives from numerous programs in communication sciences and disorders, as well as with CAA board members and site visitors.

References

Hallowell, B. (1996). Innovative models of curriculum/instruction: Measuring educational outcomes.  In Council of Graduate Programs  in Communication Sciences and Disorders, Proceedings of the seventeenth annual conference on graduate education, 37-44.

Hallowell, B. & Lund, N. (1998).  Fostering program improvements through a focus on educational outcomes.  In Council of Graduate Programs in Communication Sciences and Disorders, Proceedings of the nineteenth annual conference on graduate education, 32-56.

STRATEGIC PLANNING

• Is there a well-defined process for articulating and modifying targeted educational outcomes for the program?
• Is it clear what the program's targeted educational outcomes are?
• Are student and alumni perspectives on outcomes assessment practices incorporated?
• Are students aware of general targeted educational outcomes?
• Is there a balance of focus on both summative and formative educational outcomes?
• Do stated targeted educational outcomes represent a blend of cognitive, performative, and affective outcomes?
•  Is there a focus on outcomes across, as well as within, the curriculum?
• Are there clearly defined assessment procedures that may be used to guide student  learning, validate learning experiences, and guide program modifications?
• Is there a plan and schedule for sampling student performance or products to assess all goals within a reasonable period of time?

• Are all faculty members involved in current outcomes assessment practices within the program?
• Are they all familiar with program's stated target educational outcomes?
• If there is a person or committee responsible for overseeing assessment efforts, is that person or committee including all of the faculty in goal-setting, assessment, and report-writing efforts, and in discussion of how the program might respond to strengths and weaknesses identified through assessment processes?
• If there are separate clinical and academic faculty members, do they collaborate in establishing targeted educational outcomes, in outcomes assessment, and in discussion of how to make use of assessment results?
• Are they all involved in the development of comprehensive outcomes assessment practices for the program?
• Does the program have an action plan for addressing any faculty concerns regarding the administrative demands of outcomes assessment?
• Have effective pedagogical methods been identified to foster targeted outcomes?
• Are faculty members using effective pedagogical methods to enhance targeted outcomes?

• Is literature on educational outcomes, teaching methodologies, and assessment practices available for use by faculty members wishing to develop or implement additional teaching practices and assessment techniques?
• Does the program have access to consultants who can assist in faculty training and motivation regarding outcomes assessment issues?
• Is there sufficient administrative assistance for data collection and analysis?
• Are there institutional offices available to assist with summative data collection?  If so, are their services being used?

• Are assessment data used to demonstrate accomplishment of goals for student learning?
• Are assessment data used to evaluate specific learning experiences within the program?
• Are assessment data used to guide program modifications?
• Has the process of reviewing assessment data included action planning for additional assessment methods and instruments that would improve the assessment process?
• Have additional assessment instruments been developed, adapted, and/or refined to match targeted educational outcomes?
• If so, have they been used?
• Are formative assessment results being used to improve the experiences of individual students and of the program?
• Are summative assessment results being used to improve the program?
• Is there clear evidence of program improvements based on outcome assessment activities?

• Develop a faculty awareness of the goal of improving the focus on educational  outcomes within your programs, and engage faculty in the development of  comprehensive outcomes assessment practices for all programs.
• Address faculty concerns regarding administrative load.
• Develop a faculty resource base, including literature on educational outcomes,  teaching methodologies, and assessment practices, to be used by faculty members  wishing to develop or implement additional teaching practices and assessment  techniques.  Make this resource base available to all faculty members, and continuously expand upon it.
• Decide whether to include students' perspectives on outcomes assessment practices.
• Define your general goals for student learning, the general targeted educational outcomes.
• Define your goals for the process of assessing educational outcomes.
• Balance your focus on both summative and formative educational outcomes, in order to ensure the use of assessment activities that will: (1) lead to programmatic improvements that will influence the outcomes of students in the future; and (2) provide more immediate feedback to faculty and administrators concerning students' learning such that the experiences of individual students and classes of students can be modified to enhance the learning of students currently enrolled in each program.
• Focus on outcomes across, as well as within, the curriculum.
• Enlist all clinical and academic faculty to collaborate in defining specific targeted formative outcomes grouped according to three outcome categories:  Cognitive, Performance, and Affective.

Performance outcomes are defined as those relating to the functioning of a student, or of a program graduate, in a professional setting (e.g., ability to select appropriate formal assessment instruments for effective diagnostic problem solving for a variety of
 patient/client types.

Affective outcomes are defined as personal qualities and values that students ideally gain from their experiences during a particular educational and training program (e.g., demonstration of sensitivity to and appreciation of multicultural issues in a variety of contexts).

• Analyze and edit for redundancy the collaborative list of targeted outcomes.  Your new  targeted educational outcomes list may be used as the basis for the next phase of  outcomes assessment planning activities.
• Develop a plan to sample students' performance or products that demonstrate accomplishment of specific goals.  It is not necessary for every goal to be assessed every year or for all students.  A subset of goals may be targeted in a given year, with other goals in subsequent years.  The plan should address the sampling procedure and the schedule for assessing each goal.
• Use assessment data to demonstrate accomplishment of goals for student learning, as defined by the faculty.
• Develop and implement an analytical system for tracking: (1) the articulation of specific targeted outcomes to students; (2) the refinement of stated targeted outcomes; (3) the means of assessment that are used to guide student learning, validate learning experiences, and guide program modifications; (4) needs for enhancement of learning opportunities; and (5) the specific experiences intended to further develop targeted educational outcomes.
• Develop, refine, share, and implement additional assessment instruments to match targeted educational outcomes for all programs.
• Make use of formative assessment information to improve the experiences of  individual students.
• Make use of summative assessments to improve your program(s).
• Document evidence of program improvements based on outcome assessment  activities.

• Engage faculty
• Review and expand upon each program's general and specific goals
• Address concerns regarding administrative load
• Monitor the consistency and clarity with which you articulate targeted outcomes to students at all program levels
• Analyze needs for improved assessment practice
• Stimulate use of effective pedagogical methods to enhance targeted outcomes