Karen Richardson, M.A.
University of South Florida
Simply defined, an outcome is the result of an intervention. From a historical perspective, one of the earliest known accounts of outcomes measurement and management, was based upon work conducted by Florence Nightingale in the 1860’s (Frattali, 1998). Outcomes measurement relates to collecting, analyzing, and reporting outcomes data. Outcomes management deals with implementation of changes based on the data, which ultimately are designed to improve outcomes. Outcomes assessment has been used at some level within university educational programs for many years. Due to increased competition for public and private financial support, expanded academic and clinical program requirements, smaller faculties, and increasing student enrollment, the importance of demonstrating academic accountability and educational quality have emerged as major national issues (Rassi, 1998). Educational programs must show that academic process and products meet the highest standards of quality. Outcomes measurement provides a mechanism for demonstrating accountability in university programs.
The current competency-based certification requirements have resulted in significant changes in the model of training for audiology. As programs are developing new curriculum designs, additional emphasis must be placed on the integration and strengthening of academic and clinical education. Innovative educational training models are imperative to insure that students obtain required knowledge and skills. The manner in which Communication Sciences and Disorders educational programs implement formative and summative outcomes measurement is in need of further investigation.
Formative outcomes assessment is ongoing and periodic. It involves monitoring students’ acquisition of knowledge and skills specific to immediate goals and objectives defined within academic courses, laboratory, and clinical practicum assignments provided on a semester-by-semester basis. The end result of the educational training process, via summative outcomes assessment, is the students’ demonstration of mastery of knowledge, clinical concepts, and skills. Ultimately, summative assessment of learning outcomes at the end of educational preparation is pertinent for documentation of students’ successful progress. Formative and summative outcomes assessments are equally important to program and curriculum development. Systematic outcomes measurement of program design, course offerings, course content and objectives, and other learning experiences is essential for quality assurance and improvement of the program.
The purpose of this paper is to discuss specific teaching strategies, interventions, and evaluative techniques, as they relate to intended or actual outcomes. Student learning outcomes and program instructional outcomes that have been measured within the current program at the University of South Florida will be outlined. Specific information regarding outcomes in many areas is currently being monitored and analysis will come later as the program develops. Initial discussion will outline the various levels of formative and summative outcomes assessment related to program and instructional issues, followed by formative and summative assessment of student learning outcomes. Finally, future program goals will be discussed.
Program and Instructional Outcomes
In August 1999, the Department of Communication Sciences and Disorders at the University of South Florida (USF) initiated a four-year clinical doctoral degree of audiology. Eleven new graduate students were accepted and 15 current master’s students transitioned into the Au.D. program. Several years of research and planning for program and curriculum development were required prior to program approval at the university and state levels. Consequently, the transition from the master’s to doctoral program has been relatively smooth over the past several months. The USF Au.D. program provides an extensive curriculum offering state-of-the-art academic and clinical educational opportunities.
The academic goals of the program are fundamental to the education and training of graduate students. Strong academic courses are the foundation of a rigorous and challenging Au.D. curriculum. Innovative teaching strategies have been incorporated to accomplish the goal of producing well-trained, competent audiologists. The initial phase of formative assessment in the academic area began with a comprehensive review of course content to insure that the objectives related to current ASHA knowledge and skill outcomes were being met. A systematic study of the courses delineated which competencies were being taught within specific courses. This process identified the need to update current courses and develop new courses to reflect trends in clinical practice and technological advances in the profession of audiology.
To maintain the momentum necessary for reviewing and refining various program components, audiology faculty has met on a weekly basis during Fall and Spring semesters over the past 18 months. The major advantage of integrating the weekly meeting format into the routine semester schedule is the continuity it provides for finalizing projects and maintaining individual responsibilities. A Director of Audiology, elected from constituent faculty every two years, helps plan and organize the meetings. Specific assignments and timelines facilitate accomplishing immediate and long-range program goals. A team approach to problem-solving, brainstorming ideas, and decision making has positively impacted interactions between academic and clinical faculty. This aspect of formative program assessment has yielded excellent outcomes.
Directly involving students in the decision-making process regarding their own learning is important for quality improvement (Rassi, 1998). To this end, USF conducted a student focus group at the end of the first semester, which included first year students enrolled in the program. The purpose of the focus group was for the students to provide input regarding strengths of the Au.D. program and the areas in need of improvement. All students participated on a voluntary basis and one student served as moderator. The audiology faculty did not attend or participate in the planning of the meeting. The Center Director worked with the student moderator to develop the questioning route, review focus group procedures, confirm the role and responsibilities of the moderator, and verify the methods of data collection. The Center Director also served as assistant moderator and assisted with data analysis and reporting of the findings.
After the focus group session, content analysis procedures were used to identify patterns and trends in the existing data. Findings were disseminated to the students and faculty. The areas of advising and student orientation generated the greatest number of comments and suggestions for improvement. Students suggested that details related to the number of credits, fees/tuition, schedule of courses, and course prerequisites must be provided prior to orientation. In the area of advising, students reported that regularly scheduled, individual advising was preferred. With the initiation of the new Au.D. program in Fall 1999, many of the printed materials, such as schedules, course descriptions, and advising forms were not available until students arrived for fall orientation. The importance of these issues is apparent and strategies have been implemented to prevent these problems in the future.
Students agreed that a major strength of the course work component is the strong link between courses and clinical practice. The clinical lab was considered to provide integration of content knowledge and practical skills with learning developing in a cumulative manner. Students did note current limitations of space and lack of availability of the audiology test booths and lab, which was considered to be ineffective for maximum learning. Fortunately, the CSD Department and Communication Disorders Center was approved for a new state-of-the-art departmental building and clinical facility with the anticipated relocation date of Spring, 2001. Many concerns regarding limited space and equipment will be resolved at that time.
Faculty resources were viewed as a strength of the program. However, it was suggested that sequencing and grouping of courses and clinical assignments regarding credit hours per semester and the resulting workload should be reviewed and revised. Students also proposed the development of a mentoring program for all newly admitted Fall 2000 graduate students. The plan included sending each accepted student a letter of welcome with basic information about the program, local area housing and travel, and pairing with a second year student as a “buddy.” Follow-up on the success of the project will be completed and outcomes will be reviewed.
Overall, the information and suggestions generated through the focus group were extremely valuable toward the formative evaluation process of the Au.D. program. There was a balance of positive statements, which confirmed the strengths of the program and specific suggestions for improvement. The faculty scheduled a meeting to discuss the students’ suggestions and began formulating strategies for addressing many of the issues. Many changes have currently been implemented into the program.
Instructors’ self-directed review of their own teaching effectiveness is another aspect of formative assessment at the instructional level. Critical self-evaluation should be an ongoing process in which instructors explore alternative teaching strategies to address varied student learning styles. Instructors are expected to provide in-depth knowledge about their discipline as well as an understanding of how students learn. Utilizing active learning strategies can enhance the traditional lecture format. There is strong research evidence supporting active approaches to facilitate learning as students are more likely to internalize, understand, and remember material learned through an active learning process (Bonwell & Sutherland, 1996). USF provides several special programs for improving teaching methods and skills in multimedia technology to assist instructors through the Center for Teaching Enhancement.
To summarize, the formative outcome assessments related to program and instructional areas currently implemented at USF include an initial comprehensive review of courses and course content, weekly audiology faculty meetings, the student focus group generating student input, and instructors’ self-review of teaching effectiveness and course design. Inclusion of each of these areas has made a significant contribution to overall program success by providing information to guide periodic modification of the program design.
At the end of each semester, students complete an evaluation for each course and practicum assignment. The evaluations are anonymous and a designated student representative returns the completed forms to the Department office. Areas evaluated for academic courses include clarity of course objectives, communication of ideas and information, facilitation of learning, and stimulation of interest in the course. Clinical faculty members are also evaluated in areas such as instructor’s effectiveness in teaching and developing clinical skills, presentation of alternative viewpoints or approaches, and promotion of discussion for active problem solving. Students are encouraged also to include written comments. A summary of the results is disseminated to each individual instructor. Areas of strengths and weaknesses are reviewed and strategies for improvement are developed.
In view of the success of the student focus group conducted at the end of the first semester which involved first year students only, the faculty recommended that the student focus group be convened annually and all current students should be asked to participate. As a summative assessment tool, the focus group yields extensive information about program outcomes. The faculty must remain committed to developing and maintaining a clinical doctoral audiology program of the highest standards. Therefore, ongoing formative and summative outcome assessments must be completed routinely and the resulting data must be reviewed in an unbiased manner to facilitate development and implementation of alternative strategies.
The USF-CSD program has implemented an exit evaluation of the program for several years. The evaluation addresses several aspects of the academic and clinical program. However, the evaluation has been completed sporadically to date. Obtaining feedback from the exit evaluation would be an important addition to currently available data regarding summative assessment of the program and instructional outcomes. The audiology faculty is in the process of modifying the current exit evaluation form and developing a consistent monitoring process to assure that all students complete the evaluation prior to graduation.
In summary, summative assessments related to program and instructional
areas include individual course and practicum evaluations each semester,
annual student focus groups, and exit evaluations at end of the graduate
program. These summative evaluations occur at significant junctures
in the program and provide information relevant to the continuation/discontinuation
of program components.
Student Learning Outcomes
The unique aspects of formative assessment of student learning outcomes at USF will be emphasized in this portion of the discussion. A systematic approach to formative assessment of student learning outcomes was developed. Traditionally, student performance in many aspects of the educational process is documented at the culmination of the process. However, ongoing monitoring of students’ progress throughout the program provides them with critical feedback regarding acquisition of academic knowledge and clinical skills. Whereas many instructional formats may include a midterm and final exam only, current audiology programs must include a more dynamic assessment process. Consequently, the ongoing learning activities or projects provided should have clearly defined goals and objectives. The manner in which students are required to demonstrate evidence of their knowledge and skills can be varied to include oral, written, or practical assignments. The clinical practicum context provides a natural setting for demonstration of and assessment of critical thinking, problem solving, and decision-making skills.
One component of formative assessment of student learning outcomes requires the development of a student portfolio. Development of a student portfolio is an important aspect of each student’s self-assessment of his/her commitment to and motivation for success in the Au.D. Program. The initiation of the portfolio is part of a course requirement which guides a student through the early stages of development, sets timelines for completion, and provides grading for certain assignments. Ultimately, the portfolio will be student managed and faculty monitored. The initial design included five sections: a biographical sketch; a self-rating of the student's current level of acquisition of ASHA competencies; samples of exemplary tests, projects, and papers; a section for feedback on clinical experiences from students; and a section for reflections.
In the biographical sketch, the student is encouraged to express his/her personal commitment to graduate education and discuss why the profession of audiology was chosen as a career path. This process requires introspection and assists the student in articulating which aspects of the profession of audiology may have fostered their interest. Most students report many interesting life experiences that were responsible for acquainting them with audiology.
A five-point competency rating scale was developed utilizing ASHA competencies listed in Standard IV-D: Evaluation and IV-E: Treatment. During the first semester, students rate their perceived level of competence in each area. As part of periodic self-assessment, students again rate their perceived progress at the end of the first semester and thereafter at the end of each year. A complete copy of the ASHA Standards is reviewed with the students. This process acquaints the student early in the educational process with the expected competencies and educational outcomes required for ASHA certification.
A representative sample of the student’s work in various courses is added to the third section. Students select examples of tests, papers, and other assignments completed throughout the year, which provide support for demonstration of progress in acquisition of various competencies. As students re-evaluate their performance at the end of each semester, reference to materials gathered in this section should help support their perceived progress.
The Feedback section requires that students document their clinic shadowing and observation. Each first semester student is matched with an audiologist in different practice settings for one day to observe the range of activities in a typical workday. The audiologist discusses various aspects of diagnostic and treatment service provision, as well as office management responsibilities . Students are required to summarize their observations of clinical activities for the particular site. The objective of the assignment is to provide exposure to “real world” settings to help formulate realistic expectations for their role and responsibilities in a clinical setting. Student shares this information in class discussions with their peers.
The last section of the Portfolio is referred to as reflections. Each student writes a final essay addressing his or her progress made by the end of the semester. A description of academic and clinical experiences that support the results of the self-rating should be included. In addition, information regarding major areas for improvement and goals for next year should be included.
Some possible additional sections to the Portfolio for next year include a section for Scholarly information, Professional data, and a USF Clinical Competency Checklist. The Scholarly section will include a copy of the research proposal completed as part of the second year Research class and a copy of the approved proposal for the Professional Research Project (PRP), which is initiated in the third year. A copy of the proposed journal article completed for the PRP and approved for submission to a peer-reviewed journal should also be included. The Professional section may include any work completed as part of professional presentations or poster sessions, conferences, and seminars. The student will be asked to complete a resume/vitae by the end of second year.
An updated, revised, comprehensive Clinical Competency Checklist, which reflects the Au.D., is still under development by the clinic. The concept of a checklist that identifies each competency or skill area would provide a means for reviewing and updating competencies at the end of each semester for all students. Based on students' progress in their academic and clinical coursework, the level of acquisition of each competency will be documented and recorded on the checklist. The importance of this process is to insure that the student has clear and specific information related to his or her progress and to assist in understanding the ongoing academic and professional needs and responsibilities for success.
The student Portfolio can be viewed as extremely important documentation of students' progress throughout their doctoral education and has other potential uses. As students compete for a variety of internship opportunities during their third and fourth years, the Portfolio could be used to support their application. A sample of individual innovative and creative research projects, as well as excellent academic and clinical work, can be displayed.
Biweekly Grand Rounds is a new learning experience initiated this year in the Au.D. program and is frequently associated with medical educational methodology. The experience provides a means of formative assessment, which is practical and spontaneous. The format of the meetings rotates between student case presentations, faculty presentations, and a journal club with rotating student discussion leaders. Student case presentations and student-lead discussions of classic and current research literature provide an excellent peer teaching/learning experience. Faculty presentations summarizing national professional presentations or updating other scholarly, clinical, or research work is another important learning experience outside the classroom. Frequent and varied learning situations with all levels of the faculty are essential for reinforcing concepts, improving understanding of critical issues, and fostering confidence and independence. Consequently, the student views the teacher as a scholar and practitioner with sustained learning and teaching objectives. The recognition of life-long learning goals is important for the development of professional responsibility, personal commitment, and self-directed learning skills that are essential for audiologists and medical practitioners (Tharpe, 1995).
The array of activities included in the Grand Rounds format are also important for helping students critically evaluate diagnostic and treatment strategies, improve understanding of research findings and ongoing research opportunities, and learn to critically evaluate the past and current audiology literature. Students demonstrate these skills by completing written assignments, participating in discussions to review pertinent issues, or answering related questions as assigned by the faculty coordinator of Grand Rounds. The effectiveness of developing the intended learning outcomes by participating in these activities can be assessed through the above formative assessment techniques.
Biweekly Professional Group meetings alternate weekly with Grand Rounds. The faculty felt it was important to provide a separate meeting for the individual student cohorts in the program. The cohorts of students include the new first year students, the second year students, and the third year students. Issues confronting these three groups of students, particularly those transitioning from the master’s program into the Au.D. program are very different. These meetings were focused on student-driven topics of discussion and were moderated by different faculty teams. For example, the first year students spent additional time reviewing basic clinical evaluation strategies and procedures; the second year students focused on case management rationales and new hearing aid technology; and the third year students preferred to explore professional issues related to future fourth year externship assignments and resume writing. Most students felt these meetings were productive and provided additional information or clarification of particular areas. They also offered useful suggestions for topics for next year.
Another important aspect of academic formative assessment involves all students in a midterm evaluation with their assigned academic and clinical faculty advisors. A comprehensive review of each student’s progress in academic and clinical courses is conducted collectively by the faculty. Students are then seen for individual conferences to discuss the information and elicit feedback from them. They are assisted in formulating goals or strategies for improvement where needed. A form with a summary of the information is completed and signed by the students and advisors and placed in the permanent file. Students have expressed a very positive impression about these meetings and feel they are beneficial in helping them understand their individual needs for improvement.
A clinical laboratory class offered during each semester of the first year is a new class developed this year. The obvious benefits of incorporating a hands-on lab component provide early exposure and training in several diagnostic and clinical procedures prior to clinical practicum assignments. The lab goals and objectives are coordinated with the first year academic courses and exercises are designed to promote the development of specific skills. Innovative teaching strategies utilizing computer-assisted instruction include the following areas: diagnostic pure-tone and immittance simulation software test battery approach and management of auditory disorders, and web-based interactive learning modules. Students obtain practical experience on all basic clinic equipment and during the last semester, they participate in supervisor demonstration teaching and individual diagnostic evaluation sessions with normal hearing adult patients. The lab assignments are developmentally sequenced and students must integrate basic knowledge and skills to successfully complete the later, more advanced assignments. The formative assessment is performance based and the instructor can determine the concepts with which a student may need additional assistance. The ultimate goal of the clinical lab is to prepare students to perform successfully in their first clinic practicum assignment.
Once a student is enrolled in a clinic practicum assignment, many formative assessment strategies of earning outcomes have been incorporated into the USF clinical instructional framework. Weekly pre-session review and planning meetings are scheduled to assist the student in developing appropriate testing rationales and strategies, particularly early in the practicum experience. Post-session wrap-up meetings are extremely important for discussing performance and patient or testing related issues. Immediate feedback regarding strengths and weaknesses can be provided. Thus, daily performance assessments either written or oral can be discussed.
A formal midterm evaluation is scheduled with each student at which time specific information about performance and progress across all areas is presented. The following six areas are included in the midterm assessment: clinical procedures, pre-assessment, assessment, written documentation, interviewing/counseling procedures, and clinical problem solving. The midterm evaluation is extremely important to the formative assessment process of clinical performance outcomes. If a student is performing at a grade level of “C” or below in any two of the six areas, or obtains an overall grade of “C” based on weighting assigned to each area, the student is recommended for the Clinical Assistance Program (CAP). The CAP provides a formal intervention as a result of the formative evaluation.
The basis of initiating a CAP allows the student to obtain additional supervisory support in an effort to improve clinical performance. The CAP is generally begun in concert with the midterm review. Based upon the areas identified, the student and supervisor develop realistic individualized goals and objectives targeted to improve performance in the identified areas. Specific criteria are developed for meeting the goals and timelines are set. By the end of the semester, the student must successfully meet the defined goals and objectives by earning a grade of “B” or better in the specific areas. If an overall grade of “C” or less is obtained the student is required to repeat the practicum assignment and no clock hours are awarded for this clinical experience. Formulating and managing the CAP are additional clinical responsibilities for the individual supervisor. However, the desire to assist each student in performing successfully is of paramount importance.
At the end of each semester or prior to academic advising for the subsequent semester, faculty advisors review each advisee’s semester grades and overall grade point average (GPA). If a grade of less than “B” is received in any course, the student will be required to repeat that course. An overall GPA of “B” (3.0) or better is required at the graduate level and a student is placed on academic probation if the GPA falls below that level. Improvement in the next semester must be achieved to remain in the program. Periodic review each semester by faculty advisors helps identify students in need of academic assistance or possible course scheduling adjustments.
Formative assessments of student learning outcomes are quite extensive within the USF Au.D. program. Successful academic and clinical progress for each student is a fundamental program goal. Formative assessment strategies are essential for monitoring student progress while in the program and determining achievement for continuing in the program.
Throughout the student’s educational program, successful acquisition of knowledge and skills is documented at the culmination of the learning process via a summative assessment. A final evaluation is determined by a combination of assessments related to class projects, written and oral presentations, clinical performance, and examinations. The Au.D. program requires that a final grade of “B” or better be obtained for successful completion of all courses. As mentioned earlier, a student must maintain a GPA of 3.0 or better to successfully complete the graduate program in audiology.
In addition to demonstrating appropriate development and progress in
clinical competencies and performance related skills throughout the semester
via formative assessment, each student must pass an oral practical examination
at the end of each semester during the second year while enrolled in clinical
practicum. The oral practical format requires the student to demonstrate
through an actual hands-on task or discussion the appropriate methods of
evaluation or management of the particular problem or situation.
Utilizing this evaluation method, assessment of the student’s critical
thinking skills, problem-solving abilities, and depth of academic and clinical
knowledge may be achieved. Exploration of various options for managing
a particular issue can also be discussed. Many students are somewhat
intimidated by the oral practical examination situation, particularly the
first time. However, many students acknowledge the value and utility
of this strategy for performance assessment.
Prior to entering fourth year internships, students are required to pass the national certification examination (NESPA) adopted by ASHA. By the end of the third year of the program, students have completed all required course work. The NESPA is utilized as a summative assessment tool within the USF curriculum to document mastery of academic knowledge and professional concepts and skills. Timely scheduling for taking the examination and receiving results are pertinent to when students receive assignments for the fourth year placement.
As we approach the end of the first year of the Au.D. program at USF, a tremendous amount of work still must be completed. One of the program requirements for all entering students is passing a comprehensive oral and written examination at the end of the second year. The students must pass the examination to continue into the third year. Preliminary work has begun in identifying minimal expected competency levels at the end of the second and third years, based on the specific knowledge and skills areas outlined in the Certification Standard by ASHA. In the absence of available quantitative research data, qualitative measures have been utilized in the current research design. Early data sampling suggests significant variations in levels of performance across skill areas. Many of these variations may be related to differences in exposure and opportunities to learn a particular skill. After completing the data collection process and analysis, the development process for the examination will begin.
As mentioned earlier, The USF-CSD Department was approved for a new academic and clinical facility several years ago. The new building will be located in the medical/science quadrangle in close proximity to the Moffitt Cancer Center and the University Medical Clinics. The projected relocation date is Spring, 2001. New program affiliations are currently underway in an effort to expand the current client base and develop new clinical training opportunities for the students. The United Cerebral Palsy Center of Tampa Bay and James A Haley Veteran’s Administration Hospital will each develop an on-site satellite clinic within our building. With these new program affiliations, unique clinical training opportunities and physical resource expansion will result in significant enhancement of the Au.D. program.
In summary, the incorporation of formative and summative assessment of outcomes is important to all Au.D. programs. Audiology training programs are ultimately responsible for assessing and documenting student learning outcomes throughout the educational process and improving the quality of graduate programs. The intent of this paper was to provide examples of systematic and innovative methods of executing this responsibility. The major benefit to undergoing the intensive program review and development process for the new Au.D. degree program was the determination of program strengths and weaknesses. Once those areas were identified, components in need of improvement became clear. It was also reassuring to determine the many positive aspects of the program. In order to provide appropriate academic and clinical training opportunities that enhance students’ knowledge and performance skills, clearly defined learning objectives are essential. Creatively designed teaching modules and lessons that encourage students to “think outside the box” promote critical thinking and problem-solving skills imperative for successful, competent Doctors of Audiology.
American Speech-Language-Hearing Association. (1996). A study of clinical activities and knowledge areas for the certified audiologist. Rockville, MD.
American Speech-Language-Hearing Association. (1997). Certification standards for audiology. Rockville, MD. [Online]. Available FTP: http://www/asha.org/library/audiology_standards.htm
Bonwell, C. C. & Sutherland, T. E. (1996). The active learning continuum: Choosing activities to engage students in the classroom. In T. Sutherland & C. Bonwell (Eds.), Using active learning in college classes: A range of options for faculty. Jossey-Bass Publishers.
Frattali, C. M. (1998).Outcomes measurement: definitions, dimensions, and perspectives. In C. Frattali (Ed.), Measuring outcomes in speech-language pathology. New York: Thieme.
Rassi, J. A. (1998). Outcomes measurement in universities. In C. Frattali (Ed.) Measuring outcomes in speech-language pathology. New York: Thieme.
Tharpe, A. M., Rassi, J. A., & Biswas, G. (1995).
Problem-based learning: An innovative approach to audiology
education. American Journal Of Audiology, 4(1), 19-25.