Course Evaluation

---

Program Evaluation Rubric

Effective course evaluation requires more than just student feedback. To that end, we have also designed a rubric for use by our employees, as well as by clients, subject matter experts, and other interested third parties. This rubric uses findings by Russell K. Baker of the University of Tampa (local copy of article available here) and Harold Henke of Nova-Southeastern University (local copy of paper available here) as a basis.

Rubric Design and Operation

This rubric combines information assembled by Russel K. Baker from Ralph Tyler's 1949 book Basic Principles of Curriculum and Instruction and Benjamin Bloom's 1956 book Taxonomy of Educational Objectives: Handbook 1: The Cognitive Domain. From Harold Henke's paper, we included elements of the rubric he constructed using information from Jakob Nielsen's Top Ten Web Design Mistakes and Jones' and Okey's Interface Design for Computer-based Learning Environments. Our course evaluation rubric is therefore subdivided into several major categories.

Each category has a number of items. A rating scale of 0 to 4 accompanies each item, including a description of each what each rating entails. To use the rubric, evaluators grade each item according to the rating which best describes that item with respect to the course. The sum of the ratings is then multiplied by a certain number as directed by that category. The resulting number provides a graded percentage of the course's overall grade for that category.

Category	4 (Very Good)	3 (Good)	2 (Fair)	1 (Poor)	0 (Very Poor)
For the following ten items, add together the sum of the ratings and multiply them by 2.5 for this category's overall score.
Objectives: Verbage	Are the learning objectives written utilizing verbiage appropriate to the Bloom criterion?
	All objectives use verbiage appropriate to criterion.	Most of the objectives use appropriate verbage.	Half of the objectives use appropriate verbage.	Few of the objectives use appropriate verbage.	None of the objectives use appropriate verbage.
Objectives: Specificity	Are the learning objectives stated in a clear, well defined, and concise manner?
	All objectives use clear and concise wording.	Most of the objectives use clear and concise wording.	Half of the objectives use clear and concise wording. Alternate: The objectives are free of ambiguity but wordy.	Few of the objectives use clear and concise wording.	None of the objectives use clear and concise wording.
Objectives: Measurability	Do the learning objectives have an obvious and specific method to measure achievement?
	All objectives are clearly measurable.	Most of the objectives are clearly measurable.	Half of the objectives are clearly measurable.	Few of the objectives are clearly measurable.	None of the objectives are clearly measurable.
Objectives: Attainability	Are the learning objectives attainable by the students with effort in the time allowed?
	All objectives are attainable by students within the alloted time.	Most objectives are attainable by students within the alloted time.	Some objectives are attainable by students within the alloted time.	Few objectives are attainable by students within the alloted time.	None of the objectives are attainable by students within the alloted time.
Objectives: Relevance	Are the learning objectives focused on the desired behavior, or appropriate to Bloom's criterion?
	All of the objectives focus on the desired behavior and are appropriate to Bloom's criterion.	Some of the objectives focus on the desired behavior and are appropriate to Bloom's criterion.	Half of the objectives focus on the desired behavior and are appropriate to Bloom's criterion. Alternate: The objectives either are appropriate to Bloom's criterion or focus on the desired behavior, but not both.	Few of the objectives focus on the desired behavior and are appropriate to Bloom's criterion.	None of the objectives focus on the desired behavior and are appropriate to Bloom's criterion.
Objectives: Trackability	Are milestone measurements with time deadlines indicated?
	All objectives have milestone measurements and deadlines.	Most of the objectives have milestone measurements and deadlines.	Half of the objectives have milestone measurements and deadlines.	Few of the objectives have milestone measurements and deadlines.	None of the objectives have milestone measurements and deadlines.
Objectives: Difficulty	Are the learning objectives difficult but attainable?
	The objectives provide the right amount of challenge to the average student.	Some of the objectives provide the right amount of challenge to the average student.	Half of the objectives provide the right amount of challenge to the average student.	Few of the objectives provide the right amount of challenge to the average student.	The objectives are either far too easy or too hard for the average student.
Objectives: Communication	Are the learning objectives well communicated to the students?
	Objectives are easily accessible to students and worked into course activities.	Students have easy access to learning objectives, but are not worked into course activities.	Learning objectives are not made obvious to the students.	Learning objectives are available to students, but are difficult to find.	Students have no way of viewing the learning objectives for themselves.
Objectives: Delineation	Are learning objectives clearly outlined for the students?
	All of the objectives are obviously outlined for the students.	Some of the objectives are obviously outlined for the students.	Half of the objectives are obviously outlined for the students.	Few of the objectives are obviously outlined for the students.	None of the objectives are obviously outlined for the students.
Objectives: Ownership	Do students agree with the learning objectives? (Leave blank if not applicable.)
	Students are in full agreement with the learning objectives.	Students mostly agree with the learning objectives.	Students only somewhat agree with the learning objectives.	Students disagree with the learning objectives.	Students strongly disagree with the learning objectives.
For the following four items, add together the sum of the ratings and multiply them by 6.25 for this category's overall score.
Experiences: Practice	Do online activities require students to practice desired changes, and does a feedback mechanism exist?
	Online activities give students opportunity to practice the desired changes. A thorough feedback mechanism exists for these activities.	Students have ample opportunity to practice desired changes. Adequate feedback exists for these activities.	Students have an opportunity to practice desired changes, but there is little feedback beyond informing students what they answered correctly.	Although students have an opportunity to practice desired changes, no feedback exists.	Students are not given any sort of opportunity to practice or feedback.
Experiences: Reward	Do students receive motivation through timely-delivered rewards upon success?
	Students receive a significant and immediate reward for achievement, and rewards are not given if students do not meet the learning objectives.	Students receive reward for achievement, but it is either somewhat middling or delayed. Rewards are not given if students do not meet the learning objectives.	Students receive reward for achievement, but it is both middling and delayed. Rewards are not given if students do not meet the learning objectives.	Students receive little reward for achievement, and/or the rewards are greatly delayed. Rewards are not given if students do not meet the learning objectives.	Students receive rewards regardless of whether or not the student meets the learning objectives.
Experiences: Variety	Does the course use a variety of tools to enhance learning? Are they relevant to Bloom's criterion?
	Students receive a wide variety of experiences, all of them relevant to Bloom's criterion.	Students receive a fairly large variety of experiences relevant to Bloom's criterion. Alternate: Students receive a wide variety of experiences, but they are not entirely relevant to Bloom's criterion.	Students receive a middling variety of experiences relevant to Bloom's criterion. Alternate: Students receive a wide variety of experiences, but their relevance to Bloom's criterion is questionable.	Students receive a little variety of experiences relevant to Bloom's criterion. Alternate: Students receive a variety of experiences, but barely relevant to Bloom's criterion.	Students either receive no variety of experience or the experiences provided are not relevant at all to Bloom's criterion.
Experiences: Outcomes	Do tools restrict outcomes to the desired change? Does feedback exist to correct undesired outcomes?
	The tools selected and implemented restrict possible outcomes to achieving the desired change. Feedback also exists to correct unwanted outcomes.	Tools implemented restrict the possible outcomes to achieving the desired change, but there is little or no feedback to correct unwanted outcomes. Alternate: Tools implemented mostly restrict the possible outcomes to achieving the desired change, although adequate feedback exists to correct unwanted outcomes.	Tools implemented only somewhat restrict the possible outcomes to achieving the desired change, and there is little feedback to correct unwanted outcomes.	Tools implemented do not restrict the possible outcomes to achieving the desired change, and there is little feedback to correct unwanted outcomes.	The tools implemented do not restrict possible outcomes to achieving the desired change, and no feedback exists to correct unwanted outcomes.
For the following three items, add together the sum of the ratings and multiply them by 8.33 for this category's overall score.
Organization: Continuity	Are multiple opportunities continuously provided for students to practice the desired change?
	Students have multiple opportunities to practice skills necessary for the desired change continously throughout the course.	Students have multiple opportunities to practice skills necessary for the desired change, but only at a number of sections within the course.	Students have multiple opportunities to practice skills necessary for the desired change, but only at a couple of points within the course.	Students have only one opportunity to practice skills necessary for the desired change, but are free to practice at any time.	Students have only one opportunity to practice skills necessary for the desired change and have no option to practice at any time.
Organization: Sequence	Do activities build on each other with increasing breadth and depth? Is the desired level of complexity met?
	Activities build upon one another with increasing complexity as required by the desired level of Bloom's taxonomy.	Activities build upon one another with a subtle increase of complexity as required by the desired level of Bloom's taxonomy.	Activities build upon one another, but do not quite meet the desired level of Bloom's taxonomy.	Activities build upon one another subtly, but do not meet the desired level of Bloom's taxonomy.	Activities do not build upon one another and do not meet the desired level of Bloom's taxonomy.
Organization: Integration	Do activities exemplify the relationship of the desired change to other subjects/fields?
	Activities clearly exemplify the desired change in relation to other subjects and fields.	Activities adequately demonstrate the desired change in relation to other subjects and fields.	Activities cover the desired change's relation to other subjects and fields, but not with any detail.	Activities barely relate the desired change to other subjects and fields.	Activities do not relate the desired change to other subjects and fields.
For the following five items, add together the sum of the ratings and multiply them by 5 for this category's overall score.
Evaluation: Relevance	Do the evaluations used specifically evaluate the intended change?
	Evaluations very clearly evaluate the intended change and only the intended change.	Evaluations clearly evaluate the intended change with little deviation.	Evaluations do evaluate the intended change, but there is some deviation.	The evaluations signficantly deviate from evaluatiing the intended change.	The evaluations do not evaluate the intended change at all.
Evaluation: Baseline	Is a baseline evaluation used to assess the state of students before instruction?
	A complete baseline evaluation is performed at the beginning of the course.	A fairly thorough baseline evaluation is performed at the beginning of the course.	An evaluation is performed at the beginning of the course, but it only partially covers the desired change.	A highly abridged evaluation is performed at the beginning of the course.	No baseline evaluation is performed at the beginning of the course.
Evaluation: Successive	Are a series of evaluations used to assess student progress?
	A series of evaluations are used to assess student progress.	A fair amount of evaluations assess student progress.	A middling amount of evaluations assess student progress.	One or two evaluations exist to assess student progress.	The only evaluation to assess student progress appears at the end of the course.
Evaluation: Appropriative	Do evaluations use the same skills necessary to achieve the desired change?
	Evaluations examine all the skills necessary to the desired change.	Evaluations examine most of the skills necessary to the desired change.	Evaluations examine half of the skills necessary to the desired change.	Evaluations examine few of the skills necessary to the desired change.	Evaluations do not examine the skills necessary to the desired change.
Evaluation: Criterion Level	Do evaluations reflect mastery of the change at the desired Bloom criterion level?
	Evaluations are very well suited to the complexity of the desired criterion.	Evaluations are adequately suited to the complexity of the desired criterion.	Evaluations are somewhat suited to the complexity of the desired criterion.	Evaluations are little suited to the complexity of the desired criterion.	Evaluations are not at all suited to the complexity of the desired criterion.
For the following ten items, add together the sum of the ratings and multiply them by 2.5 for this category's overall score.
Design: Use of Frames	Does the course use frames, and do they interfere with bookmarking and navigation?
	Course does not use frames at all in its layout.	Course uses frames, but the design does not interfere with bookmarks.	Frame layout causes mild interference with bookmarking.	Frame layout causes mild interference with bookmarking.	Frames cause major issues with navigation and bookmarking.
Design: Overuse of New Tech	Does the course feature new, experimental, and potentially unsupported technologies?
	New and experimental technology used only as required by the subject matter.	New and experimental technology used for more than one major feature of course.	Significant and unnecessary amount of new and experimental technology used in course.	New and experimental technology unnecessarily comprises majority of course features.	Overreliance on new and experimental technology.
Design: Overused Animation	Does the course include distracting animations, scrolling text, or marquees?
	No scrolling text, marquees, or constant animation.	A few examples of scrolling text, marquees, or constant animation.	Significant amounts of scrolling text, marquees, or constant animation.	Large amount of scrolling text, marquees, or constant animation.	Highly distracting amounts of scrolling text, marquees, or constant animation.
Design: Complex URLs	Does the course have long URLs, URLs with mixed case letters, or both?
	Very simple URLs that can be entered by hand.	Course features a few complex URLs, or URLs for minor course features.	Course features a few complex URLs, particularly for major feature locations.	Course features a large number of highly complex URLs.	URLs are too complex for a student to quickly and reliably enter manually.
Design: Orphaned Pages	Are there any pages that lack navigation to the index or isolated due to dead links (orphaned)?
	Course has no orphaned pages.	Course has one or two orphaned pages.	Course has a significant number of orphaned pages.	Many orphaned pages exist in the course.	The majority of the course is orphaned.
Design: Scrolling Pages	Does the course feature pages longer than the height of one screen?
	The course contains no scrolling pages.	The course contains a few scrolling pages.	The course contains a significant amount of scrolling pages.	The majority of the course features long-scrolling pages.	Course includes many long-scrolling pages.
Design: Navigation Support	Does the course provide maps and navigation links on each page?
	All course pages have navigation links. Course map is also available.	All course pages have navigation links.	Most course pages have navigation links.	Few course pages have navigation links.	Course is mostly unnavigable from page to page.
Design: Link Decorations	Does the course feature inconsistent link colors and decorations?
	Course uses some form of standard link colors (blue for unvisited, purple or red for visited).	Course uses nonstandard link decorations, but these colors are internally consistent.	Course has a few different forms of nonstandard link decorations.	Course has many different forms of nonstandard link decorations.	Link decorations and colors have no form of consistency.
Design: Information	Does the course contain any information that is outdated?
	Course uses the most recent versions of references and has no outdated information.	Course has one or two outdated sources and references.	Course contains several oudated sources and references.	Course contains a significant amount of oudated sources and references.	Course relies on many sources and references which are outdated.
Design: Download Time	How long does it take to download the course on an average student's connection?
	Course takes less than ten seconds to access on the average student's connection.	Course takes more than ten seconds to access on the average student's connection.	Course takes more than fifteen seconds to access on the average student's connection.	Course takes more than thirty seconds to access on the average student's connection.	Course takes more than a minute to access on the average student's connection.
For the following seven items, add together the sum of the ratings and multiply them by 3.57 for this category's overall score.
Browsing: Selectable Areas	Does the course provide selectable areas to allow students access to information?
	All navigable tools are coherently and clearly marked.	Most navigable tools are coherently and clearly marked.	Some of the navigable tools are coherently and clearly marked.	Few navigable tools are coherently and clearly marked.	None of the navigable tools are coherently and clearly marked.
Browsing: User-Determined Order	Does the course allow users to access information as per the user's own preference?
	User has total control over the order in which information is accessed.	User can mostly control the order of accessed information.	User has some control over the order in which information is accessed.	User has little control over the order in which information is accessed.	User has no control over the order of accessed information.
Browsing: Site Maps	Does the course provide maps so that users can find where they are as well as links to other points of interest?
	Course provides a very detailed map which allows users to navigate to all segments.	Course provides a detailed map which allows user to navigate to most segments.	Course provides a map with most major points of interest included.	Course provides a map with very few navigable sections.	Course does not provide maps.
Browsing: Course Feedback	Does the course provide users with feedback on any significant time delays on accessing information?
	Course does not have any significant time delays. Alternative: The course provides notice for all significant delays.	Course provides notice for most delays.	Course provides notice for roughly half of the delays.	Course provides notice for only a few delays.	Course does not provide notice for any delays.
Browsing: Progress Feedback	Does the course inform users on their progress through the content?
	The course provides enough information for users to easily track their progress in the course.	The course provides adequate information for users to track their progress in the course.	The course provides some information for users, causing moderate difficulty in tracking their progress in the course.	The course little information for users, making it very difficult for them to track their progress in the course.	The course does not provide any information for users to their progress in the course.
Browsing: Arrangement	Is information arranged in a non-threatening manner so that users are not overhwelmed?
	All information is "chunked" in easily comprehensible portions.	Information is "chunked" to a reasonable degree, but could be optimized.	Information is "chunked" into segments that are a little too large.	There are only a few subdivisions within the course content.	Course content is monolithic in presentation and very overwhelming.
Browsing: Visual Feedback	Does the course provide visual feedback when a user makes a choice?
	Course provides ample feedback and comments for all chocies made.	Course provides adequate feedback.	Course provides some feedback, but not enough to keep a user well-informed.	Course provides very little feedback.	Course provides no visual feedback.
For the following four items, add together the sum of the ratings and multiply them by 6.25 for this category's overall score.
Media: Integration	Does the course integrate content across different media types?
	The course integrates multiple media types in an effective manner.	The course integrates an adequate number of media types effectively.	The course integrates only one or two media types, such as graphics.	The course integrates multiple media types, but does so in a distracting manner.	The course contains text only.
Media: Relevance	Does the course use media that is relevant, appropriate, and credible?
	All media used in the course is relevant and credible.	Most of the media used in the course is relevant and credible.	Half of the media used in the course is either irrelevant, inappropriate, or outdated.	Most of the media used in the course is either irrelevant, inappropriate, or outdated.	None of the media used in the course is relevant and credible.
Media: Consistency	Does the course provide access to each type of media in the same manner?
	Course is fully consistent with the methods used to access media.	Most of the media used has a consistent methd of access.	Half of the media used has a consistent methd of access.	Course has little consistency in the methods used to access media.	Course has no consistency in the methods used to access media.
Media: Control	Does the course use similar control icons for all types of media?
	All types of media used have the similar control icons.	Most types of media used have the similar control icons.	Half of the media used features similar control icons.	The control icons used for media varies greatly.	No two examples of media used in the course feature similar control icons.
For the following four items, add together the sum of the ratings and multiply them by 6.25 for this category's overall score.
Metaphor: Themeing	Is the course designed around a consistent metaphor or theme?
	A single metaphor or theme strongly ties the course content together.	The course content is mostly tied together by one or two themes.	There are a few themes or metaphors used throughout the course.	The course contains a number of themes or metaphors, weakly unifying the content.	The course has no unifying theme or metaphor.
Metaphor: Obviousness	Is the metaphor or theme obvious to users?
	The metaphors are easily understandable to the average user.	The metaphors are mostly understandable to the average user.	The metaphors are somewhat understandable to the average user.	The metaphors are too esoteric for the average user to understand.	The metaphors make no sense to the average user.
Metaphor: Applicability	Are the metaphors used applicable to the subject of the course?
	The metaphors fit very well with the course subject.	The metaphors are adequately relevant to the course subject.	The metaphors are somewhat fitting to the course subject.	The metaphors have little to do with the course subject.	The metaphors are not applicable to the course subject in any way.
Metaphor: Navigation	Are navigation tools within the course consistent with the metaphor?
	Navigation tools are fully relevant to the themes or metaphors used.	Navigation tools are mostly relevant to the themes or metaphors used.	Navigation tools have some relation to the themes or metaphors used.	Navigation tools have little relation to the themes or metaphors used.	Navigation tools have no relation at all to the themes or metaphors used.
For the following two items, add together the sum of the ratings and multiply them by 12.5 for this category's overall score.
Access: Searchability	Are users provided with search tools?
	The course includes multiple search tools, including an interactive search engine.	The course includes a few search tools, including an interactive search engine.	The course provides a few non-interactive search tools.	The course provides a single non-interactive search tool such as a master table of contents.	The course does not provide any search tools.
Access: Media	Does the course provide a way to search for specific media used within the course?
	The course includes search tools, including interactive search tools, that allow users to find any piece of media used.	The course includes search tools, including interactive search tools, that allow users to find most pieces of media used.	The search tools provided are either non-interactive or can only search for one type of media.	The course includes only one non-interactive search tool for media, such as a list of figures.	The course does not allow users to search for specific pieces of media.
For the following six items, add together the sum of the ratings and multiply them by 4.16 for this category's overall score.
Guidelines: Maps and Cues	Does the course provide cues such as menus and maps to help users conceptualize course content?
	Course enables users full control over content navigation at any time, and provides a full table of contents.	Course enables user control over content navigation most of the time, and provides a full table of contents.	Course enables user control over content navigation some of the time, and provides a an adequate table of contents.	Course enables little user control over content navigation, and provides an adequate table of contents.	Course only provides a table of contents, if any.
Guidelines: Fixed Format	Does the course use a consistent format or layout to keep information in the same place across course content?
	Course has complete consistency in its design. All subjects have a layout coherent with one another.	Course has adequate consistency in its design. Most subjects have a layout coherent with one another.	Course has some consistency in its design. There is notable deviation in layout among subjects.	Course has little consistency in its design. There is a large amount of deviation in layout among subjects.	Course has no consistency in its design.
Guidelines: Location	Are users provided with information to let them know their location in the course's content?
	Course layout provides full contextual information for users. Users know were they are at all times.	Course layout provides adequate contextual information for users. Users know were they are most of the time.	Course layout provides some contextual information for users. Users know were they are roughly half of the time.	Course layout provides little contextual information for users. Users do not know where they are most of the time.	Course provides no sort of contextual information for users.
Guidelines: Indicators	Does the course use location indicators and progress reports to inform users of their location?
	The course has many indicators, including an on-demand progress report, to aid students in determining their progress in the content.	The course has a number of indicators, including a progress report, to aid students in determining their progress in the content.	The course has some indicators to aid students in determining their progress in the content.	The course has few indicators to aid students in determining their progress in the content.	Course does not feature any sort of location or progress indicator.
Guidelines: Consistency	Does the course use consistent fonts, font sizes, and font decorations to represent certain types of information?
	Course uses fully consistent font styles for certain types of information throughout the course content.	Course uses mostly consistent font styles for certain types of information throughout the course content.	Course has a fair amount of consistency in font styles used throughout the course content.	Course has little consistency in font styles throughout the course content.	Course does not feature any sort of consistent font style throughout the course content.
Guidelines: Descriptions	Does the course provide immediate or adjacent descriptions of program controls wherever controls are used?
	Course provides immediate and adjacent descriptions of program controls wherever such controls appear.	Course provides on-demand descriptions of program controls wherever such controls appear.	Course provides descriptions for only some of the program controls used in the course content.	A help page provides descriptions of program controls. Users must navigate away from course content to access this page.	Course does not provide any descriptions of program controls.