share what you know!

Internal quality review process

My institution has recently adopted Quality Matters standards for Online courses. We will begin by applying the standards/ rubric, but developing our own internal process. Does anyone have anything to share on how they have developed a similar process. Lessons learned, best practices? Any feedback is much appreciated.

1 member likes this

▶ Reply to This

Replies to This Discussion

Permalink Reply by Dan Feinberg on January 12, 2015 at 1:11pm

Hi Tracy--

Here at Open SUNY's Center for Online Teaching Excellence, we’ve had some experience developing a process, and are in the middle of trying to do it again. We aren’t a campus ourselves, but we work with almost all the campuses across SUNY’s 64-member system.

In the past we tried setting up a cross-campus review process, which met limited success. We did manage to train a couple hundred faculty and instructional designers in how to apply the quality matters rubric, but we completed very few reviews. The issue was faculty time and motivation. We set up a process where faculty members would review each other’s’ courses for free, as a sort of reciprocal agreement. Faculty were all very well intentioned, but many of them failed to complete the reviews in a timely manner. They meant to help, but life got in the way! Some of this is the nature of people’s busy lives, but some of this was due to that fact that we did not incentivize or reward participation properly. We never figured out exactly how to reward and recognize participants.

We’ve recently set out again to create a system of regular review and refresh of courses, with a few major changes:

We abandoned the Quality Matters rubric in favor of a ‘home grown’ alternative. This was a long debate and a lot of effort, but we wanted a rubric better matching our specific needs. We built into the rubric mechanisms for data collection and suggestions on how to address any identified issues.
The process is campus driven. Our campuses have different cultures and needs, and a top-down, system-wide initiative wasn’t going to work. Campuses have better ability to customize the rubric and process to their needs, as well as incentivize participation.
We have a better infrastructure for recognition. We are launching an extensive ‘badging’ or micro-credentialing platform that will recognize people’s efforts. We have also set up system-wide events where participants can be honored. We are hoping that these more formalized elements will increase the appeal to reviewers.
We made the process more focused on actions. Quality Matters reviews are assigned a score that passes or fails overall. We have done away with scores, and focused more on developing an action plan that offers suggestions for improvement. The course now doesn’t ‘pass’ our process at all, but instead every course has elements highlighted where a ‘refresh’ could help.

We haven’t fully rolled out the entire program, so I’m sure there will be unexpected pitfalls and pleasant surprises along the way. All of our work is licensed with a Creative Commons license, so feel free to take a look. If you would like to talk more, feel free to reach out to us at Contactcote@suny.edu or me directly at dan.feinberg@suny.edu

Take care,

Dan Feinberg

Open SUNY COTE

▶ Reply

Permalink Reply by Tracy Cofiell Miller on January 12, 2015 at 1:34pm

This is so helpful!!! Thank you. I'll be sure to tell the team about it and share as we learn more ourselves.

▶ Reply

Permalink Reply by Robert Piorkowski on January 12, 2015 at 2:24pm

In addition to Dan's comments above, I'd like to offer a few "lessons learned" from our experience with training faculty to apply the QM rubric:

1. Faculty need a clear explanation of the narrow scope of the rubric, in that it looks at instructional design only, not other elements of the teaching/learning process (course management, feedback to students, instructor effectiveness, institutional resources, etc.);

2. It is helpful to clarify the communication path of "the score," as for some, it could be a reason for anxiety;

3. Faculty appreciate recognition for participating in the process. It is a professional development activity and should be recognized by the campus as such. If reviewing another's course, this could be deemed community service as the process requires time and skill;

4. The timing of a review is an important factor in implementation. October and February/March work well;

5. Encouraging informal self-reviews could be helpful in raising awareness and "tuning in" faculty to the rubric;

6. Incentives for reviewers go a long way toward successful participation. Stipends, release time, etc.

Good luck!

Rob Piorkowski

Assistant Director of Online Learning

Open SUNY COTE

▶ Reply

Permalink Reply by Tracy Cofiell Miller on January 13, 2015 at 10:50am

Thank you Rob!

Regarding point 1. Yes I can see where the rubric review process could leak into some sort of evaluation of the teaching /learning process. We have recently implemented a Teaching Online Self Assessment (outside the QM process). The first round of the self-assessment was encouraged by a dean and voluntary. The results were very favorable. Right now we are creating a personalized framework of workshops, resources, and consultations for the responders if they wish to take advantage of them. Faculty members at a self-determined level of mastery, would be good candidates for internal reviewers, either from there experience or their own development from going from good to great, specifically in the area of design.

▶ Reply

Permalink Reply by Ian August on January 13, 2015 at 4:57pm

Hey guys,

I love the idea of the open SUNY rubric not being a pass or fail. Having a low score on a QM review can leave a faculty in bad spirits.

I also love the idea that open SUNY's rubric looks at course delivery. I never understood why QM only looked at course design as design is only part of the story.

I think incentives go a long way to have faculty assist other faculty. And the other option, as Tracey mentioned, is self assessment.

Ian