Online Assessment: To Measure or Not to Measure

Assessment, both formative and summative is an important part of education. Unfortunately, distance education programs currently often lack appropriate strategies to test their effectiveness, which is one of the principle reasons for using formative assessments (Lockee et. al., 2002). It is essential that we consider ways to evaluate the effectiveness of online programs to ensure learning objectives are being met. 

Instructional design processes call for the development of assessment strategies early in the planning stages of a learning unit, whether online or face-to-face (Morrison et. al, 2011). Developing assessments along with your learning objectives can help you clarify what you want learners to learn, and can help you find appropriate ways to ensure your instruction is effective in meeting these objectives.

While it is necessary to measure the effectiveness of online courses, it is not always necessary or even productive to measure the effectiveness of every activity performed by students in an online environment. When attempting to measure student performance, some elements of online learning are difficult to score. So much goes on in online education, scoring all of it can be a monumental task. Bonk (2010) cautions listeners against attempting to grade everything students do online. While some tasks lend themselves to automated scoring, such as objective tests, constructed-response activities including blogs, discussion boards, and written assignments currently require human intervention to score (Saint-Germain, 2009). This type of assessment requires construction of a rubric to measure student performance. Scores are still often very subjective in nature. Bonk (2010) suggests alternatives to time-intensive grading of every such activity. Scoring of discussion board activity, for example, might simply be a participation grade. Students receive points for doing the activity, regardless of quality. Other scores might be based on a mixture of quality and quantity. Students expect and even demand feedback on their performance. When you do choose to score an online activity, be sure to supply meaningful feedback along with the score.

Effective online instruction will have built-in assessment measures to ensure the course is effective at meeting learning objectives, but effective online educators cannot and should not spend all their time scoring student activities for the sake of assigning a grade. It could be argued that new learning methods call for new educator strategies, and perhaps the time for measuring student performance by a score. Online education provides the opportunity for students to document learning through other means, such as through an ePortfolio or wiki site. Colleges and prospective employers can review such indicators to see a student’s growth over time, and these methods provide a much greater insight into the depth and diversity of a student’s learning experiences than a transcript of grades could every provide.

References

Bonk, Curtis (2010). Assessing Student Online Learning. Indiana University. Retrieved April 27, 2012 from http://www.youtube.com/watch?v=d0nfDOPaw_8

Lockee, Barbara, Moore, Mike, and Burton, John (2002). Measuring Success: Evaluation Strategies for Distance Education. Educause Quarterly. Retrieved April 27, 2012 from http://bb91a.tamut.edu/bbcswebdav/pid-8367-dt-content-rid-107047_1/xid-107047_1

Morrison, G., Ross, S., Kalman, H., and Kemp, J. (2011). Designing Effective Instruction. John Wiley & Sons, NJ.

Saint-Germain, Michelle (2009). Assessment Quickies #6: Matching Assessment to Teaching and Learning. California State University. Retrieved April 27, 2012 from http://www.youtube.com/watch?v=0S22TOBZGIM&feature=related

Action Research Draft Implementation Plan

To see a PDF of my Action Research Draft Implementation plan, click here.

Action Research Plan - Description of the Problem

What effect do substitute teachers have on student achievement and what are the ways schools can better utilize substitutes to minimize student impact?

I will start by examining quantitative data by comparing TAKs data with teacher absences by teacher at one of the local ISD's to determine if students performed better in classes where teachers had fewer absences.

I also plan to use focus groups and Survey Monkey to discuss with teachers and substitutes the ways substitutes are currently trained and utilized and what changes could be made to better prepare substitutes or plan better for teacher absences to ensure students are still learning when the substitutes are in charge.

The motivation for this study comes from my own work and personal experiences as a substitute. I often find that teachers have left no lesson plan or have planned for students to complete "busy work" while they are absent. I define "busy work" as work which will likely not be graded and which has little to do with the subject matter students are currently learning. This could include "fun" movies, repetitious writing, word searches, crossword puzzles, mazes, or coloring sheets. I have often felt I was being utilized more as a babysitter or a police warden than as a teacher. 

My concerns have been increased by the initial research I have conducted, which indicates that students spend on average the equivalent of a complete school year studying with a substitute during their K-12 education. These studies indicate that a direct correlation exists between lower student achievement and teacher absence (Elizabeth, 2007; Glatfelter, 2006; Damle, 2009; Granowicz, 2010).

I realize that many substitutes do not have the teacher training which I have. However, I feel confident that with some basic training on the part of the substitute and some better planning on the part of the teacher, substitutes can better carry out their assigned tasks.  This should ensure that the impact on learning is minimized under their care. My literature research also demonstrated that in school districts where substitutes are trained and teachers are taught how to prepare for them, the impact is minimized (O’Connor, 2009, Deay & Bontempo, 1986; Cardon, Tippetts, & Smith, 2003; Elizabeth, 2007).

References:

Cardon, Peter W.; Tippetts, Zachary; Smith, Geoffrey G. (2003). “The Effectiveness of Substitute Teacher Training: The Results of a Utah Study.”

Damle, Ranjana (2009). “Investigating The Impact Of Substitute Teachers On Student Achievement: A Review Of Literature.”

Deay, Ardeth & Bontempo, Barbara (1986). “Helping Substitute Teachers Contribute to School Effectiveness,” The Clearing House.

Elizabeth, Jane (2001). “A Substitute for Education: Classroom-crippling shortage leads to unusual solutions,” Pittsburg Post-Gazette.

Elizabeth, Jane (2001). “Institute helps teacher replacements learn to teach,” Pittsburg Post-Gazette.

Glatfelter, Andrew Gary (2006). “Substitute Teachers as Effective Classroom Instructors,” A Doctoral Dissertation, UCLA.

Granowicz , Stephanie (2010). “Impact of Substitute Teachers on Student Achievement,” Knight Life.

O’Connor, Kevin (2009). “No Substitute Left Behind,” Principal.

Reflections on choosing an Action Research Topic

I was very interested in the subject of Chapter 2 in Leading With Passion and Knowledge. Chapter 2 covered "Passions That Drive Your Journey," and discussed how you could find topics with which you could begin your action research. I found that all nine of the suggested areas could as easily apply to a teacher's research as they could to an administrator's, although the examples given were largely more applicable to administration (Dana, 2009, pp. 65-66).

Dana's caution about making sure that my wonderings did not focus on changing other people was good advice (p. 64). The only person I can change is me. I may wonder about what I can do to make homework more relevant for my students, but I cannot wonder how I can make my students do their homework and turn it in.

I also appreciated the advice concerning choosing a question which was open-ended. I found myself drafting my original wonderings in the form of yes/no questions (p. 67). That sort of question cannot lead to a great research project because there is only one right answer to a question like that.

The third piece of advice Dana gives that hit home with me was the advice to be sure that my topic would have an impact on learning activities (p. 67). Since this is the most important issue faced by schools today, projects centered around student achievement are going to get the most attention and the most support. Projects which focus on teacher learning are the second most important issue since teachers need to constantly learn and improve to be effective in educating students.

I really felt like I learned a lot from this week's activities. I feel better prepared to choose and begin my research project after the activities this week, and more confident that my topic will gain the interest and support of the districts where I work.