Online Assessment: To Measure or Not to Measure

Assessment, both formative and summative is an important part of education. Unfortunately, distance education programs currently often lack appropriate strategies to test their effectiveness, which is one of the principle reasons for using formative assessments (Lockee et. al., 2002). It is essential that we consider ways to evaluate the effectiveness of online programs to ensure learning objectives are being met. 

Instructional design processes call for the development of assessment strategies early in the planning stages of a learning unit, whether online or face-to-face (Morrison et. al, 2011). Developing assessments along with your learning objectives can help you clarify what you want learners to learn, and can help you find appropriate ways to ensure your instruction is effective in meeting these objectives.

While it is necessary to measure the effectiveness of online courses, it is not always necessary or even productive to measure the effectiveness of every activity performed by students in an online environment. When attempting to measure student performance, some elements of online learning are difficult to score. So much goes on in online education, scoring all of it can be a monumental task. Bonk (2010) cautions listeners against attempting to grade everything students do online. While some tasks lend themselves to automated scoring, such as objective tests, constructed-response activities including blogs, discussion boards, and written assignments currently require human intervention to score (Saint-Germain, 2009). This type of assessment requires construction of a rubric to measure student performance. Scores are still often very subjective in nature. Bonk (2010) suggests alternatives to time-intensive grading of every such activity. Scoring of discussion board activity, for example, might simply be a participation grade. Students receive points for doing the activity, regardless of quality. Other scores might be based on a mixture of quality and quantity. Students expect and even demand feedback on their performance. When you do choose to score an online activity, be sure to supply meaningful feedback along with the score.

Effective online instruction will have built-in assessment measures to ensure the course is effective at meeting learning objectives, but effective online educators cannot and should not spend all their time scoring student activities for the sake of assigning a grade. It could be argued that new learning methods call for new educator strategies, and perhaps the time for measuring student performance by a score. Online education provides the opportunity for students to document learning through other means, such as through an ePortfolio or wiki site. Colleges and prospective employers can review such indicators to see a student’s growth over time, and these methods provide a much greater insight into the depth and diversity of a student’s learning experiences than a transcript of grades could every provide.

References

Bonk, Curtis (2010). Assessing Student Online Learning. Indiana University. Retrieved April 27, 2012 from http://www.youtube.com/watch?v=d0nfDOPaw_8

Lockee, Barbara, Moore, Mike, and Burton, John (2002). Measuring Success: Evaluation Strategies for Distance Education. Educause Quarterly. Retrieved April 27, 2012 from http://bb91a.tamut.edu/bbcswebdav/pid-8367-dt-content-rid-107047_1/xid-107047_1

Morrison, G., Ross, S., Kalman, H., and Kemp, J. (2011). Designing Effective Instruction. John Wiley & Sons, NJ.

Saint-Germain, Michelle (2009). Assessment Quickies #6: Matching Assessment to Teaching and Learning. California State University. Retrieved April 27, 2012 from http://www.youtube.com/watch?v=0S22TOBZGIM&feature=related

Playing Games in the Classroom – the Role of Simulations and Game Play in K12

I have long been a proponent of the use of video-games in education. As a mother of students with learning challenges, I was often surprised at the intensity my sons applied to their video-game pursuits; failing and retrying a multitude of times without hint of frustration. This same intensity was absent from their school applications. My sons frequently failed to apply themselves to a school based assignment out of fear of failure no more threatening than that which they faced undauntingly during gaming. Seeing this contradictory behavior made me realize that something about the virtual environment of their games engaged them more deeply in the process while disengaging them from the stigma associated with failure. They approached the game with a determination to conquer it. I am convinced that this engagement and determination is what is missing from traditional educational approaches. If we can find a way to simulate that through educational use of game play in the classroom, I believe we can revolutionize education.

Few studies have been conducted on the effective use of video-based game play in the classroom. Part of the reason for this is the negative attitude toward game play by educators in general. “Many school leaders and teachers react negatively to video games and gaming culture, bashing video games as diversionary threats to the integrity of schooling or as destructive activities that corrupt moral capacity and create a sedentary, motivation-destroying lifestyle” (Halverson, 2005). Halverson goes on to say that the problem is exacerbated by the standards-driven environment we have embraced in the light of No Child Left Behind and other such legislation. “Standards specify what to teach; school leaders and teachers construct efficient pedagogies and learning environments to teach it.”

An and Bonk (2009) discuss the components necessary for developing educational games that will engage student in learning. They are proponents of a context-based rather than content-based approach. By this the authors mean that learning needs to be authentic and meaningful, surrounding a realistic situation or problem, rather than disconnected facts to be committed to rote memory. “Context is more important than content since learning is a process of ‘developing abilities to see, think, do and be in the world,’ rather than accumulating discrete facts (Squire, 2005b, p. 19).” This seems to be in direct opposition to the standards-based curriculum of the classroom, which demands a content-first approach. However, if approached correctly, I believe video-games and simulations can be used to teach directly to standards and can improve student retention and learning.

Steen (2008) cites research that demonstrates that learning increases proportionately with our interaction with the material. According to his example, a teacher utilizing visuals with lecture and textbook reading assignments might expect students to retain 50% of the materials being taught. This teacher can dramatically increase student retention to 70% by incorporating a class or small group discussion. Learning increases to 80% if students are allowed to experience the material. This is the realm of the video-game.

Part of the resistance to video-game use in the classroom is that there is not an efficient way to assess or measure learning, and in fact, students might learn at different rates or fail to apply what they have learned to school-related concepts. Halverson (2005) states that video games “provide inefficient and unpredictable environments for learning school-based material and have learning outcomes that are difficult to map onto curriculum standards. Learning in endogenous video games can be a protracted and indirect affair with a steep learning curve when compared with standard curriculum units on mathematical fractions, Egyptian history, or European expansion.”

Halverson (2005) goes on to explain that the cure for this lies in the way teachers facilitate the lesson. He feels that teachers can extract valuable lesson plans from existing commercial video games through several steps. His first suggestion involves mapping the learning potentials of commercial games to existing standards-based content. “Commercial endogenous games require an integrated lesson design that incorporates the depth of gaming insights into standards-based school environments.”

The second step outlined is to change the structure of the traditional classroom to allow for facilitation of learning from the game to derive the desired content. “The role of the learning environment in a traditional school setting is to provide a context to make structured content accessible to students; the role of the learning environment in an endogenous game-based setting is to scaffold prompts for helping students construct legitimate analogies between what can be learned in the game and what schools need to teach” (Halverson, 2005).

Next, Halverson (2005) suggests using the built-in risk-taking and controlled failure of the games as an authentic measurement for assessment of learning. “Designing environments to integrate games into schooling can thus draw on the assessment devices already built into games. The technology of multi-player gaming, for example, generates tangible records of prior game moves in the form of discussion threads that can be used to spark reflection on the assumptions behind earlier game moves (see, for example, the Rise of Nations Universe site). Learning environment designers can use these public representations of game-based information to discuss school-based learning outcomes. The arguments players develop online to defend in-game moves open valuable windows into the players' thinking processes. The outcomes of game-play also provide authentic artifacts of student learning that can be used as summative evaluations of learning.”

Halverson’s (2005) final suggestion may seem a bit over the top, but it makes sense if educators are to attend to the other suggestions. He proposes that in order to best learn how to use commercial games in education, educators need to play them. “…nowhere is the current generational gap in technology greater than in game literacy, and while asking school leaders and teachers to play commercial video games may be a stretch, integrating game-based learning experiences in their professional development may help them see the merits of gaming from the inside.”

I am convinced that educators need to devote some time to research the potential for use of commercially available video-games for education, and instructional designers need to become aware of the need for authentic game-based learning experiences and build games designed to meet standards-based instructional needs which are engaging for students to play.

References

An, Yun-Jo and Bonk, Curtis J. (2009). Finding that SPECIAL PLACE: Designing Digital Game-Based Learning Environments. TechTrends, Vol. 53, No. 3.

Halverson, Richard (2005). What Can K-12 School Leaders Learn from Video Games and Gaming? Innovate. Retrieved April 22, 2012 from http://www.innovateonline.info/index.php?view=article&id=81.

Steen, Henry L. (2008). Effective eLearning Design. Merlot Journal of Online Learning and Teaching, Vol. 4, No. 4.

Teachers Learning Online – Online Professional Development Courses about Online Teaching and Learning

Online learning, e-learning, u-learning, distance education. Whatever you call it, online education is one of the buzzwords in the educational community today. Debate over the effectiveness of online education has the educational community reexamining best practices and rethinking instructional strategies and methodology to provide excellent online educational experiences that in many cases meet or exceed the expectations for student performance in traditional face-to-face courses. In order to meet the demand for high-quality, highly engaging online courses, teachers need training about online education. Demands on teacher time and a lack of appropriate local educational opportunities has prevented teachers from improving their online teaching skills in the past, but now classes are being offered for these teachers the same way their students get the classes – online over the web.

Educating online teachers via online courses makes sense in a number of ways. When a teacher takes an online course, it allows them to experience online education from a student’s perspective. Teachers can experience the same frustrations or the same conveniences their students discover. Teachers increase their own online skills when their training is provided over the web. Online professional development meets the demand for just-in-time, just-enough, just-for-me training. Teachers can pick and choose those courses which apply to their interests, technology comfort level, and domains so training covers technology applications which they will actually use. These courses are also often offered on-demand so teachers can schedule training around their already crowded schedules, and can take courses from virtually anywhere they have internet access, including from their web-enabled phone.

Beyond the convenience, online professional development has the capacity to meet ongoing needs of teachers attempting to implement what they have learned into their courses in real-time. Dede et. al. (2009) discuss the trend that creates this need. “The need for professional development that can fit with teachers’ busy schedules, that draws on powerful resources often not available locally, and that can create an evolutionary path toward providing real-time, ongoing, work-embedded support has stimulated the creation of online teacher professional development (oTPD) programs.”

Online professional development is increasingly in demand and may become essential in providing the quality and kinds of training teachers need. Lebec and Luft (2007) comment, “Motivations for teachers to seek such opportunities are numerous. Dilemmas such as heavy instructional demands with minimal preparation time (Darling-Hammond & Cobb, 1996), accessibility to professional development in rural settings, and lack of institutional funds to send instructors to high quality courses or to cover their time away (Barkley & Bianco, 2001) often limit opportunities for teachers seeking additional training… This impetus, coupled with existing regional shortages of certified instructors in domains such as the physical sciences (Choy, 1993), makes the easily accessible online environment attractive as an expedient means of gaining discipline-specific training (Bowman, Boyle, Greenstone, Herndon, & Valente, 2000; Herbert, 1999).”

There are many excellent resources available to teachers today looking for professional development options online. The Sloan Consortium (http://www.sloanconsortium.org) is committed to excellence in online education. Their mission includes training for online educators which is available both face-to-face as workshops and online as webinars. Powerful Learning Practice (http://www.plpnetwork.com) is an online professional development provider which trains teachers in 21^st century teaching skills, including teaching online. This resource includes a year-long course called the Connected Learner Experience which provides webinar trainings and collaboration with other educators. There are also individual online classes offered.

Educators have their own social network at Classroom 2.0 (http://www.classroom20.com). Forums and groups here give educators the opportunity to participate in professional learning communities online, while virtual conferences and live and recorded webinars offer professional development opportunities.

References

Classroom 2.0 (2012). http://www.classroom20.com

Dede, Chris, Ketelhut, Diane J., Whitehouse, Pamela, Breit, Lisa, and McCloskey, Erin M. (2009). A Research Agenda for Online Teacher Professional Development. Journal of Teacher Education. Retrieved April 17, 2012 from http://jte.sagepub.com/cgi/content/abstract/60/1/8

Lebec, M., & Luft, J. (2007). A mixed methods analysis of learning in online teacher professional development: A case report. Contemporary Issues in Technology and Teacher Education, 7(1), 554-574. Retrieved April 17, 2012 from http://www.citejournal.org/vol7/iss1/general/article1.cfm

Powerful Learning Practice (nd). http://www.plpnetwork.com

The Sloan Consortium (2012). http://www.sloanconsortium.org

Examples of Technology as a Scaffolding Tool for K12 Educators

Scaffolding is defined as “tutoring or other assistance provided in a learning setting to assist students with attaining levels of understanding impossible for them to achieve without assistance” Brush and Saye (2002). Scaffolds in a classroom setting can be provided in a variety of ways, from manipulatives to handouts to teacher interventions such as questioning techniques.

Brush and Saye (2002) determined there are two types of scaffolding, hard and soft. They define hard scaffolding as those techniques which are static and can be anticipated and produced in advance of learning, such as handouts and manipulatives. Soft scaffolding is dynamic in nature. The teacher uses these techniques to support student learning on the spot based on the needs of the individual student. Questioning techniques would be an example of this type of scaffolding. Technology has some powerful implications as a potential tool to assist teachers in providing scaffolding for learners, particularly in the area of hard scaffolds.

Sharma and Hannafin (2007) discuss the use of technology-enhanced learning environments (TELEs) to support learner needs. “Two important affordances of computer systems are the ability to constrain user actions through predefined rules and the ability to store large amounts of data. By directing attention on important task features, software scaffolding may prevent learners from engaging in unnecessary, misleading, or unproductive tactics.” They go on to describe practical ways to use technology to focus on the areas students are likely to misunderstand and to provide access to a variety of problem solving strategies so students can attempt to solve problems using different approaches.

I have seen technology used in this fashion. For example, Pearson’s SuccessMaker software uses a variety of approaches in assisting students learning mathematics. Students are introduced to math word problems through written text and speech. Important numbers are highlighted to draw students’ attention, and students are asked to fill in numbers and symbols in an equation to be solved. If students request help or answer the problem incorrectly, the program offers additional scaffolding by first filling in the equation for students to solve. If this scaffold is insufficient to allow for student success, further scaffolding is provided in the form of pictures or other concrete examples representing the problem. For example, if the problem involved determining how many birds were left on a fence after some flew away, the software might display an animation of the original birds sitting on a fence and the suggested number flying away, allowing students to visualize the concept of subtraction in a concrete way. This is an example of hard scaffolding, as the program can only provide those examples pre-programmed by the instructor, but it is more dynamic than some uses of technology for scaffolding in that the amount of scaffolding provided varies based on the needs of the individual student, and the scaffolding is designed to fade over time as student proficiency increases.

GE and Land (2004) suggest that technology can be used to scaffold problem-solving processes by providing an opportunity for reflection and metacognition. One such use of technology is found in blogging. “A blog can become much more than an online diary and has countless instructional applications” (Kajder & Bull, 2003). In addition to offering an opportunity for students to reflect or participate in in-depth discussions among themselves, Kajder goes on to state that blogging is powerful because it provides students with an authentic writing platform with a real audience, immediate visibility, the opportunity to receive feedback, a chance to review how their writing has developed over time, and the ability to experiment with multiple forms of communication such as multimedia.

Ragan (nd) describes requiring a timely response to learning from students as one of ten “best practices” for online instructors. Classroom instructors can use blogging as a method of requiring a timely response from students. An expansion on the idea of a written journal, students can post reflections to their learning, questions they still have about what was learned, important vocabulary, examples, and more on their blog pages. They can connect what they have learned to new ideas online through the use of hyperlinking. But perhaps the most powerful use of this technology is the ability for students to interact with each other’s posts, commenting and expanding on the ideas of their peers. By requiring a regular blog entry to be kept, instructors can see areas where students have faulty understanding or need additional supports, and peer interaction provides a built-in scaffold system. Thus this tool provides both hard and soft scaffolds for students.

Technology promises to provide additional opportunities to support student learning as teachers become familiar with new applications and expanded technology capabilities. Computers are becoming more complex and “smarter”, increasingly more responsive to the needs of the user. As new applications become available, scaffolding will more truly mimic the support of an expert teacher, providing a greater range of soft scaffolds. This will increasingly free the time of the teacher to facilitate additional student needs and fill in learning gaps. While technology will never replace the need for human interaction, the amount and complexity of scaffolding tasks that technology is able to provide can only increase based on current innovations. 

References

Brush, Thomas A. and Saye, John W. (2002). A Summary of Research Exploring Hard and Soft Scaffolding for Teachers and Students Using a Multimedia Supported Learning Environment. The Journal of Interactive Online Learning, Vol. 1, No. 2. Retrieved April 15, 2012 from http://ncolr.org/jiol/issues/pdf/1.2.3.pdf. 

GE, Xun and Land, Susan M. (2004). A Conceptual Framework for Scaffolding Ill-Structured Problem-Solving Processes Using Question Prompts and Peer Interactions.

Kajder, Sara and Bull, Glen (2003). Scaffolding for Struggling Students: Reading and Writing with Blogs. Learning & Leading with Technology, Vol. 31 No. 2. Retrieved April 15, 2012 from http://heartlandaeatoc.pbworks.com/f/Scaffolding+for+Struggling+Students+Using+Blogs+and+Wikis.pdf.

Sharma, Priya and Hannafin, Michael (2007). Scaffolding in Technology-Enhanced Learning Environments. Interactive Learning Environments Vol. 15, No. 1, pp. 27-46.

Ragan, Lawrence (nd). 10 Principles of Effective Online Teaching: Best Practices in Distance Education. Distance Education Report.

Exemplary Online Educators and a Community of Inquiry

The readings this week centered around best practices for educators in online teaching. Two of the readings (Garrison et. al., 2000; Perry & Edwards, 2005) discuss the importance of creating a “community of inquiry” in an online learning environment. Garrison defines three intersecting areas which create this community; namely a social presence, a cognitive presence, and a teaching presence.

Although the remaining reading by Liu et. al. (2005) examined actual practice rather than developing theory, the authors categorized instructor roles into four categories to determine how instructors interact with students and the learning environment. Three of the four categories, in my opinion, overlap the areas established by Garrison. The four roles listed by Liu were Pedagogical, Social, Managerial, and Technical.

The pedagogical role intersects with Garrison’s idea of the cognitive presence. The instructor interacts with the course to provide content and interacts with the students to facilitate learning. “The pedagogical roles of online instructors revolve around facilitating educational process for students’ understanding of critical concepts, principles, and skills. Such tasks include encouraging students’ knowledge-sharing and knowledge-building through interactive discussion, designing a variety of educational experiences, providing feedback, and referring to external resources or experts in the field.” Perry and Edwards (2005) focus on the cognitive presence of the teacher, examining three characteristics of online teachers within the cognitive domain. Exemplary online teachers were found to be challengers, affirmers, and influencers. These traits overlap well with the roles of feedback-giver and interaction-facilitator described by Liu in the pedagogical arena.

I feel that the courses I’ve taken with Dr. Aworura have had strong cognitive presence. Dr. A. does an excellent job of creating a “triggering event” (Garrison et. al., 2000) and facilitating students’ discussion and exploration of the topic to allow us to connect and apply new ideas.

Obviously, the social presence and the social roles of instructors overlap. Interestingly, although Garrison shows this to be of equal and overlapping importance to the other two areas in a community of inquiry, Liu’s report indicates that this area tends to be looked at as of less or minor importance by many of the online educators involved in that study. Liu cited lack of awareness of the importance of this role, concern about time constraints, and lack of technology as reasons for this apparent apathy toward developing a social presence.

In our program, I feel that great importance has been placed on the social aspects of our learning community. We take time at the beginning of each course to greet and get to know one another. Because of this, I’ve come to know many of my fellow students in a casual way and have a picture of them in my mind when I’m discussing course material with them. Our class discussions are strong, lively, and interactive. Students engage in “expressive but responsive, skeptical but respectful, challenging but supportive” (Garrison et. al., 2000) discussions where we encourage one another to think outside the box and dig deeper for understanding. I have enjoyed these exchanges and grown as a person because of them. Group activities have allowed us opportunities to work with one or more students in a closer way. Although the dynamics of asynchronous communication often complicate this process (and in some cases make it downright impossible), I have worked in several groups where we have had a great flow of ideas and have complemented one another’s strengths. This collaborative learning meets the goal specified in Garrison as drawing “learners into a shared experience for the purposes of constructing and confirming meaning.”

The teaching presence isn’t a perfect match with the managerial roles. Some of what Garrison defines as a teaching presence overlap more with the pedagogical role, but instructional management is a part of the teaching presence as defined, and the parallel is apparent. “When education based on computer conferencing fails, it is usually because there has not been responsible teaching presence and appropriate leadership and direction exercised” (Garrison et. al., 2000). The roles of conference manager and organizer and planner expressed by Liu are useful in establishing leadership and direction, but Garrison’s reference to building understanding may fall more into the cognitive presence.

In our program, I have seen Dr. A. establish and maintain a strong teaching presence. Even during my first couple of courses, when Dr. A. experienced a situation which took her outside the country into areas of limited internet connectivity, although she struggled to stay on top of all her teaching responsibilities, she was quick to respond to student messages, offered direction, and provided a well-organized environment and well-facilitated discussions. In contrast, I have taken online courses where the teaching presence was weak and where I felt isolated, like I was learning on my own. I do not feel these courses were as successful in stimulating my critical thinking skills, and I felt slighted, like I wasted my time in the class and could have learned as much without the tuition charge.

I work with teachers in other departments whose online courses amount to little more than “a correspondence course via email” (Roberts & Br anna n as cited in Perry and Edwards, 2005). These courses fail to establish any social presence, and the cognitive and teacher presences are limited. Students work in isolation on problems. Teaching is often limited to reading the text and completing assignments and assessments. Those teachers with a stronger understanding of the dynamics of online learning may offer multimedia support, additional offline resources, Powerpoint slide presentations of instructor notes, etc., but fail to establish any kind of community and do not facilitate discussion among students. It may be that the courses being offered do not lend themselves well to a rich online environment. I would be interested to sit in the face-to-face courses of these instructors to find out if their traditional classrooms are focused on lecture and also lack discussion and student interaction.

The only area discussed by Liu for which there was not a parallel described in the Garrison model is the technical role. This role also lacks a parallel in traditional education. Educators have been asked to fill a role for which they are under-qualified and unpaid. This is an unfortunate side-effect of online learning. Even under the best of circumstances with well-designed courses on robust learning platforms, students and teachers can experience technical difficulties beyond their control. Operating system and browser incompatibility, problems with plug-ins, viruses, and scheduled or unscheduled server shut-down can interrupt the flow of an online course, prevent the instructor from using powerful tools, and wreak havoc with student access. When students lack the skills necessary to troubleshoot their problems, the teacher is forced into the role of technical support and often has to try and resolve student issues, redirect their inquiries, or find a work-around solution. In a small college like A&M Texarkana, there may be a lack of funds or resources to provide actual technical support staff. In future definitions of exemplary online educators, there may be a greater emphasis placed on the instructor’s ability to support the students with technical problems, although I do not feel this should ever become a routine part of their job description.

References

Garrison, D. Randy, Anderson, Terry, and Archer, Walter (2000). Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education 2(2-3). Elsevier Science Inc.

Liu, Xiaojing, Bonk, Curt, Magjuka, Richard, Lee, Seung-hee, and Su, Bude (2005). Exploring Four Dimensions of Online Instructor Roles: A Program Level Case Study. American Education and Communication Technology (AECT) International Conference.

Perry, Beth and Edwards, Margaret (April 2005). Exemplary Online Educators: Creating a Community of Inquiry. Turkish Online Journal of Distance Education-TOJDE Volume: 6, Number: 2.