A: to get to the “does not compute.”

Bad news for those who want to apply completely automated, industrial processes to writing assessment! A recent UK experiment with computerized marking progrmas has given them a failing grade. The Chartered Institute of Educational Assessors recently put the computer marker through its paces by feeding it samples of essays written by the likes of Hemmingway and Churchill. The machine did not take kindly to the test and gave poor marks to some of the best writers of the 20th century. Not that this news is surprising, for computers are still fairly clumsy at natural language processing – just ask your grammar checker in MS Word. To imagine that highschool essays like those required in the British A-levels and CGSEs do not fall into the category of natural language is foolish. Foolish because in many of the rubrics designed to help teachers assess student writing at the highschool level – six plus traits, for instance – to demonstrate mastery, the student must break out of the mold of rigid formula. They must show spark by using metaphor, descriptive language, unconventional sentence structure, and all of the other things that set great writing apart from merely competent writing. How can a parser understand whether a metaphor works, or falls flat? How can a machine see the difference between stylish concision and lack of vocabulary?

COMARKER™ skates by this problem because from the beginning we understood that putting teachers and markers out of work was antithetical to the educational enterprise. Academic writing tests our ability to generate meaning from the disparate and the abstract and that is preceisely what the computer cannot yet generate or assess. So to generate feedback and authentic assessments and evaluations for students COMARKER™ relies on a more creative and intelligent system : human teachers.

We still need beta testers, so contact us if you want to help us make a system that improves on the shortcomings of full automation.

Last week Brian Panulla, one of our Flex developers, flew up to Vancouver from Portland to meet with us about something we have been talking about since the summer: creating a a document standard for rubrics and a program that allows teachers to create, edit, and share those rubrics with their colleagues.  The meetings were a resounding success, one of the several that have come out of the OpenEd 2009 conference in Vancouver last summer. In two days of meetings we hammered out the requirements, some user stories, and the basic architecture. Brian will begin iterating later this month, and if all goes well, we will have a working prototype in about two or three months.

More sophisticated prototypes are to follow

The rubric module will not just be for users of COMARKER™, but for teachers everywhere to create and share their rubrics. Of course, rubrics built with the new system will be fully interactive in COMARKER™, but since we are all about mass collaboration, everyone will be invited to share.

The most frequent and forceful complaint that we have had about our program from our testers, so far, has been that the window in which markers read and mark the student’s essays is too small. I even lined up one tester who refused to mark with the program given the tiny window. We have righted this wrong with a substantial full screen mode. I just used it and it looks great, it is easy to use, and it doesn’t mess with the format of the essays. Tatiana, our programmer, has also recently completed the Rubric function of our program, so markers can quickly and easily grade the essays that they have marked up. Right now we have the Six-Plus Writing Traits rubric in the system. We use this rubric in our writing school. And as soon as Tom gets back from India, he and I are going to get to work designing an additional module that markers can use to upload and use their own rubrics (then share them with everybody else, just like comments). I am really looking forward to our next semester at A+ Academic Writing, when we can finally use the new features that we have added since the end of the last one.

P.S. I am in the land of the Ice and Snow right now. -30 degrees Celsius with no wind to speak of. At least I get to see my family this year. Happy Holidays!