Web-CAT Reading List

This reading list contains citations and links to a number of related publications by other educators on teaching software testing and on automated grading systems. If you have any additional references, please add them to this page.

Web-CAT

Web-CAT readings are listed under OurPublications.

Teaching Software Testing

• Michael H. Goldwasser. A gimmick to integrate software testing throughout the curriculum ACM. In Proceedings of the 33^rd SIGCSE Technical Symposium on Computer Science Education, ACM, 2002, pp. 271-275.

  • Describes an approach to grading testing effort by running all student test suites against all student programs, sort of like a "programming contest."

• Edward L. Jones. SPRAE: A framework for teaching software testing in the undergraduate curriculum. In Proceedings of ADMI 2000, Hampton, VA, 1-4 June 2000.

  • Describes the SPRAE framework.

• Edward L. Jones. Software testing in the computer science curriculum--a holistic approach ACM. In Proceedings of the Australasian Conference on Computing Education, ACM, 2000, pp. 153-157.

  • Proposes the integration of software testing across the curriculum, in conjunction with an automated grading approach.

• Edward L. Jones. A perspective on teaching software testing ACM. Journal of Computing in Small Colleges, 16(3): 92-100, March 2001.

  • Describes how educators should approach teaching software testing.

• Edward L. Jones. Integrating testing into the curriculum--arsenic in small doses ACM. In Proceedings of the 32^nd SIGCSE Technical Symposium on Computer Science Education, ACM, 2001, pp. 337-341.

  • Proposes integrating testing activities across the curriculum, and discusses TestLab experiences.

• Edward L. Jones. An experiential approach to incorporating software testing into the computer science curriculum IEEE. In Proceedings of the 31^st Annual Frontiers in Education Conference, vol. 2, pp. F3D-7-F3D-11, 2001.

  • Describes TestLab experiences at Florida A&M.

• John Rosenberg and Michael Kölling. Testing object-oriented programs: making it simple ACM. In Proceedings of the 28^th SIGCSE Technical Symposium on Computer Science Education, ACM, 1997, pp. 77-81.

  • Describes the difficulties of testing for students using traditional languages and approaches. Advocates the simplicity of "interactive" object manipulation in Blue (and later, BlueJ).

• Terry Shepard, Margaret Lamb, and Diane Kelly. More testing should be taught ACM. Communications of the ACM, 44(6): 103–108, June 2001.

Automatic Grading Systems

• Michael Blumenstein, Steven Green, Ann Nguyen, and Vallipuram Muthukkumarasamy. An experimental analysis of GAME: a generic automated marking environment ACM. In Proceedings of the 9^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM, 2004, pp. 67-71.

  • "Describes the Generic Automated Marking Environment (GAME) and provides a detailed analysis of its performance in assessing student programming projects and exercises. GAME has been designed to automatically assess programming assignments written in a variety of languages based on the 'structure' of the source code and the correctness of the program's output."

• James D. Bowen and Charles E. Price. An automated grading system for teaching MATLAB to freshman engineers. In Proceedings of the 2003 American Society for Engineering Education Annual Conference and Exposition, ASEE, 2003.

• Stina Bridgeman, Michael T. Goodrich, Stephen G. Kobourov, and Roberto Tamassia. PILOT: an interactive tool for learning and grading ACM. In Proceedings of the 31^st SIGCSE Technical Symposium on Computer Science Education (Austin, Texas, United States, March 7-12, 2000). S. Haller, Ed. SIGCSE '00. ACM Press, New York, NY, 2000, pp. 139-143.

• Marcel Bruch, Christoph Bockisch, Thorsten Schäfer, Mira Mezini. eAssignment--A Case for EMF. In Proceedings of the Eclipse Technology Exchange (eTX) Workshop at OOPSLA 2005, October 2005, to appear.

  • Discusses an Eclipse-based system for processing electronic programming exercises. The focus of eAssignment is on automating or bullet-proofing some of the tasks involved with setting up, distributing, collecting, and processing submissions.

• William L. Bynum, Robert E. Noonan, and Richard H. Prosl. Using a project submission tool across the curriculum ACM. In Proceedings of the 5^th Annual CCSC Northeastern Conference on Computing in Small Colleges (Ramapo College of New Jersey, Mahwah, New Jersey, United States). J. G. Meinke, Ed. Consortium for Computing Sciences in Colleges. pp. 96-104.

• L. E. Deimel, Jr. and B. A. Clarkson. The TODISK-WATLOAD system: a convenient tool for evaluating student programs ACM. In Proceedings of the 16^th Annual Southeast Regional Conference (Atlanta, Georgia, April 13-15, 1978). ACM-SE 16. ACM Press, New York, NY, 1978, pp. 168-171.

• Bob Fithen. Building A web based system for automated grading of computer modeling assignments. In Proceedings of the 2001 American Society for Engineering Education Annual Conference and Exposition, ASEE, 2001.

• J. Archer Harris, Elizabeth S. Adams, and Nancy L. Harris. Making program grading easier: but not totally automatic ACM. Journal of Computing Sciences in Colleges, 20(1): 248-261, October 2004.

• David Heaney and Charlie Daly. Mass production of individual feedback ACM. In Proceedings of the 9^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM, 2004, pp. 117-121.

  • Describes a web-based system that provides "a graphical tool that the tutors can use to produce student-friendly feedback."

• Misook Heo. A learning and assessment tool for web-based distributed education ACM. In Proceedings of the 4^th Conference on Information Technology Curriculum (Lafayette, Indiana, USA, October 16-18, 2003). CITC4 '03. ACM Press, New York, NY, 2003, pp. 151-154.

• Curt Hill, Brian M. Slator, and Lisa M. Daniels. The grader in ProgrammingLand ACM. In Proceedings of the 36^th SIGCSE Technical Symposium on Computer Science Education, ACM, 2005, pp. 211-215.

• Thomas G. Hill. MEAGER: Microsoft Excel automated grader. Journal of Computing Sciences in Colleges, 18(6): 151-164, June 2003.

• Thomas G. Hill. Excel grader and access grader. SIGCSE Bulletin, 36(2): 101-105, June 2004.

• Peter C. Isaacson, and Terry A. Scott. Automating the execution of student programs ACM. SIGCSE Bulletin, 21(2): 15-22, June 1989.

• Julia Isong. Developing an automated program checker ACM. Journal of Computing in Small Colleges, 16(3): 218-224, March 2001.

  • Describes a conventional automated grader.

• David Jackson. Using software tools to automate the assessment of student programs. Computers and Education, 17(2): 133-143, 1991.

• David Jackson. A semi-automated approach to online assessment ACM. In Proceedings of the 5^th Annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education (Helsinki, Finland, July 11-13, 2000). ITiCSE '00. ACM Press, New York, NY, 2000, pp. 164-167.

• David Jackson, and Michelle Usher. Grading student programs using ASSYST ACM. In Proceedings of the 28^th SIGCSE Technical Symposium on Computer Science Education, ACM, 1997, pp. 335-339.

• Edward L. Jones. Grading student programs--a software testing approach ACM. Journal of Computing in Small Colleges, 16(2): 187-194, January 2001.

  • Discussed the SPRAE framework, where testing concepts are used to specify assignments that are easy to automatically grade.

• Mike Joy and Michael Luck. Effective electronic marking for on-line assessment ACM. In Proceedings of the 6^th Annual Conference on the Teaching of Computing and the 3^rd Annual Conference on Integrating Technology into Computer Science Education: Changing the Delivery of Computer Science Education (Dublin City Univ., Ireland, August 18 - 21, 1998). ITiCSE '98. ACM Press, New York, NY, 1998, pp. 134-138.

  • Describes the BOSS system and its support for electronic marking support.

• Robert N. Lass, Christopher D. Cera, Nathaniel T. Bomberger, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, and Paul Zoski. Tools and techniques for large scale grading using Web-based commercial off-the-shelf software ACM. In Proceedings of the 8^th Annual Conference on Innovation and Technology in Computer Science Education (Thessaloniki, Greece, June 30-July 2, 2003). D. Finkel, Ed. ITiCSE '03. ACM Press, New York, NY, 2003, pp. 168-172.

• Michael Luck and Mike Joy. A secure on-line submission system. Software—Practice and Experience, 29(8): 721-740, 1999.

• Lauri Malmi, Ari Korhonen, and Riku Saikkonen. Experiences in automatic assessment on mass courses and issues for designing virtual courses ACM. In Proceedings of the 7^th Annual Conference on Innovation and Technology in Computer Science Education, ACM, 2002, pp. 55-59.

  • Presents experiences on using automatic assessment in large scale courses of introductory programming, data structures, and algorithms over a period of 10 years. Discusses some of the issues besides automatic feedback that are necessary for achieving good outcomes.

• Pardo, A. A multi-agent platform for automatic assignment management ACM. In Proceedings of the 7^th Annual Conference on Innovation and Technology in Computer Science Education (Aarhus, Denmark, June 24 - 28, 2002). ITiCSE '02. ACM Press, New York, NY, 2002, pp. 60-64.

• Kenneth A. Reek. A software infrastructure to support introductory computer science courses ACM. In Proceedings of the 27^th SIGCSE Technical Symposium on Computer Science Education, ACM, 1996, pp. 125-129.

  • Describes an automated submission system that also supports auto-grading. Basically a bunch of Unix shell scripts.

• Kenneth A. Reek. The TRY system—-or-—how to avoid testing student programs ACM. In Proceedings of the 2o^th SIGCSE Technical Symposium on Computer Science Education, ACM, 1989, pp. 112-116.

• Graham H.B. Roberts and Janet L.M. Verbyla. An online programming assessment tool. In Proceedings of the Fifth Australasian Conference on Computing Education (Adelaide, Australia). T. Greening and R. Lister, Eds. Conferences in Research and Practice in Information Technology Series. Australian Computer Society, Darlinghurst, Australia, 2003, pp. 69-75.

• Riku Saikkonen, Lauri Malmi, and Ari Korhonen. Fully automatic assessment of programming exercises ACM. In Proceedings of the 6^th Annual Conference on Innovation and Technology in Computer Science Education, ACM, 2001, pp. 133-136.

  • Describes an automatic grading system for Scheme programs.

• Tom Schorsch. CAP: an automated self-assessment tool to check Pascal programs for syntax, logic and style errors ACM. In Proceedings of the 26^th SIGCSE Technical Symposium on Computer Science Education (Nashville, Tennessee, United States, March 2-4, 1995). C. M. White, J. E. Miller, and J. Gersting, Eds. SIGCSE '95. ACM Press, New York, NY, 1995, pp. 168-172.

• Dennis R. Stevenson. Batch grading of Excel homework assignments. In Proceedings of the 2002 American Society for Engineering Education Annual Conference and Exposition, ASEE, 2002.

• Kumiko Tanaka-Ishii, Kazuhiko Kakehi, and Masato Takeichi. EMMA: a web-based report system for programming course--automated verification and enhanced feedback ACM. In Proceedings of the 9^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM, 2004, p. 278.

  • A one-page description of a web-based system for handing submission, grading, and feedback of programming assignments. The system is from the University of Tokyo.

• Terence Weigel. Automated grading of design problems. In Proceedings of the 2005 American Society for Engineering Education Annual Conference and Exposition, ASEE, 2005.

  • Describes a system used in a civil engineering course used to grade designs of steel building parts.

Grading and Marking Techniques

• Vicki L. Almstrum, Nell Dale, Anders Berglund, Mary Granger, Joyce Currie Little, Diane M. Miller, Marian Petre, Paul. Schragger, and Fred Springsteel. Evaluation: turning technology from toy to tool: report of the working group on evaluation ACM. In Proceedings of the 1^st Conference on Integrating Technology into Computer Science Education (Barcelona, Spain, June 2-6, 1996). ITiCSE '96. ACM Press, New York, NY, 1996, pp. 201-217.

• Janet Carter, Kirsti Ala-Mutka, Ursula Fuller, Martin Dick, John English, William Fone, and Judy Sheard. How shall we assess this? ACM. In Working Group Reports From ITiCSE on innovation and Technology in Computer Science Education (Thessaloniki, Greece, June 30-July 2, 2003). D. Finkel, Ed. ITiCSE-WGR '03. ACM Press, New York, NY, 2003, pp. 107-123.

• Charlie Daly and John Waldron. Assessing the assessment of programming ability ACM. In Proceedings of the 35^th SIGCSE Technical Symposium on Computer Science Education (Norfolk, Virginia, USA, March 3-7, 2004). SIGCSE '04. ACM Press, New York, NY, 2004, pp. 210-213.

• John English and Phil Siviter. Experience with an automatically assessed course ACM. In Proceedings of the 5^th Annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education (Helsinki, Finland, July 11-13, 2000). ITiCSE '00. ACM Press, New York, NY, 2000, pp. 168-171.

• John English. Experience with a computer-assisted formal programming examination ACM. In Proceedings of the 7^th Annual Conference on Innovation and Technology in Computer Science Education (Aarhus, Denmark, June 24-28, 2002). ITiCSE '02. ACM Press, New York, NY, 2002, pp. 51-54.

• Debra Trusso Haley, Anne De Roeck, Bashar Nuseilbeh, Marian Petre, and Pete Thomas. Gathering requirements for a grid-based automatic marking system. Presented at the 1^st International ELeGI Conference on Advanced Technology for Enhanced Learning, Vico Equense (Naples), Italy. March 15-16, 2005.

• Colin Higgins, Pavlos Symeonidis, and Athanasios Tsintsifas. The marking system for CourseMaster ACM. In Proceedings of the 7^th Annual Conference on Innovation and Technology in Computer Science Education (Aarhus, Denmark, June 24-28, 2002). ITiCSE '02. ACM Press, New York, NY, 2002, pp. 46-50.

• James W. Howatt. On criteria for grading student programs ACM. SIGCSE Bulletin, 26(3): 3-7, September 1994.

• David V. Mason and Denise M. Woit. Providing mark-up and feedback to students with online marking ACM. In the Proceedings of the 30^th SIGCSE Technical Symposium on Computer Science Education (New Orleans, Louisiana, United States, March 24-28, 1999). SIGCSE '99. ACM Press, New York, NY, 1999, pp. 3-6.

• Grigori Melnik, and Frank Maurer. The practice of specifying requirements using executable acceptance tests in computer science courses ACM. In Companion to the 20^th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, ACM, 2005, pp. 365-370.

• Brenda C. Parker, Thomas J. Cheatham, and Justin Milling. Using technology to teach technology ACM. Journal of Computing Sciences in Colleges, 17(4): 133-141, March 2002.

• Blaine Price and Marian Petre. Teaching programming through paperless assignments: an empirical evaluation of instructor feedback ACM. In Proceedings of the 2^nd Conference on Integrating Technology into Computer Science Education (Uppsala, Sweden, June 01 - 05, 1997). J. E. Miller and G. Davies, Eds. ITiCSE '97. ACM Press, New York, NY, 1997, pp. 94-99.

  • This paper considers how facilities afforded by electronic assignment handling can contribute to the quality of Internet-based teaching of programming. It reports a study comparing the nature, form, and quality of feedback provided by instructors on 90 paper and electronic assignments in an introductory Computing course and notes effective strategies for electronic marking.

• Fritz Ruehr and Genevieve Orr. 2002. Interactive program demonstration as a form of student program assessment ACM. Journal of Computing Sciences in Colleges, 18(2): 65-78, December 2002.

• Pete Thomas, Kevin Waugh, and Neil Smith. Experiments in the automatic marking of ER-diagrams ACM. In Proceedings of the 10^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (Caparica, Portugal, June 27-29, 2005). ITiCSE '05. ACM Press, New York, NY, 2005, pp. 158-162.

• Pete Thomas. The evaluation of electronic marking of examinations ACM. In Proceedings of the 8^th Annual Conference on Innovation and Technology in Computer Science Education (Thessaloniki, Greece, June 30-July 02, 2003). D. Finkel, Ed. ITiCSE '03. ACM Press, New York, NY. 2003, pp. 50-54.

• Denise Woit and Dave Mason. Enhancing student learning through on-line quizzes ACM. In Proceedings of the 31^st SIGCSE Technical Symposium on Computer Science Education (Austin, Texas, United States, March 07 - 12, 2000). S. Haller, Ed. SIGCSE '00. ACM Press, New York, NY, 2000, pp. 367-371.

Peer Evaluation

• John Hamer, Kenneth T. K. Ma, and Hugh H. F. Kwong. A method of automatic grade calibration in peer assessment. In Proceedings of the 7^th Australian Conference on Computing Education - Volume 42 (Newcastle, New South Wales, Australia). A. Young and D. Tolhurst, Eds. Conferences in Research and Practice in Information Technology Series, vol. 106. Australian Computer Society, Darlinghurst, Australia, 2005, pp. 67-72.

• Jirarat Sitthiworachart and Mike Joy. Effective peer assessment for learning computer programming ACM. In Proceedings of the 9^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM, 2004, pp. 122-126.

  • Describes experiences with web-based peer assessment in an undergraduate CS course.

• Abhijeet Trivedi, Dulal C. Kar, and Holly Patterson-McNeill. Automatic assignment management and peer evaluation ACM. Journal of Computing Sciences in Colleges, 18(4): 30-37, April 2003.

  • Discusses web-based system for anonymous electronic peer review of assignments.

• Deborah A. Trytten. A design for team peer code review ACM. In Proceedings of the 36^th SIGCSE Technical Symposium on Computer Science Education (St. Louis, Missouri, USA, February 23-27, 2005). SIGCSE '05. ACM Press, New York, NY, 2005, pp. 455-459.

• Andreas Zeller. Making students read and review code ACM. In Proceedings of the 5^th Annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education (Helsinki, Finland, July 11 - 13, 2000). ITiCSE '00. ACM Press, New York, NY, 2000, pp. 89-92.

Static Analysis of Student Programs

• Susan A. Mengel and Vinay Yerramilli. A case study of the static analysis of the quality of novice student programs ACM. In Proceedings of the 30^th SIGCSE Technical Symposium on Computer Science Education, ACM, 1999, pp. 78-82.

  • Describes an experiment using Verilog Logiscope to perform static analysis of student programs. The aim was to see if this approach could lead to automated quality assessment for grading purposes (but it didn't look promising).

• Nghi Truong, Paul Roe, and Peter Bancroft. Static analysis of students' Java programs. In Proceedings of the Sixth Conference on Australian Computing Education, Australian Computer Society, Inc., 2004, pp. 317-325.

  • From the abstract: "This paper introduces a static analysis framework which can be used to give beginning students practice in writing better quality Java programs and to assist teaching staff in the marking process. The framework uses both software engineering metrics and relative comparison to judge the quality of student's programs and provide feedback about how solutions might be improved.".

Collecting Data on Student Development

• Dorota M. Huizinga. Identifying topics for instructional improvement through on-line tracking of programming assignments ACM. In Proceedings of the 6^th Annual Conference on Innovation and Technology in Computer Science Education (Canterbury, United Kingdom). ITiCSE '01. ACM Press, New York, NY, 2001, pp. 129-132.

• Matthew C. Jadud. A first look at novice compilation behavior using BlueJ. In 16^th Annual Workshop of the Psychology of Programming Interest Group (PPIG 2004), Institute of Technology, Carlow, Ireland, April 2004.

• Jaime Spacco, David Hovemeyer, and William Pugh. An Eclipse-based course project snapshot and submission system ACM. In Proceedings of the 2004 OOPSLA Workshop on Eclipse Technology Exchange (Vancouver, British Columbia, Canada, October 24 - 24, 2004). eTX '04. ACM Press, New York, NY, 2004, pp. 52-56.

• Jaime Spacco, Jaymie Strecker, David Hovemeyer, and William Pugh. Software repository mining with Marmoset: an automated programming project snapshot and testing system ACM. In Proceedings of the 2005 International Workshop on Mining Software Repositories (St. Louis, Missouri, May 17-17, 2005). MSR '05. ACM Press, New York, NY, 2005, pp. 1-5.

  • Discussed Marmoset, an electronic submission and processing system developed at the University of Maryland. It is based on CVS, and uses Eclipse's CVS plug-in to automatically commit each file save a student makes in order to collect very fine-grained snapshots of student work undergoing development.

Plagiarism Detection

• Alex Aiken. Moss, a system for detecting software plagiarism.

  • This is the home site for Moss on the web.

• Charlie Daly and Jane Horgan. Patterns of plagiarism. In Proceedings of the 36^th SIGCSE Technical Symposium on Computer Science Education (St. Louis, Missouri, USA, February 23-27, 2005). SIGCSE '05. ACM Press, New York, NY, 2005, pp. 383-387.

• Martin Dick, Judy Sheard, Cathy Bareiss, Janet Carter, Donald Joyce, Trevor Harding, and Cary Laxer. Addressing student cheating: Definitions and solutions ACM. In Working Group Reports From ITiCSE on Innovation and Technology in Computer Science Education (Aarhus, Denmark, June 24-28, 2002). ITiCSE-WGR '02. ACM Press, New York, NY, 2002, pp. 172-184.

• Moumita Ghosh and Ranadhir Ghosh. Automatic assessment marking and plagiarism detection using SOM, fuzzy logic, and decision trees. In Proceedings of the 9^th World Multiconference on Systemics, Cybernetics and Informatics, July 10-13, 2005, Orlando, Florida, USA, 2005.

• Ghosh, M., Verma, B. and Nguyen, A.An automatic assessment marking and plagiarism detection. In Proceedings of the International Conference on Information Technology and Applications, ICITA’02, pp. 274-5.

• Guido Malpohl. JPlag: Detecting software plagiarism.

  • This is the home site for JPlag on the web. Apparently, they have just added a web service as well.

• Lutz Prechelt, Guido Malpohl, and Michael Philippsen. Finding plagiarisms among a set of programs with JPlag. Journal of Universal Computer Science, 8(11):1016-1038, November, 2002.

Specification and Assertions

• David Ginat. Embedding instructive assertions in program design ACM. In Proceedings of the 9^th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM, 2004, pp. 62-66.

  • Discusses the benefits of embedding natural, informal but precise assertions in the design process.