This page provides answers to frequently asked questions about Web-CAT. If you have a question that does not appear hear, add it to the end of the list and check back for an answer later.

Our FAQ is just getting started, so feel free to contribute questions (or answers!).

1. What Is Web-CAT?

For a quick tour, watch the SubmissionWalkthrough movie (24Mb, 5min).

The name Web-CAT stands for Web-based Center for Automated Testing. It was originally conceived as a portal for student-focused services to help teach software testing, but the first (and most successful) plug-in was one for automated grading (the Grader Subsystem). Web-CAT has since become well-known because of its automated grading capabilities, which seem to have a greater impact on student learning than any of the other services platform for providing a number of student support services to help students learn software testing. Although the Grader Subsystem supports traditional models of automated program grading--by comparing the output produced by student code against some reference--it also supports grading of assignments where students do their own testing and helps encourage test-driven development.

1.1. What is Test-driven Development?

Test-driven development (also called test-first coding) is a development technique where one writes unit tests and new code features in alternation. For each small feature or addition to be made, the programmer first writes a corresponding set of unit tests that express the expectations and intent for the feature's behavior, and then writes new code. The tests are used to check that the code meets the expressed expectations. Ideally, one constantly bounces back and forth between writing a test or two, and writing a few more lines of code. See the TestDrivenDevelopment page for more information and references. The Web-CAT Grader allows a student to submit his or her test cases along with the solution, and grades on test validity and test completeness as well as code correctness.

1.2. Does Web-CAT Support Assignments That Don't Require TDD?

In short, yes.

The actions used to compile, execute, and evaluate student code can be tailored by the instructor, who can upload one or more "grading plug-ins" to be used. Further, the feedback generated for students is completely controlled by these plug-ins. As a result, virtually any assessment and evaluation scheme desired can be used with Web-CAT, including traditional "output comparison"-style grading.

Since the author of Web-CAT advocates the use of TDD, most existing grading plug-ins follow that approach, but any other approach can be used as well. In fact, a fairly sophisticated "output comparison" back-end written has been written in Java for use with Web-CAT (but is currently still in development status).

In addition, the JavaTddPlugin allows instructors to set up assignments where student-written tests are purely optional, and scoring is based only on reference tests provided by the instructor (although the default configuration is to require student-written tests). When an assignment is configured this way, students will get extra feedback if they write their own tests (including full coverage analysis, for example), but whether or not they include tests, and how well they cover their own code will not affect scoring.

There also is a grading plug-in for C++ that provides for more traditional stdin/stdout execution of student programs. Students can write their own input file(s), and Web-CAT will show them the results. Instructors can provide "reference" input file(s) as well, with results that are only visible to course staff. The CppStdioPlugin does not include student testing performance in its scoring (instead, the CppTddPlugin provides for grading C++ programmers on how well they test their own code).

At the same time, however, remember that the learning advantages that have been experimentally documented for students using Web-CAT are based on assignments where students did write their own tests using TDD. If you don't require students to write their own tests, or grade them on how well they do this, your students may not get as much value out of their assignments. See OurPublications for more details.

As more instructors use Web-CAT, we anticipate that a wider variety of grading plug-ins will become available.

1.3. Why Should Students Use TDD?

When students write tests for their own code, they initially see it as just extra work. However, after using it for a while, students quickly begin to see benefits they can appreciate themselves. Students are less likely to turn work in late, have greater confidence that their own code works the way they intended, have an easier time finding errors as they are introduced, and avoid "big bang" integration. See OurPublications for articles describing empirical evidence for these claims (particularly our article in JERIC).

1.4. What Programming Languages Does Web-CAT Support?

Web-CAT provides flexible, extensible programming language support for any programming language that can be installed on the server. The actions used to compile, execute, and evaluate student code can be tailored by the instructor, who can upload one or more "grading plug-ins" to be used. As a result, to use a new language, all the instructor needs to do is ensure the language is installed on the server, and then provide the necessary grading and feedback plug-in(sor use existing ones where possible. Plug-ins have already been written for Java (1.4 and 1.5), Scheme (both MIT Scheme and DrScheme), Prolog (SWI), Pascal (FreePascal), Standard ML (SML/NJ), and c++ (g++ 3.4.x), and new plug-ins can be adapted from existing ones.

1.5. Do I Need to Use Eclipse?

No. Anyone can submit a zip/jar/tar/tgz file directly using a web browser, as shown in the SubmissionWalkthrough movie (24Mb, 5min).

For students who do wish to use Eclipse, we have a submission plug-in they can download and install, but it is purely for convenience.

The plug-in will add a "Submit..." link to the Project menu. When clicked, this action will query the Web-CAT server to find out what assignments are open and let the student pick one from the list. It will then jar up their eclipse project, submit it to Web-CAT, and pop up the results within Eclipse (or in your web browser, depending on your OS).

You can see a screen shot of the menu addition as well as a screen shot of the submission wizard.

The plug-in installs the normal way (unzip it in your eclipse/plugins directory). Configuration is via Window->Preferences->Electronic Submission, and all you have to do is set the URL to the Web-CAT server.

The submission plug-in can be downloaded from our sourceforge site. Read the recipe on how to ConfigureEclipseSubmitter.

1.6. Can I Use BlueJ's Submitter?

Yes, it is also possible to use BlueJ's submitter extension to submit directly to Web-CAT. That extension ships with stock BlueJ distributions. All you need is a one-line config file to point it to the Web-CAT server in order to use it. Read the recipe on how to ConfigureBlueJsSubmitter.

1.7. Can I Supply My Own Code or Data Files for Student Assignments?

Yes. Instructors who are setting up assignments have their own personal "file space" where they can upload resources that they wish to use for one or many assignments (data files, supplementary jar files, configuration files for static analysis tools, etc.). You can manage this file space through Web-CAT's web interface, including uploading files, creating subdirectories, downloading/deleting items or entire folders, and even direct editing of text files on-line.

The Java grading plug-in allows instructors to provide a supplementary jar or an (arbitrarily complex) subdirectory of class files that you would like to have added to the classpath for grading assignments. You can set up a common set of options that you'd like to reuse across several assignments where you describe these kinds of parameters just once (and can have as many of these as you'd like). For any individual assignment, you can also specify an additional supplementary jar or classpath dir that applies to that assignment only.

The C++ grading plug-in provides a similar mechanism for instructors to supply a directory of header files to place on the include path, compiled library files to include at link time, or extra data files to be placed in the current working directory while student code is evaluated.

On the VT server, we preserve the contents of this "personal file space" from semester to semester, so that once you take the time to upload your favorite resources, they are always around for you to use on your assignments (until you decide to change). This is also the same place where instructors can upload reference tests to use in automatic grading.

1.8. How Do Instructors Specify Their Tests?

For the Java grading plug-in, standard JUnit test classes are used (as *.java source files to be compiled against student code). You can have as many as you want for a given assignment. There are no limitations except for some basic naming conventions. Students use the same format for the test cases they write. JUnit is widespread enough that there are many useful resources for teaching and learning how it works.

In the future, we are imagining optionally expanding this to provide our own custom subclass of junit.framework.TestCase that transparently adds some useful features that current users have requested. Instructors who want to take advantage of these features will only have to change the name of the baseclass they use, but this will be optional.

The C++ grading plug-in uses CxxTest-style tests. Other plug-ins use different formats, depending on the programming language used for submissions.

1.9. As an Instructor, Can I Run My Own ANT Script on Student Code?

The current Java grading plug-in actually uses a carefully designed ANT script (see http:/ant.apache.org/) to control the build and analysis process for student code. It is pretty general, in that it can handle virtually any project structure that folks are likely to use. However, it does not support the use of any code generation tools to generate Java files from other non-Java sources.

However, instructors can actually upload their own plug-ins and use them for assignment grading all by themselves, using just the web interface. If you want to use a custom ANT build script, the best thing to do might be to take the current Java plug-in, add to or enhance its build.xml file, and upload it as a custom plug-in for your own use. You can edit/modify/re-upload a new build.xml file for your custom plug-in at any time if you need to change its behavior.

The C++ grading plug-in also uses ANT to control the build and analysis process for student code. We use the various ANT tasks from the ant-contrib project at http://ant-contrib.sourceforge.net/ to compile and link C++, regardless of the directory hierarchy or project structure used by students.

2. Using Web-CAT

2.1. What's It Like to Submit an Assignment?

For a quick tour, watch the SubmissionWalkthrough movie (24Mb, 5min).

Students can upload their submissions via a web browser, and no other software is required to use Web-CAT. Multi-file submissions are accepted in the form of zip or jar archives, and are automatically expanded as needed. All submissions and their results are archived independently. Further, automated submission support is also provided for users of BlueJ and Eclipse. Extending submission support to other environments is also possible.

2.2. What's It Like to Set Up an Assignment?

For a quick tour from the instructor's side, watch the CreateAnAssignmentWalkthrough (34Mb, 9min).

2.3. What Options Can Instructors Control?

Because assignment processing and feedback generation are completely plug-in-based, the exact options available to an instructor depends on the plug-ins that are being used for an assignment. However, for an example, you can browse through the JavaTddPlugin/ParameterDescriptions, which lay out the controls available for our most-widely used plug-in (for Java).

2.4. How Do Students Receive Feedback?

Students receive their feedback via their web browser. The form of the feedback is completely customizable by the grading plug-ins used by the instructor, but typically takes the form of an integrated web-viewable feedback report that includes a scoring summary and a color-coded, syntax-highlighted view of each source file. For languages where static or dynamic analysis tools are provided, this report may also include an execution count for each line, with under-tested lines highlighted, as well as individual line-by-line errors or diagnostics. Instructors, TAs, or grading staff can directly add comments and point deductions to the color-highlighted source code view using only a browser. For a visual example of typical feedback produced by the JavaTddPlugin, watch the SubmissionWalkthrough movie (24Mb, 5min).

2.5. What Happens When Hundreds of Students Submit Simultaneously?

We typically run 500-600 users per semester on our main server at Virginia Tech. Of course, these are spread over multiple courses, and usually no more than 100-150 have simultaneous deadlines. Load runs fine most of the time.

The only load problems students see are when an assignment deadline is approaching and many students are submitting, especially if multiple courses have accidentally assigned work due at the same time.

Processing a single Java assignment takes between 15-45 seconds, on average. Of course, this depends on many factors, including the likelihood that students will write code with infinite loops or other non-terminating behaviors. The server only grades one submission at a time for a number of technical reasons. Handling extremely large number of student sessions and submissions is not a problem, but submissions queue up and are processed on a first-come-first-served basis.

So the problem is that if 100 students all submit at the same time, it will take about 1500-4500 seconds (or 25-75 minutes) for them to all receive feedback. Students can continue to access the server, look up results (say, on earlier submissions), and make new submissions, but none of that will speed up when results come back on work they've already submitted.

Web-CAT uses several techniques to ensure single individuals don't monopolize the queue, and also to make sure that administrative requests (say, to regrade assignments) don't affect new submissions from students.

The bottom line is that students learn very quickly (and we tell them early) about the load as an assignment deadline approaches. I tell my students explicitly that if they're making a submission within 1 hour of the deadline, they should expect to wait--possibly a long time--before they see their feedback. Instead, we encourage them to start early and submit early if they want to take advantage of the feedback. In other words, the delays near deadlines are very predictable, and are the price for waiting too late. If you want to avoid them, it is simple--don't wait until the last minute.

In practice, this actually works quite well. Students are very good at moderating their own behavior once they know what is going on. The key is to make sure they have reasonable expectations about what will happen, so that they don't get surprised when they can least afford it.

2.6. Can Server Response Times Be Improved by Configuration?

A faster server that is dedicated just to Web-CAT use is best. A multi-processor server is even better. But there's not much you can do beyond that, I don't think. It is possible to move the database off to a separate server, use a web server (like apache) to serve static content instead of forcing Tomcat to do that also, and even move the web server off to a separate server. However, there will still be queue problems if hundreds of students are shooting for the same deadline.

Overall, though, this hasn't been a problem in practice. While our class sizes have gone down, we used to regularly use Web-CAT in courses of up to 300 students, and one of the longest-running non-VT servers regularly has classes that are even larger. Once students learn to expect long queues when a deadline approaches, they can deal with it.

If you're proactive about communicating expectations and explaining how to avoid the long waits, that is w-a-y more effective than server tweaking. I personally haven't seen queue sizes > 10 in any of my classes in the past year, even with classes well over 100 students in size. But I attribute that to students who have some experience with Web-CAT and who know to anticipate (and avoid) the delays.

We are experimenting with cluster computing this year so that we can put a pool of CPUs on the grading queue, but at best this gives just a linear speedup.

3. Getting a Copy of Web-CAT

3.1. Is Web-CAT Open Source?

Yes, Web-CAT is licensed under the terms of the GNU General Public License. Source code and binary releases for both Web-CAT and our EclipsePlugins are available via our SourceForgeProject.

When we originally started the project, there were a number of obstacles to creating an OpenSourceWebCatProject, but we have overcome most of them in the intervening time.

3.2. Can I Install Web-CAT at My Institution?

Yes, if you are brave. In February, 2006, we publicly released our first alpha distribution of Web-CAT. It is a servlet application that runs under a suitable container, such as Apache Tomcat, and on top of an appropriate JDBC-compatible database, such as MySQL. The servlet has a built-in self-installation wizard that runs the first time you visit the application. The self-installer walks you through the process of setting up database connectivity, creating database tables, picking your authentication mechanism, creating an administrator account, and populating the database with sample data.

If you want to try to install Web-CAT, realize that we are still in the process of writing our installation manual. You'll have to be willing to wrestle with installing some other open-source packages, and be patient with our development team, who will be happy to help you get off the ground if you're willing to invest the required effort. In return, we hope that you will contribute to the Web-CAT project by helping us write our installation manual--in the form of recipes in our WebCatCookbook--right here on this wiki. For more information, read HowToContribute.

4. Using Web-CAT in Your Courses

4.1. Can I Try It to See What It is Like?

You can TryOutWebCat from our server to see how it works.

4.2. If I Cannot Get a Copy, How Can I Use It?

After you TryOutWebCat, you can volunteer to UseWebCatRemotely from our server for your course. We currently host courses at three different universities from our main server, which has processed around 100,000 submissions.

4.3. Why Does Web-CAT Generate Warnings on Most of My Source Lines?

Some of the "tryout" assignments you can use to take Web-CAT for a test drive are configured with Checkstyle and PMD settings we use in our freshman classes, and which may differ in some minor ways from your preferred coding style. In particular, the Checkstyle options expect code to be indented every 4 columns using spaces instead of tabs (tabstops are interpreted as being every 8 columns). Checkstyle is also configured to enforce the use of braces in all control structures. Further, opening braces are expected to begin on a new line immediately beneath the linguistic structure they are a part of, instead of being "cuddled" on the right edge of the previous line.

Feel free to use the alternate versions of the "tryout" assignments that are not labelled as "tighter formatting" (and which have more liberal settings), or adjust your coding style for the "tighter" tryout assignments. If you decide to use Web-CAT in a class, you are free to configure the Checkstyle and PMD settings to values that suite your institution's preferred style conventions.

5. Contributing to Web-CAT

5.1. Can I Help Develop the Code?

Yes! See a discussion of HowToContribute. You can also contribute through our SourceforgeProject.