The Marmoset Project

Marmoset Demo Server The Marmoset Demo Server allows you to register an account with our demo server and then sample our software.
Marmoset Publications Links to Marmoset papers and bibtex entries.

What is Marmoset?

Marmoset is an automated snapshot, submission and testing system developed at the University of Maryland.

Goals for Marmoset

We had two main goals in developing Marmoset:
  1. To create a better experience for students learning to program.

    Learning to program is difficult! Students in introductory programming courses are asked to solve difficult problems using unfamiliar concepts in an unforgiving environment. We want to provide more feedback and better feedback.

  2. To better understand how students learn to program.

    Have you ever asked questions like "What the heck are students doing when they pull marathon all-nighters?" or "It looks like many students are making the same mistake; can I find these errors automatically so my students don't waste their time?".

    We've found that the data Marmoset collects allows us to answer these questions, and many others.

Collecting Fine-grained Snapshots

We use an Eclipse plugin called the Course Project Manager to capture snapshots of students' projects to a centralized CVS repository whenever they save their files.

The resulting fine-grained revision history provides a unique perspective into the development process for students. And because the plugin works automatically and transparently, we don't burden introductory students with version control--yet another piece of software with an unintuitive interface.

Automated Testing

Students can also explicitly submit projects to the SubmitServer, a centralized submission and testing server, that will run these submissions against a suite of unit tests developed by the instructors to evaluate the functional correctness of a submission.

Of course we don't show the students all the results from the unit tests. It's pretty clearly unproductive to show students all the test results, for fear they'll tweak their code in a seemingly random fashion until it passes enough test cases. We call this programming by "Brownian motion".

The typical compromise is to have public test cases that the students can see and secret test cases that the students can't see. The problem here is that the secret tests cases are usually not executed (and often not even written!) until after the project has been submitted.

So a student submits something that passes all the public test cases, and two weeks later finds out she got 50/100 because she failed all the secret tests. This feedback is two weeks late to be useful in fixing the program!

Release Testing

We have added a new class of test case called a release test, giving use three classes of unit tests:

Data Collected so far...


Table: Data collected by Marmoset for 4 consecutive semesters of CS-2 at the University of Maryland
semester # students # projects # snapshots # submissions # test outcomes
Fall 2004 84 8 44,771 3,919 627,320
Spring 2005 103 8 45,558 3,932 451,727
Fall 2005 69 7 14,214 1,505 152,359
Spring 2006 101 6 43,052 2,680 940,406