The Marmoset Project
-
Marmoset Open source project
-
The Marmoset source code repository. Marmoset is open
source under the Apache 2.0 license. At the moment,
it is undergoing rapid evolution and stabilization;
we are hoping to find several other universities
willing to try using it for the Spring 2012 semester.
-
Survey on collection and grading of student programming projects
-
A survey about to assess current practices in the collection and grading of student programming projects. Results of the survey will be made available from this web page.
-
Marmoset
Publications
-
Links to Marmoset papers and bibtex entries.
What is Marmoset?
Marmoset is an automated snapshot, submission, testing and code
review system
developed at the University of Maryland.
In Fall 2011, it was used in 27 courses by 1,770 students for a total of 170 programming
assignments. More than 69,000 project submissions were collected during the semester (students can submit multiple times for a particular project, getting
limited feedback each time).
Marmoset is described as part of a talk given by Professor William Pugh
on Innovation in teaching software development skills Dec 2nd, 2011 at SUNY Stony Brook.
Look for more information and updates on this web page shortly.
Some of the information below is rather dated, but provided until we can provide more up to date information.
Goals for Marmoset
We had two main goals in developing Marmoset:
- To create a better experience for students learning to program.
Learning to program is difficult! Students in introductory
programming courses are asked to solve difficult problems using
unfamiliar concepts in an unforgiving environment. We want to provide
more feedback and better feedback.
- To better understand how students learn to program.
Have you ever asked questions like "What the heck are students doing
when they pull marathon all-nighters?" or "It looks like many students
are making the same mistake; can I find these errors automatically so
my students don't waste their time?".
We've found that the data Marmoset collects allows us to answer these
questions, and many others.
Collecting Fine-grained Snapshots
We use an Eclipse plugin called the Course Project Manager to capture
snapshots of students' projects to a centralized CVS repository
whenever they save their files.
The resulting fine-grained revision history provides a unique
perspective into the development process for students. And because
the plugin works automatically and transparently, we don't burden
introductory students with version control--yet another piece of software with an
unintuitive interface.
Automated Testing
Students can also explicitly submit projects to the SubmitServer, a
centralized submission and testing server, that will run these
submissions against a suite of unit tests developed by the instructors
to evaluate the functional correctness of a submission.
Of course we don't show the students all the results from the
unit tests. It's pretty clearly unproductive to show students all the
test results, for fear they'll tweak their code in a seemingly random
fashion until it passes enough test cases. We call this programming
by "Brownian motion".
The typical compromise is to have public test cases that the
students can see and secret test cases that the students can't
see. The problem here is that the secret tests cases are usually not
executed (and often not even written!) until after the project
has been submitted.
So a student submits something that passes all the public test cases,
and two weeks later finds out she got 50/100 because she failed all
the secret tests. This feedback is two weeks late to be useful in
fixing the program!
Release Testing
We have added a new class of test case called a release test,
giving use three classes of unit tests:
- Public tests are distributed to students directly.
They can run these tests and see the results at any time.
- Secret tests are only used for grading purposes. They are
not distributed to students and the results of individual tests
are not revealed to students before the deadline. After the
deadline, secret test results can be optionally revealed to students.
- Release tests are not distributed to students. However,
students may request the results of running them on a particular
project submission. For a release test request to be granted,
students must have a release token available. Students are
given a limited number of tokens, each of which regenerates in a
fixed period of time, e.g. 3 tokens every 24 hours. Viewing
release results allows the student to see the number of
release tests passed and failed as well as the names of the
first two tests failed. For example, for a project requiring the
evaluation of various poker hands, a student may discover that her
submission has passed 6 out of 12 release tests and that the first
two tests failed were
testFullHouse and testThreeOfAKind.
Data Collected so far...
Table:
Data collected by Marmoset for 4 consecutive semesters of
CS-2 at the University of Maryland
| semester |
# students |
# projects |
# snapshots |
# submissions |
# test outcomes |
| Fall 2004 |
84 |
8 |
44,771 |
3,919 |
627,320 |
| Spring 2005 |
103 |
8 |
45,558 |
3,932 |
451,727 |
| Fall 2005 |
69 |
7 |
14,214 |
1,505 |
152,359 |
| Spring 2006 |
101 |
6 |
43,052 |
2,680 |
940,406 |
|