CIS 422 Project 1
Automated Turnin Program

A.Hornof - April 3, 2003

This project is due on April 30, at 10 PM, including all source code files and documentation. Follow the submission instructions. The initial SRS/SDS/ProjectPlan is due in class on Tuesday, April 15, along with a class presentation of the document.

Problem Statement

Students taking computer science classes need to routinely submit their completed programming assignments to the instructor for evaluation and grading. There are many advantages to submitting these projects electronically so that the instructor can compile, run, and test the programs.

Electronic submission of assignments presents a number of administrative challenges. The students are one set of stakeholders, and the problem can be considered from their perspective. Each student must have a clear and straightforward way to submit their work, and for their submission to be clearly separated from those of other students. Each student's submission must be secure; that is, other students should not be able to see it. Each student should also have a means of confirming that their submission was successful, and that they submitted the intended files.

The instructor is another stakeholder. He or she needs to be able to compile, run, and test all of the programs submitted by students in a secure location that the students cannot see. He or she needs to provide feedback to the students on how their programs performed, and to maintain a list of all the scores, both internally with the students' names, and externally so that the students can see their progress, but by using a randomly assigned name or number. Also, the "instructor" will often be a small staff of more than one person, including GTFs (graduate teaching fellows, or teaching assistants) and graders.

The systems support administration staff, who maintain the shared computer system used by the students and instructor, are another set of stakeholders. This staff's job is to maintain a working and secure shared system. They will not endorse a turnin system that creates any sort of security weakness.

There are numerous challenges in solving this problem. The system is not a "one off," or something that will be just set up and left to run indefinitely. The solution will need to be reconfigured every term for numerous classes by numerous instructors, with little or no training. The system will need to accommodate multiple classes at the same time. Instructors will need to reconfigure the system for each homework assignment, and archive data at the end of the term. A turnin system that fails, such as by losing data or creating a security breach, will be banished from a department. Instructors will have little patience with a turnin system that imposes itself on the design of a class, such as by not accommodating a project with thirty source code files, or a system that requires source code files to be named to fit a fixed scheme.

The problem grows even more difficult as you start to consider the special cases that will arise. For example, students may sometimes be required to submit a hierarchy of files that must be reconstructed for grading (such as a Java program organized into packages), and instructors will have different late policies, perhaps even from assignment to assignment, and may impose different penalties even from project to project.

Many Possible Solutions

There are many possible technical solutions to the problem, and your job is to identify, design, build, test, document, and deliver one working solution. You may use any programming language or packages, or any combination of such, and reuse any code that you find, provided that you clearly cite your sources and clearly delineate and cite the source of your reused code. But your solution should follow from your design, rather than your design conforming to some reusable code that you find.

A number of turnin programs are already available and used in the CIS department. You may look to these for ideas and inspiration, provided you cite the sources of your ideas as best as possible. Each of the current programs has its strengths and weaknesses. Your solution should provide as many of the strengths and as few of the weaknesses as is possible.

Solution #1: Sugih Jamin's Unix-daemon-based "turnin" program

Professor Sugih Jamin, at the University of Michigan, wrote two C programs that work together to facilitate the automated submission of assignment files. The system has been successfully used by the current 422 instructor as well as by other faculty in the CIS department here at the UofO, and will in fact be used to submit projects in this version of 422. The first C program is "turnind". The 'd' stands for daemon, a program that lies dormant in Unix waiting for a condition to occur. The instructor must start up the daemon before students can submit their assignments. The second program is "turnin", which students invoke from the Unix command line, along with a few arguments, to submit their code. The daemon has already been started for this project. Feel free to try it out.

Advantages

  • The program is elegant in its simplicity.
  • Students can be given relatively simple instructions such as Submission Instructions, and few if any students have serious trouble following the instructions.
  • The students can confirm their file arrived by viewing the turnin directory online.
  • The instructor only needs a relatively simple How to Use Turnin and Turnind document to remember how to set it up and configure it for each assignment for each term.

Disadvantages

  • Students must have an account on the Unix server so they can run the "turnin" program. Thus, the system cannot be used for 200-level programming courses because students in these courses do not have CS accounts.
  • If the machine running the daemon is rebooted, the daemon process is terminated and students can no longer turnin their projects until the daemon is restarted.
  • There is a fair amount of administrative file manipulation required by the instructor, including unpacking tar files, examining submission dates and times for late submissions.
  • Students cannot see what they turned in other than to confirm that the tar file arrived at a particular date and time.
  • A unique process ID must be provided for every unique instance of the daemon running concurrently. If different instructors use it simultaneously, a unique process-numbering policy must be established or trouble will occur.

Solution #2: David Atkins' web-based "e-turnin" system

Professor David Atkins, here at the University of Oregon, wrote a web-based "e-turnin" system that he and other 200-level instructors (including the current instructor of this course) have used successfully for collecting homework projects in these introductory programming courses. The program is written in MAWL. It is a web-form-based system that runs as a CGI script. It is a more complex system than the Unix-daemon-based Solution #1, but it also has more features.

Advantages

  • Web-based, so students without a CIS account can use it.
  • Students create their own accounts, limiting the setup time required by the instructor and creating a sort of secondary class roster for the instructor, complete with student IDs and email addresses (though the information is also available to via DuckWeb).
  • Students can view their solutions on line after they are submitted.
  • The instructor can add a grading or comments file that the student can view through the system (though this is an undocumented feature, and is a little tedious for the instructor to set up).
  • There are a number of other useful features built in, such as the facility to give the students a series of survey questions when they create their account.
  • The setup is relatively straightforward with a few e-turnin notes, but note that one user's experience (mine) was that unexpected problems arise during setup that require a fair amount of troubleshooting time. But once the system was set up initially, it worked pretty much just fine.

Disadvantages

  • Does not lend itself to group submissions, in part because students are asked to create individual accounts that include their student ID number.
  • Very limited technical and user documentation.
  • Written in a somewhat obscure language, limiting its extensibility.
  • The filenames must all be specified in advance by the instructor.
  • Source code must be cut and paste into fields, which is tedious and leads to user errors, especially when students are submitted more than just two or three files, and prevents students from submitting non-text files such as graphics files.
  • Not as robust and well-documented as a commercial off-the shelf (COTS) product.
  • Partly because of so many useful features that the program already has, it is easy to think of additional useful features.

The e-turnin system has been configured and set up for CIS 422, but just for experimental purposes. A user named "test" has been created, with the password of "test". Since it will not be used for submitting any homeworks, feel free to try out this account or to create a test account of your own, but do not use your real student ID number because it will be visible to other students.

The source code is available. Also, the solutions directory has been made publicly readable, so you can look inside and see what the instructor would see when he or she goes to do the grading. Normally, the solutions directory would not be public-readable. This is why it is important that you don't use your real student ID. The student ID is part of each student's directory name.

Study the Existing Solutions and Devise a Better Solution

A non-functional copy of the main administrative web page is viewable at admin_sample.html. A nonfunctional copy of the administrative student list that gets created is at student_list_sample.html. Since you did not get to these pages by logging on as the administrator, you will quickly run into errors if you try to execute any of the functions that appear to be on these pages.

You can also set up either of these systems by following the setup instructions for each, and thus see the full-blown instructor experience of using each system. Your solution to Project 1 should be at least as good as either of these solutions, and preferably better. You should also produce better user and technical documentation. The real test will be whether instructors in the department--and the instructor of this 422 class--can quickly and easily install, configure, set up, and test the reliability of your system to the point that he or she would be completely comfortable having students in a class submit their homework assignments via this system, and would prefer it over the two other systems currently available. Note that fancy colorful graphics, bold claims, advertising-type language boasting the features of a system, nicely-formatted but content-free and useless user or technical documentation will not help your product a bit, and would in fact likely detract from the quality of the product. The most important thing is that your system work flawlessly under all possible scenarios (file types, browsers, command shells, through rebooting of systems, multiple simultaneous projects and classes, etc.), provide a useful set of features, and be easy to configure and easy to use.

Sugih Jamin's Unix-daemon-based "turnin" program is very well-written, simple, and concise. It is barely documented at all, but this may be its only shortcoming as a distributable, reusable piece of software. It is a great tool. It gets the job done well, and reliably.

Possible Requirements Derived From a Field Study Using "e-turnin"

In the course of using Professor Atkins' "e-turnin" program last term for a programming class of about 50 students, a number of issues came up. Many or most of these would be excellent candidates for requirements to satisfy in your project. They will be listed here in no particular organization or order. You may prioritize them, break them into functional and non-functional issues, or use them in any way you see fit. Note that few of them are currently written as requirements, but more just as ideas.

  • The ideal system would allow for late days. The current system sets a single fixed deadline. It was not feasible to permit students to turn in projects with late penalties, in part because the system just did not facilitate it.
  • A number of perl or shell scripts were often needed immediately upon starting to grade the homeworks, to organize the submitted files into a directory structure that reflected the package organization that they were required to use, and to compile and run the programs. The ideal turnin program would also facilitate the compilation and running of programs, and ease the flow of control needed during the grading process, such as by gracefully handling situations in which a program being tested "hangs".
  • The ideal system would also facilitate e-turnback.
  • The programming team delivering the system must successfully demonstrate that no backdoor has been built into the system.
  • The ideal system would permit the turning in of files either by cutting and pasting, or by selecting files from a command line or file dialog box, or perhaps even dragging and dropping files.
  • The National Science Foundation, interestingly, now handles all of its grant applications and reviews via a computerized system, which itself includes a number of turnin subsystems. Perhaps some ideas could be gleaned from this system. The system is called FastLane. FastLane had a very rough start. It's getting better, but it's still not perfect.
  • To facilitate 200-level students, the system should probably work over the web.
  • Perhaps the system should require, after each user submits a file, that the user confirms they submitted what they intended. This is an interesting (and slightly annoying) feature of FastLane.
  • Perhaps the system should work for all browsers, with java, javascript, and cookies either on or off.
  • The system should make it easy to add and drop students in the middle of the class, but without losing the data from the students who dropped.
  • Perhaps the system should facilitate the assigning of random Student ID names or numbers.
  • The system could assist in posting grades to web page.
  • Student IDs must not be compromised. They should not be printed out with solutions that could be passed back in class.
  • There must be a convenient way to print out all the files of all the students, along with an execution trace. For example, if the instructor wants to grade them on paper. The e-turnin system is actually very good for this.
  • The system must set all read/write and ownership permissions correctly for both the instructor and GTFs for the class, including the Unix group permissions.
  • The necessary and relevant data must easy and straightforward to backup at the end of the term. Assume the instructor will forget the system password two weeks after the class ends.
  • The system must be verifiably secure.
  • Many types of files can be submitted and need to be set up for automated compilation
  • Perhaps the instructor should be able to modify programs slightly, such as if they find a bug they would like to fix, but without disturbing the original file that the students submitted. They should be able to, in effect, just go down the line grading project after project. They should be able to create notes files that the students can see eventually, and annotate submitted code.
  • It must be easy to backup the submitted files during and at the end of the term.

Distributing the Work Within Your Team

One of the biggest challenges in this project is going to be finding good ways to divide up the programming and other technical work. Aside from this, you must also assign responsibilities to team members. It is not a good strategy to simply share all responsibilities. Although all team members should contribute to some extent to every aspect of the project, it is essential to have one person with central responsibility for each major part of the task. Among the responsibilities to be assigned include:

  • Manager. This person is primarily responsible for administration of the project schedule. This person should be well-organized.
  • Quality control. This person is responsible for administering the quality plan. Thoroughness and attention to detail are important here.
  • System architect. This person is responsible for the overall architectural design of the system. Good abstract thinking skills are useful.
  • Technical documentation. This person is responsible for documents meant primarily to be read by software developers and maintainers. Written and verbal communication skills are important.
  • User documentation. This person is responsible for documents meant primarily to be read by people other than software developers, such as end users. Good communication skills are important, as well as the patience to try things over and over again from the user's perspective.
  • User interface. This person is responsible for human factors and ensuring a good "user experience." This person should also be able to see things from the user's perspective, and should thoroughly understand the users' tasks.
  • Configuration control and product build. This person is responsible for managing the construction of software products, including coordination of multiple developers. This person will need to maintain order and enforce policies, ideally setting up a shared data repository, such as with CVS.

The mapping of people to roles is not necessarily one-to-one. For example, it is common in small teams for the user documentation and user interface roles to be combined. You may wish to identify additional roles. You may also want to spell out responsibilities more clearly, e.g., shall the user documentation or technical documentation person be in charge of the final presentation?

The person with ultimate responsibility for one of these functions does not have to do the whole job alone. For example, the system architect does not design the whole system alone, and the product build manager does not do all the implementation; they are managers of different aspects of the project.

Risk control is an important part of project management. One of the largest risks to any software project is the loss of a key person. Therefore, while it is important that a single person be ultimately responsible for each role, it is a very good idea to assign a "backup" for each role also. The backup person assigns the main person responsible for a role, and should be knowledgeable enough about that role to take over responsibility if the primary person is lost or unable to fulfill his responsibility. The backup role is also an opportunity for "cross-training" in preparation to take a lead role in a later project.

The SRS / SDS / Project Plan

The SRS / SDS / Project Plan document is a small combination of elements that might appear in several different documents in a larger project: A proposal, a feasibility study, a project plan, a requirements statement, a specification and/or external design, and an architectural design overview. This document should convince management, a client, or an investor that this project is worth funding. The quality and content of the document will communicate the likelihood of success of the project if it were to be approved.

Your SRS / SDS / Project Plan should include at least the following:

  • A clear and concise one-paragraph problem statement. What is the problem to be solved? The problem, including the task requirements, should be described independently of the solution, the piece of software you will build.
  • A description of the product you intend to build. This should describe the externally visible behavior of your product as precisely as possible, but it should be concise and clear.
  • An overall design description. What are the major parts of your system, and how do they fit together? What are the main organizing principles that you used to break your system into parts?
  • A management plan. How is your team organized? How is the work divided among team members? How does your team make decisions? How do you check progress against your plan? How will your team meet and how will it communicate?
  • A build plan. What is the sequence of steps you will take to build the system? When will each "build" of the system take place.
  • A rationale for the overall design and build plan. Why have you broken the system into these parts, and why have you chosen these particular steps to build the system?

Each group will prepare an in-class presentation that covers each of these six topics. The presentation, as with the document, should be intended to convince the audience that your project is worth their support. In both the document and the presentation, solid ideas and good content will go much further than e-marketing mumbo jumbo.

Project Management

The group should keep a record of meetings. This record should briefly note the agenda of the meeting, the date and time, who showed up (and who showed up on time), and what was accomplished during the meeting. Set ending times for meetings to help you get through the agenda items.

The group should also keep a record of each of the tasks that are assigned to each group member. This should be in the form of a spreadsheet or table with the following columns: The task, assigned to whom, when assigned, when due, when completed, who did it, who signed off on it.

Reuse Guidelines

The cheapest, most dependable and least risky software components are those you don't build. You can find many nifty things freely available on the web. I strongly suggest you take advantage of them. On the other hand, you must do so in a way that is legal and ethical, and while I won't set an upper bound on how much of your project code can be reused, you must certainly provide some "value added" and not merely repackage software available elsewhere.

To be legal, you must obey all copyright restrictions in software you use. Beware that a document or file need not contain an explicit copyright statement to be protected by copyright law; you have a right to copy or reuse something only if the author has specifically granted you that right. I am absolutely firm on this, and will not hesitate to fail an individual or a whole team for unethical conduct as regards intellectual property. If you have any questions about what you may or may not do, ask me.

Your product must be freely distributable under the Gnu copyleft agreement. In some cases this may mean that you cannot make use of some software which is otherwise perfect. In other cases it may mean that your product will depend on other software packages that you cannot directly distribute. (Be careful of such dependencies, especially on commercial software, as they can make your product more difficult to install and use.)

To be ethical, you must clearly document the original source of all software and other documents. Every source file must contain header comments clearly identifying its author(s). Derivative work (e.g., code written by you but adapted from a book) must clearly cite each source used in its creation. Falsely identifying yourself as the author of something that is someone else's work, or failing to properly cite a reference on which you based part of your work, is plagiarism and will be dealt with very severely.

It is entirely possible to follow these guidelines, making only legal and ethical use of other people's work, and still to avoid a lot of design and coding that would be required if you built this project "from scratch." Sometimes you will find that, even if you cannot directly reuse code (e.g., because it is written in a different programming language), you can still reuse design. You should properly cite the sources of reused design as well as reused code.

Approximate Schedule

Week 1 (April 3-9)

Before meet with your team, start thinking about the project on your own. Design work and brainstorms are usually more productive if group members first do some thinking on their own, without the inertia of groupthink to pull everyone down a single path. Work on the issues that will be in your product concept document due at the end of week 2. Further analyze the strengths and weaknesses of the current systems. Try to find other systems to consider as well. What is the competition? Also, identify useful components that you can reuse. Think about the most effective development strategy. Should a couple team members learn MAWL and overhaul Atkins' e-turnin? Should you launch two independent development strategies in case one fails?

Week 2 (April 10-16)

You are going to need at least a couple of intense team meetings to agree on your product concept, in addition to more individual research. Think seriously about the feasibility issues: How is your team going to divide up the work and tackle the problems? Produce the SRS/SDS/Plan document and present it to the class on Tuesday, April 15.

Week 3 (April 17-23)

You should by now have already made substantial progress towards the final product. Delivery is less than two weeks away, and deadlines that looked easy before are starting to get scary. Don't panic. Do make a plan that includes early production of a running prototype (no matter how limited in functionality, it just needs to work) and frequent revisions. Make contingency plans for the failure of anything that isn't already running. You really, really want a running prototype before class on Thursday of this week, so that you are ready to discuss any remaining problems.

Week 4 (April 24-30)

It's crunch time. You're on a daily build-and-smoke schedule now. You have an agreed meeting time each day for putting together everyone's pieces and testing the current system version. The build-master is sweating bullets, but has it under control so that, if it all blows up on Wednesday, the Tuesday build is still good to go. You schedule intense reviews of documents and outstanding design issues. On Wednesday, you declare victory and turn in your project. On Thursday, May 1, you will present your project to the class.

Acknowledgments

The reuse guidelines and approximate schedule are derived from a previous assignment designed by Michal Young.