Web Team Todo

From LU
Jump to: navigation, search

Notes after Splash Season 2011

A bunch of bugs turned up at Splash 2011, notably to MIT and Stanford's sites. We're keeping a list of bugs at Splash 2011 Bugs.

Feature requests for the next release (January 2012) should be collected here.


These are sysadmin related tasks that don't pertain to the codebase, but are equally important for delivering our services.

  • Backup and availability
    • Fix and test database replication
    • Test and deploy duplicity backups to S3
    • Get live backup sites working
    • Write failover/restore scripts and define policy
    • Add more Nagios monitoring, especially for our backup processes
    • Fix Nagios monitoring of Dropbox (enforce one process per site by directly counting dirs in /lu/dropboxes, so we don't have to constantly update the target process count)
  • E-mail
    • Add A and MX records to chapter sites so no one gets SMTP "unrouteable address" errors
    • Set or disable archiving and/or privacy settings as needed on new Google groups
    • Think about risks in our current e-mail system and see if we need any backup schemes
  • Other services
    • Set up automated testing of the code against copies of our production databases
    • Simplify and document file uploading to chapter sites (e.g. videos too large for Dropbox)
    • Work with interested parties on improving CRM situation. Drupal and CiviCRM setups need to be started fresh or a new CRM software should be identified.
    • Analyze performance risks (e.g. network bandwidth, inefficient code, overloaded database) and how to mitigate them
    • Move code.learningu.org (and maybe some other stuff) to hippo server; ideally diogenes would be for chapter sites only
  • Long term
    • Figure out what servers to use for what and where
    • Make everyone aware of how infrastructure works (or even "in case of problem X, do Y")
    • Reduce/control costs


Bug fixes

Please provide a short (1 sentence) description of the error, including the error message if you got one. Specify who reported the bug and on which site it was noticed.

Some of these may already have been taken care of. If so, please move the list items to the bottom of the page.

  • Editing a QSD repeatedly seems to yield outdated (cached?) content sometimes (MIT) Alex Dehnert 16:51, 25 February 2011 (UTC)
    • See mail from Adam to Web-team on 2011-03-10

Chapter-specific requests


  • If you update your profile e-mail and no other user has the same old e-mail address, remove the old e-mail from announcements@; and always add the new one to announcements@.
  • The Teach and Learn front pages don't give a good idea of what's coming up. If I want to plan ahead to teach for ESP (and it usually does take a considerable amount of planning ahead), I basically have to guess which link (Splash, Spark, Junction, etc.) is a program that's coming up next. The front page should instead show info about one or two programs that are coming up. Perhaps this could automatically include the program front pages or something. Rspeer 01:06, 13 May 2010 (UTC)
  • Sometimes the "Teach"/"Learn" links go somewhere nonsensical. For example, go to "Admin" and then click "Teach", and you get a 404. It should go to the main teaching site instead. Rspeer 00:58, 13 May 2010 (UTC)
  • Closing the lottery is hard (its opened by the same recursive userbit that you'd want to set to allow most stuff) Alex Dehnert 01:37, 8 March 2011 (UTC)
  • (ES?) Set up the new Boston Area ESP mailing list and support for people to choose between a particular chapter's list and the Boston Area list.


Stanford maintains their own notes on what to do with the Web site:


These are now broadly organized into categories... Please rearrange each category based on descending order of priority.

New pages/views

  • Add an interface for updating grades for students who skipped a grade.
  • Make an onsite class changes form that autocompletes based on the student's name and on the title/emailcode of the class.
  • Add schedule constraint generator for disallowing gaps in schedules. (Requested by Granite Hills; medium/low priority) Note that there is currently no view/page for generating schedule constraints of any kind.
  • Add view to get numbers for text message reminder (or better yet, send the text).
  • Make a way to extend deadlines for individual users.
  • At a glance way to see how many people are Priority/1, Interested, Enrolled, etc. in each class in a program Alex Dehnert 19:14, 27 February 2011 (UTC)
  • Add a view for adding teacher training and interview signup settings, so they don't have to be done via the non-intuitive admin interface.
  • Add a live teacher check-in page, much like the onsite student check-in pages.

Added features for existing pages

  • Add a "gender" field to the student/teacher profiles (non-required? "Other" or other accommodating option for T/Q folks?)
  • Do more complete checking of the combinations of section length and quantity. For example, if there are 2 3-hour blocks and 2 2-hour blocks available, don't allow someone to register a 3-section class where each section is 3 hours.
  • Add info to user view page: showing detailed information on current and past classes (like times, rooms, enrollment, etc. -- this helps us when teachers say things like "The room you gave me last time was too small"); showing current and past program schedules for students. This will reduce the need for morphing.
  • When someone creates a new program, it should automatically add the Student/Catalog deadline so that the catalog is viewable from when student registration opens to 9999; and it should add the deadline for teachers to view their classes from when teacher registration opens up through the day of the program.
  • have a checkbox that says "I understand that by signing up I'm committing to attending Splash" (Stanford/Ben)
  • Allow teachers to take themselves off a mailing list from their teacher page [?] (MIT/Paul)
  • Show documents from previous programs as suggestion (MIT/michael)
  • Add comm panel lists for teachers based on their training and interview selections

Behavior customization

  • Add Tag to set whether shirt statistics are collected for all teachers, or show stats for both approved teachers and all teachers.
  • Allow an arbitrary cap on the number of classes per student, instead of just "one-class registration." (Chicago/Douglas)
  • Allow caps on the number of students per school per program. (Chicago/Douglas)
  • Mailing lists generated through the website should act more like mailing lists; most notably, make the mail still appear to have come from the list when it gets to the recipients, rather than rewriting the address

User interface/workflow improvements

  • Use list_display for financial aid app modules and allow searching by program.
  • Add link to "manage volunteers" from main program management page. Add link to CSV export too.
  • Template overrides
    • Provide an easy way to edit the latest version of a particular template override, and/or provide a warning when someone attempts to edit a template override that is not the latest version. (MIT/Alex)
    • Store edit time for templates, and display it in the admin Alex Dehnert 01:34, 8 March 2011 (UTC)
    • See "Useful Django tools +1: Automatic table history" on 2011-02-24 by Adam
  • Make /myesp/switchback redirect to admin_morph/all after unmorphing (but allow this to be overridden by a Tag).
  • In the dashboard, make rejected classes red and italic, and unreviewed classes bold and blue, so that you can quickly see which is which. Put a key to this at the top of the dashboard.
  • Have onsite be permanently open (perhaps the appearance of an obvious link can be limited to the day of the program). The user and bits should be automatically created and perhaps customizable. (Duke/Dan)
  • onsite reg is kind of a nightmare. too complicated -- "what classes are available" by various criteria is really hard to find. (Stanford/Ben)
  • Fix up program modules main page
  • Top bar Ajax login box (Stanford/Mike)
  • Allow editing of class titles and descriptions only (Stanford/JD)
  • Display a popup 'Are you sure?' message when a teacher tries to submit a class with a grade range of 7-12 with no prereq's (MIT/Adam)
  • Add link to room page on teacher page (MIT/survey respondent)
  • We're creating this awesome "how to" guide for the website. It would be great if a given page/step on the website linked the the relevant instructions. [sbachar]

Technical/architectural improvements

  • Change program modules findModule() to use aux_call and main_call decorators and no longer store those fields in the database and module_properties.
  • Prevent people from doing too much with the site when their account is disabled. Alternatively, give people finer control over when they show up (generated lists, comm panel e-mails, etc.).
  • Archive and/or allow re-sending of account activation e-mails.
  • Make "manage class" load faster.
  • Make it so someone has to approve e-mails sent through comm panel (Stanford/Pdox)
  • Eliminate duplicate e-mail addresses in comm panel sending (Chicago/Louis)
  • Some form of automated scheduling would be great, but i realize that's difficult. Automated notifications of things like full classes and no availability might be more managable though. (Stanford/Ben)
  • Clean up, improve Ajax scheduling with Tony
  • Clean up communications panel and add more filters (number of class hours, text message, etc.)
  • Add comm panel filter for "Guardians of students in the program", to send morning reg announcements to. Or make a mailing list for guardians of students in each program. -jmoldow, 07/02/2011
  • Clean up accounting: update FinancialAidRequest, figure out what to do about dynamic finaid cost amounts (i.e. balance posting breaks; manually chosen amount received doesn't autoupdate). Invoice cost reflects full cost of program ignoring finaid_amount
  • Add tests for aggressively cached functions
  • Add tests for onsite (including onsite user creation)
  • Make UserHasPerms use caching that expires at the right time (MIT/Michael)
  • Events and attendance tracking (MIT/Michael)
  • Improve model/controller separation (Pdox)
    • For each controller/handler, determine how to augment the model so that the same task can be done with a series of model calls.
    • Move the appropriate code into the model and out of the handler.
    • When finished, the Model API should be far more powerful (it will contain functions to do complete tasks, such as adding a class, canceling a section & emailing students, etc).
  • Split out DataTree/UserBit into more usage-specific models like StudentRegistration. (Pdox)
  • Improve performance (Pdox)
    • Make a unified stress testing system for evaluating performance improvements/regressions and registration preparedness.
    • Make performance improvements based on the output of automated tests.


  • Remove old templates that are not used due to code rot, cache_inclusion_tag etc.
  • Remove old code/files that we aren't using.

Claimed/In Progress

  • Custom forms (Summer of Code - Vishal): Has its own Todo page at Custom Forms Todo.
  • Data viewing/export (Summer of Code - Jordan).


Email Login and Preferences

  • Login by email address (in addition to username)
  • Control of mailing preferences, so that users can opt-out of emails.
  • Add an opt-out line at the bottom of all emails sent out.

QSD Editing, Versioning, and Hidden Area

  • It should be possible to flip through existing versions of a page for reference.
  • WYSIWYG QSD editing
  • Give each QSD page a "hidden text area" for storing notes about the page, such as instructions on how to update the page, or markup that needs to be temporarily not shown.
  • Slay the markdown beast.

Specific Bug Fixes

  • Ajax login should refresh navigation bar.


Previously completed

  • Walk-in seminars can be registered for Alex Dehnert 01:37, 8 March 2011 (MIT/Jordan)
  • When a class is accepted on the "Manage Class" page, the class section gets the max size filled in. This means that if you later go back to change the size, you need to change it on both the class and class section. If the class section size is null to start with, it should be null after saving the "manage class" page unless it is explicitly saved. (MIT/Alex)
  • Allow teachers to register for teacher training right up until it happens. (MIT/Adam)
  • Redirect to /myesp/register from parent account does not work
  • In OnSite reg, on the new-student signup page, change "grade as of Fall 2008" to "grade as of Fall $YEAR" (for some value of $YEAR that will probably take some Python magic to get right, since it's this academic year, not this numeric year)
  • Set up diogenes backups (improve on archimedes system) - Until telekinesis is fixed up I'm using BackupPC on my VM kilentra.net (which also runs Nagios). I think LU should get a geographically distinct VM to use as a backup and monitoring server, but it's not worth spending money on yet. -Michael
  • Alter replication system to simplify DB-only setup for backups - This is easy enough now that you can edit/create a configuration file and run setup_repl.sh to create the management scripts. -Michael
  • Fix the way graduation years/school/employer information is handled (with Pdox) - We've been storing graduation year as a 4-letter character field and no one has complained recently. We'll see what people think.
  • Provide understandable messages for user name taken, or too long (all/Michael) - Doesn't seem to cause errors anymore, at least in main
  • Comm panel error: Key u'not_classreg' not found in <QueryDict (Stanford/Michael) - Can't reproduce this in main
  • esp/esp/../esp/program/tests.py", line 114, in testUserInfoPage self.assert_(self.admin.first_name in response.content) AssertionError - Usersearch/view looks fine in main
  • Deadlock in catalog class capacity related? (Chicago/Michael) - Fixed by MIT folks after they ran into the same issue
  • Code branch unification
  • Moved class registration (e.g. makeaclass) code to separate controller
  • Make a page to give someone administrator access.

Bug fixes

  • 6/26/2011 (Michael): Template:Program.schedule yields something titled a "student schedule" for 'teacher' users. It's accurate otherwise, so this is low priority.
  • 6/26/2011 (Michael): Fix "type object 'User' has no attribute 'ajax_autocomplete'" when you don't click something on the AjaxForeignKey dropdown
  • 6/15/2011 (Michael): Fix behavior of makeaclass when "display times" is checked in class reg moduleinfo
  • 6/9/2011 (Michael): Fix Mailman list creation (command failed: /usr/sbin/postalias /var/lib/mailman/data/aliases (status: 127, Key has expired))
  • 6/9/2011 (Michael): Get rid of Django 1.3 DeprecationWarnings
  • 6/9/2011 (Michael): Add user search box to templates.
  • 5/30/2011 (Michael): MIT: Fix Google Analytics
  • 5/30/2011 (Michael): Give MIT and Northwestern better user morph status/control like Stanford.
  • 5/30/2011 (Michael): Remove the "dummy program for profile information." What a bad idea.
  • 5/29/2011 (Michael): Error when invalid regular expression entered into user search (e.g. "last_name=*brhan") needs to be caught
  • 5/29/2011 (Michael): When a program is created with extra modules and those modules are deselected later, the ProgramModuleObjs are still there and cause the modules' main view to be shown if the module is required. We should perhaps use Program.getModules() in ProgramModule.findModule() rather than doing a query for ProgramModuleObjs.
  • 5/29/2011 (Michael): Fix the dev server setup script to drop the DB user so it can be run multiple times and replace the existing install.
  • 5/??/2011 (Michael): Make Varnish willing to wait longer than 2 minutes for responses like dashboard/printables. -> Pound is currently set to 1 hr, Varnish to 500 minutes
  • 5/??/2011 (Michael): Make student grade field default to blank so people don't accidentally leave it at 7
  • 5/??/2011 (Michael): Add simplejson, django-extensions to INSTALL dependencies list; fix syncdb/migrate directions
  • 5/??/2011 (Michael): Fix TemplateDoesNotExist on include of nonexistent program module templates (Vishal found that it doesn't silently fail)
  • 3/??/2011 (Michael): Put ajax_requests,ajax_restypes in TeacherClassRegModule module_properties
  • 3/??/2011 (Michael): /manage/Splash/Splash_2011/surveys/review: Survey review'Program' object has no attribute 'num_teachers'
  • 2/22/2011 (Michael): Prevent rapidcheckin form from checking in people other than students of the current program, or from checking in people twice.
  • 2/22/2011 (Michael): Fix silly caching in adminvitals shirt stats code.
  • 2/22/2011 (Michael): Fix Javascript error reporting
  • 2/22/2011 (Michael): Fix MIT specificity in password reset e-mail.
  • 2/15/2011 (Michael): Make template overrides update properly on save (have to tell other processes) to avoid needing Apache restart
  • 2/??/2011 (Michael): Fix MIT specificity in link on teacher application questions page.
  • 2/9/2011 (Michael):Add ipython to dependencies in dev_server_setup script.
  • 2/8/2011 (Michael): Remove "Class Application 1" column on generated lists unless it is requested
  • 2/8/2011 (Michael): Ensure everyone has the CSS and JS they need to properly use all-static QSD without killing the admin edit bars.
  • 2/7/2011 (Michael): Set up Site object correctly in DB initialization (not "example.com")
  • 2/7/2011 (Michael): Ensure that startup script creates users_userbitimplication_created_bits
  • 2/7/2011 (Michael): Fix trailing slash handling of dev server script (Stanford/Ted)
  • 1/25/2011 (Michael): Jared Davis and Amol Naik showed up on the dashboard as teaching, but didn't always show in lists I generated.... Jared showed in fewer lists than Amol. (Illinois/Anya) -> These teachers had registered using accounts (*_2007) that were disabled. The broader fix is to prevent people from doing much with the Web site unless their account is enabled (now in the queue).
  • 8/26/2010 (Michael): Make the studentinfo admin pane searchable by username
  • 8/16/2010 (Adam): Fix CSRF protection errors on: inline QSD editing, class registration (Illinois/Michael)
  • 8/4/2010 (Michael): Revert use of MIT splash page as default homepage view (Illinois/Michael)
  • 8/22/2010 (Michael): Add check for students not being in other sections already in cannotAdd (MIT/Adam)
  • 6/28/2010 (Adam): Clean up Adam's "hack" for Mailman lists in commit 9ab7dc0
  • 2/28/2010 (Michael) Can we make the default setting for Availability all-checked rather than all-unchecked? This would eliminate the problem of classes being unschedulable because one of the teachers forgot to provide availability. (Stanford/Ben) - added code in makeaclass()
  • 2/28/2010 (Michael) Need to automatically instantiate StudentclassRegModuleInfo (Stanford/Pdox) - added getModules() call in program creation code
  • 2/28/2010 (Michael) Multiple TeacherClassRegModuleInfos for user, use get_or_create (MIT/Michael) -> added unique_together constraint
  • 2/28/2010 (Michael) Make login-by-birthday sign up save your birthday for the profile form (all/Michael)
  • 2/27/2010 (Michael): Change reply-to from esp-bounces@mit.edu on non MIT sites (NU/Adam)
  • 2/26/2010 (Michael): Fix SAT Prep section division algorithm to accomodate different room sizes (MIT/Tony)
  • 2/??/2010 (Yuri) E-mail receipts (MIT/Michael)
  • 2/??/2010 (Michael): Can't pickle <class 'decimal.Decimal'>: it's not the same object as decimal.Decimal (Duke/Michael) -> WSGI process groups had to be isolated
  • 2/??/2010 (Michael): error 19 from memcached_get: SOME ERRORS WERE REPORTED (Chicago, Stanford/Michael) -> needed newest pylibmc+memcached
  • 1/19/2010 (Andrew): Double-counted students on classes with application questions (Harvard/Lester)
  • 1/23/2010 (Michael): Fix error on catalog preview (also make arrows less purple) (Duke/Reena)
  • 1/23/2010 (Michael): ProgrammingError at /teach/Cascade/2010_Winter/teacherreg (ERROR: permission denied for relation users_useravailability) (Chicago/Michael)
  • 1/23/2010 (Michael): SATPrep: classgen should approve classes/sections
  • 1/23/2010 (Michael): SATPrep: classgen should use get_or_create on rooms (by name/timeslot), set size
  • 1/30/2010 (Adam): TypeError at /admin/ajax_qsd/ save() got an unexpected keyword argument 'user'
  • Have newprogram create '/Classes' node (MIT/Adam)


  • 6/26/2011 (Michael): Make manage class page check for problems with teacher availability, mismatched schedules, etc. when modifying classes. See Ted's check_conflicts script.
  • 6/21/2011 (Vishal): Look into specifying alternative DB schema for models (e.g. for custom forms).
  • 6/??/2011 (Michael): Student search box for admins on the left underneath where the login box is placed...when convenient (Chicago/Alex)
  • 2/16/2011 (Michael): Convert cached inclusion tags to new caching API.
  • 2/9/2011 (Michael): Add a link (and wiki directions) for managing [co]teachers from the admin side.
  • 2/6/2011 (Michael): Add Tag to set whether teachers have shirt options
  • 9/??/2010 (Michael): Figure out how to decide which account survives when a user's multiple accounts are merged. -> We will be selecting the most recently created account, although we can rename accounts afterwards if a user requests.
  • 9/??/2010 (Michael): Make one-screen onsite interface -> /onsite/[program]/[instance]/status
  • 9/12/2010 (Michael): Make it possible to set (with a Tag) a minimum number of available hours for teachers to set before they go about submitting classes.
  • 9/11/2010 (Michael): Add a volunteer sign-up page like J.D's.
  • 9/6/2010 (Michael): Make the 'disposal' area on the Ajax scheduling page larger.
  • 9/6/2010 (Michael): Add availability and resources columns to the all_classes_spreadsheet view to match what the Stanford folks have done by hand.
  • 9/6/2010 (Michael): Add frequently-requested statistics page.
  • 9/6/2010 (Michael): Add "Go back to managing [program]" link when you're morphed. Note: The loginbox_ajax template needs to be included in the main.html override for this to show up.
  • 8/??/2010 (Andrew): Somehow make there be fewer duplicate accounts. (Stanford/Ben) -> user merging and forwarding
  • 8/11/2010 (Adam): Add a bit that lets you force all users to verify their account (by clicking an e-mail link) (MIT/Michelle)
  • 4/29/2010 (Michael): Improve deadline management page (NU/Liza)
  • 2/28/2010 (Michael): Make class manage page have continuously variable class size, as opposed to the discrete choice on the teacher reg page (MIT/Adam)
  • 2/28/2010 (Michael): Make it possible to send HTML e-mails through comm panel (Stanford/Michael)
  • 2/28/2010 (Michael): Customize class size editing options: "Class Size should be a free-response box that requires a positive integer less than 160. Failing that, class sizes of 40, 60, 80 and 100 should be added to the existing drop-down box. " (Stanford/Ben)
  • 2/28/2010 (Michael): Add landscape support to render_to_latex
  • 2/24/2010 (Michael): Add proper pylibmc to replication
  • 2/24/2010 (Michael): Add compiling of newest memcached to replication
  • 1/24/2010 (Michael): Convert functions to caching API

Difficulties/discussion needed

  • Add CREATE LANGUAGE plpgsql; to datatree initial data (All/Michael)

This should be done globally once per database. The command can't be issued from the initial data. I added this to the directions in INSTALL.

  • Add memcached binary protocol to replication

I tried it and it caused "error 12 from memcached_add: CONNECTION DATA EXISTS"

  • Customize what happens after you fill out your profile (forwarding to program registration, static page, showing list of programs, etc.) (Stanford/Pdox)

I made a registration_redirect view in esp/esp/web/views/myesp/main.py which applies the default logic from the profile editor and can be used (default URL /myesp/redirect). Should we just customize this function or add options in the database? If we add options in the database, what would they be?

Not actually problems

Return to the Web Team or How to use the website