Support for Upgrading Eden Deployments
Goal
Target user: System Administrator of a production system
Goal: This is one part of making operations a big fan of our software: Make it very easy for them to upgrade.
Upgrade Issues
The current process is manual: UserGuidelines/Admin/Upgrade
For other software, people are used to automatic upgrades. Even the (very complex) Windows operating system auto-upgrades. What can make an Eden upgrade problematic? There are two types of changes that are visible to the sysadmin or user:
- Structural changes: These are changes that affect data structure and file layout.
- Database schema changes
- Directory structure changes
- Configuration process and parameter changes
- Functional changes: These are changes that would alter the site behavior or layout, in non-trivial ways. This would affect users, and might require retraining.
These issues would be present even if the site were running an unmodified version of Eden using a supplied template. If the site has made modifications to the code, even in a template, and those modifications conflict with changes in the upgrade, then the two sets of modifications need to be reconciled. Note configuration changes are expected, and need to be supported in the normal case above.
Expected issues beyond the above, that can arise if the site has code modifications, are:
- An API change may require the site to alter their code to use the modified API.
- Alteration by the site of code outside of the site's template may lead to merge conflicts.
- If both the site and upgrade have database schema changes, then those changes need to be reconciled and merged.
Of these issues, the most potentially destructive is a database schema change, as without proper preparation, running the upgraded version of Eden can lead to data loss. The Web2py migration does not understand changes such as renaming a column -- that is interpreted as deleting a column and adding a new one. This can happen regardless of whether the site has made any changes whatsoever in their code. Also, requiring the sysadmin to investigate the upgrade's schema changes, and write scripts to convert their database, is a significant burden. Addressing database migration is thus likely to be the largest reduction in upgrade effort.
Support for upgrades can proceed in stages:
- Deal with the case in which there are no significant site code changes.
That is, changes are limited to configuration and relatively cosmetic UI changes.
- Provide support for database schema changes.
- Automate configuration changes.
- Assume additional changes are confined to the site's template, but may include use of a now-deprecated or altered API.
- Deal with site database schema changes.
- Consider what can be done to ease conflict resolution in UI code.
- Likewise, for conflict resolution in model and controller code, beyond schemas.
Some potential conflicts may be avoided up front by convincing sites not to alter framework code, and to avoid directly editing models and controllers. For changes that may be of benefit to other users, encourage sites to make their changes generic and contribute them back to the main repository.
Here, we consider the first stage, in which the upgrade issues are due to changes in Eden rather than in site changes. Since directory layout changes are rare, this can be deferred until such a change is planned, unless there are sites deferring upgrades due to previous directory reorganizations.
Configuration Changes
The site may have modifications to models/000_config.py, or they may have changes in their template's config() function in config.py. These are problematic as they are Python code. Obtaining the site's settings.* values would require parsing the files in some manner.
An alternative is to replace the current Python code for configuration settings by a consolidated format such as a JSON or XML structure, or by a standard type of configuration file used in other Python products.
Options (which may be somewhat dated) are described here:
Database Schema Changes
Since this is typically the most difficult change for a site to cope with, and the most dangerous (since it risks data loss or corruption), it may be best to tackle this first.
When the database schema changes in a way that requires conversion of the database, then we would like to provide a script to perform the conversion. This is not an unusual requirement. For instance, in a Ruby on Rails system, migration scripts are mandatory -- the schema is defined as the result of applying all migration scripts in order. In an Eden, or more generally, Web2py system, the schema is defined directly, so there is no onus on the developer to write a migration script. Developers typically delete their database and reload sample data from csv files, so do not have a need to migrate existing databases. Thus, writing a migration script seems a burden.
The goal is thus to assist the developer in producing migration scripts, which they would then include along with the changes that necessitate the migration, when they submit those changes to the main repository.
A site performing an upgrade would then run an overall migration script that would create a copy of the database, and apply all the individual migration scripts starting from the site's current Eden revision, to the upgrade revision. The site would then test using the new database, and if ok, retire the previous database.
Detailed discussion of database migration (originally for a GSoC project): BluePrintDatabaseMigration
GSoC 2012 database migration project: Event/2012/GSoC/DatabaseMigration
Current state and more discussion: BluePrint/DatabaseMigration
