wiki:SysAdmin

Version 74 (modified by Fran Boon, 12 years ago) ( diff )

--

SysAdmins

Table of Contents

  1. Deployment Cycle
  2. Setting up a New Instance
    1. Copy from Template
  3. Upgrading an Instance
  4. Continuous Integration
  5. Trac Maintenance
    1. Backups
    2. Customizing Trac
      1. Settings
      2. Interface
      3. Plugins
  6. Mail Queues
  7. References
  8. Pootle
  9. ToDo
    1. Windows Installer
    2. Infrastructure Virtual Machine Template
    3. Fabfile
    4. Test
    5. Demo
    6. OSGeo LiveDVD
    7. Pootle
    8. Trac
    9. Release Process
    10. Translation Process
    11. MeetBot

We can always use SysAdmin support to help out with our infrastructure: developing tools & maintaining sites.

Deployment Cycle

Setting up a New Instance

Eden project have 2 servers kindly donated by ​Zen Internet.
These run KVM to provide different virtual machines.
We use ​Proxmox to manage these VMs:

  • Type: Fully Virtualized (KVM)
  • Installation media: Debian-507-amd64-netinst.iso (if not using the template)
  • Disk Storage: Images (Host1) data (Host2)
  • Start at Boot: tick
  • Disk type: VIRTIO
  • Disk size: 22GB
  • Guest Type: Linux-2.6
  • Network Card: virtio
  • Bridge: vmbr0

These documents are secured - ask for access if you are going to contribute:

Copy from Template

We normally copy an instance from a predefined template rather than starting with a fresh OS build.

  • consider changing this with scripts similar to ISO/VM builds?

Instances run WSGI & MySQL.

  1. Turn off template VM instance if on
  2. SSH into the host machine

If using Host1:

3. 

cd /spare/images/images
cp 106/vm-106-disk-1.raw 1xx/vm-1xx-disk-1.raw

If using Host2:

3. 

#zcat /spare/template/vm_release.gz | dd of=/dev/data/vm-1xx-disk-1 bs=8M
dd if=/dev/data/vm-106-disk-1 of=/dev/data/vm-1xx-disk-1 bs=8M
  1. That'll take a while. You can get it to notify progress by doing a kill -USR1 {process id of the dd}
  1. start your VM after the copy is complete
  2. Open VNC Console
  3. read /home/INSTALL in the vm
  4. Set the IP (using the spreadsheet)

Upgrading an Instance

Need to decide whether the site will be upgraded straight from Trunk, or whether upgrades are staged via a Test server.

Simple upgrades can be done using the aliases:

We manage complex upgrades using Fabric:

  1. Add the site to fabfile.py
  2. If no keys generated yet, then generate some: 
    fab generate_keys
  3. deploy keys 
    fab <site> distribute_keys
  4. Run the upgrade (as/when required) 
    fab <site> deploy

Continuous Integration

CI Server results: ​http://82.71.213.53/eden/admin/result

To run the test suite manually: 

sudo sh /etc/cron.custom/web2pystarter

Open another console then: 

sudo sh /etc/cron.custom/testsuite

This first updates Eden and Web2py from Github, then runs the functional tests followed by smoke tests.

To just run smoke tests: 

cd /home/web2py
xvfb-run -a python web2py.py -S eden -M -R applications/eden/modules/tests/suite.py -A --html-path /home/web2py/applications/eden/static/test_smoke/ --suite smoke --force-debug --link-depth 16 -V 4

We also have an install of Jenkins which should ideally be configured to run this.

Trac Maintenance

To clear blocks from MathCaptcha failures:

To reset tickets which have been assigned to nobody: 

apache2ctl stop
sqlite3 /home/sahana/trac/db/trac.db
UPDATE 
  ticket
  SET status = 'new'
  WHERE owner = '' and status = 'assigned';
.q
apache2ctl start

To delete a ticket: 

apache2ctl stop
sqlite3 /home/sahana/trac/db/trac.db
SELECT * FROM ticket WHERE id is 0000;
DELETE FROM ticket WHERE id is 0000;
.q
apache2ctl start

Backups

​http://bitten.edgewall.org/wiki/TracBackup 

trac-admin /home/sahana/trac hotcopy /root/trac_backup
NOW=$(date +"%Y-%m-%d")
tar cvf trac-$NOW.tar /root/trac_backup
bzip2 -9 trac-$NOW.tar
rm -rf /root/trac_backup

Customizing Trac

Just dropping in a few notes. --Pat
Trac and Trac plugins are installed in:

/usr/local/lib/python2.6/dist-packages/

Trac itself is: Trac-0.12.1-py2.6.egg

The Eden Trac "project" is at:

/home/sahana/trac

Settings

Many settings are available in: /home/sahana/trac/conf/trac.ini
Some interface changes can be made here as well, e.g. icon, order of items in the top menu, labels for same (search for mainnav and metanav).

Interface

​http://trac.edgewall.org/wiki/TracInterfaceCustomization

  • The official Trac means of customizing individual pages is to use Genshi xsl to replace items in the original pages, which are left untouched. The replacement is done in: /home/sahana/trac/templates/site.html
  • One can also put entire replacement templates in /home/sahana/trac/templates but then one will have to merge these when Trac is updated. Putting entire templates here is problematic for plugins, which may define their own local data -- that won't be available outside the plugin context.

Plugins

Watch this space...

Mail Queues

Check if your IP is blacklisted for Spam:

View queue: 

mailq

Flush queue (inc Frozen messages): 

exim -qff

Remove Frozen messages (remove -r to remove all): 

exiqgrep -z -i -r [user]@domain | xargs exim -Mrm

References

Pootle

UserGuidelinesLocalisation#AdministeringPootle

ToDo

Windows Installer

See if we can upgrade the Windows Installer to Python 2.7 to have Foreign-Key support (#900)

Use Esky to provide an auto-update method:

Infrastructure Virtual Machine Template

  • /home/INSTALL
  • /root/ChangeTemplate.sh
  • scripts in /usr/local/bin
  • Convert to PostgreSQL/PostGIS
  • Update Eden
  • Update Web2Py (as-required)
  • Update OS
  • Increase swap to 2048Mb (from 750Mb)
    • can we do this in image? Add an emergency swapfile?
    • Add extra disk for those instances which need extra capacity

Fabfile

  • SQL indexes need restoring after DB upgrades
  • Add support for PostgreSQL
    • Add support for PostgreSQL to dbstruct (or have an equivalent file)

Test

We have an instance of Jenkins to automatically run through our Selenium tests: SysAdminJenkins

  • Fix the current builds
  • Trigger builds upon every commit
  • Email notifications: set up a mailing list which people can register/unregister themselves
  • Add timings to the output
  • Add a VPN Server (e.g. PPTPd) to allow NAT'd desktops to contribute tests (e.g. IE7)

Demo

  • Have a set of reference data available

OSGeo LiveDVD

Pootle

Trac

Errors: 

/usr/local/lib/python2.6/dist-packages/TracScreenshots-0.8dev-py2.6.egg/tracscreenshots/core.py:13: DeprecationWarning: the sets module is deprecated
  import sets
Traceback (most recent call last):
  File "./sqlite2pg", line 335, in <module>
    sys.exit(main(sys.argv[1:]))
  File "./sqlite2pg", line 331, in main
    Main(opts)
  File "./sqlite2pg", line 250, in Main
    rc = tmigration.migrateTable(tname) or rc
  File "./sqlite2pg", line 72, in migrateTable
    rc = self.default_copy(table)
  File "./sqlite2pg", line 93, in default_copy
    pgcur.execute(insert_into, row)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block
  • Add Timing & Estmation Plugin
  • Investigate a fix or alternative to ​http://trac-hacks.org/wiki/MathCaptchaPlugin for allowing Trac users to register bugs anonymously whilst not locking out our testing team.
  • See if we can not send emails out to users for the updates they make themselves (maybe fixed in 0.12.x?)
  • Investigate a fix for the last section edit doubling-up the footers (maybe fixed in 0.12.x?)

Release Process

Translation Process

  • Script to update language files: 
    cd web2py
python web2py.py -S eden -R applications/eden/static/scripts/tools/languages.py
  • Scripts to automate the conversion 2 PO & upload/download to/from Pootle:

Unfortunately it doesn't seem that we can currently automate the upload back to Pootle:

MeetBot

We could use a MeetBot to log our IRC meetings better:

Projects/Infrastructure

Note: See TracWiki for help on using the wiki.