Git

Sahana Eden uses the Git DVCS & hosts active branches on ​GitHub

Developer Workflow

Fork the Sahana Eden Repository

If you've not used GitHub before:

1. Sign-up for an account on ​GitHub
2. Set up git on you your computer by following ​these instructions.
   Note: In order to contribute code back to the Eden project, you need to "push" your changes to GitHub, and GitHub requires you to authenticate for that. The instructions describe authenticating using a password. If you don't want to do do that, you can use SSH keys instead, but the ​SSH setup is longer, and they do not provide instructions for Windows. If you're using Windows, you can follow ​these instructions. They're for Launchpad, not GitHub. The only difference is, while you're doing Step 5 (which has you copy the "OpenSSH" formatted public key to the clipboard), go over to the GitHub instructions, to the "Add your SSH key to GitHub" section. Ignore the Linux commands and do the part that saves your public key in your GitHub account. (You should also save that public key in a local file.)
3. The way you refer to repositories on GitHub git commands in differs depending which method you're using to authenticate to GitHub. If you are using a password, then the address of a repository has the form:

   https://github.com/username/repository.git
   

   If you're using ssh keys, then the address will be:

   git@github.com/username/eden.git
   

4. There is ​on-line help at GitHub

Get your own copy of the Eden repository (repo) on both GitHub and your local machine:

1. Fork the Eden repo at: ​https://github.com/flavour/eden/fork
2. Use git to clone your own new fork down to your PC. (Use the appropriate address form depending on whether you're using a password or ssh keys, and substitute your own GitHub username.)

   cd web2py/applications
   git clone git@github.com:mygitusername/eden.git
   cd eden
   git remote add upstream git@github.com:flavour/eden.git
   

​https://docs.google.com/drawings/d/1TppJKr9Qrq6I2KpkljRixx5gYh1seRqCXk8KDWdeIF0/edit

Ongoing Coding

Summary of coding workflow using git

cd web2py/applications/eden

# Update your local repository with latest code from the trunk repository on GitHub
git pull --rebase upstream

# Write Code

# Quick review of code (no test code left in, etc)
git diff

# Check for any new files which need adding
git status
git add .

# Commit Code (Note, no pushes to GitHub here)
git commit -am "My Changes: Part 1"
.
.
.
git commit -am "My Changes: Part N"

# Pull in changes from trunk
git pull --rebase upstream

# Resolve any conflicts (see below for how)

# Commit fixed code
git add .
git commit -a

# Review commits
git log

# Squash commits
# Use rebase to squash commits to as few as possible to keep revision history clean & make it easier to review the work
# http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html
git rebase -i HEAD~N # Where N = Number of commits to squash

# Push to your branch on GitHub
git push

Graphical representation of workflow

​https://docs.google.com/drawings/d/1KElPNMq_annNSi_2f5r9grD-c0c_aZh1KBAOqSAtuPA/edit

Once you have pushed to your branch on GitHub, you will likely want this to be merged with Trunk - this should be done via a Pull Request. This is done on GitHub:

• ​https://github.com/mygitusername/eden/pull/new/<mychanges>(the branch name)

​https://docs.google.com/drawings/d/1QmGeuQvFpg3pDQpM9xu81V7MlVy4uU8EzVIRqQ14MwU/edit

Detailed descriptions of workflow steps

Rebasing from Trunk

As you are working on a change, other changes will likely be added to the trunk repository. You may need some of those changes, or may just want to be sure your work fits in smoothly with them, or you may be preparing to submit your changes as a pull request to trunk. In these cases, you'll want to update the branch you're working on to include new revisions from trunk that are not yet in your local repository. The process for doing this is first "fetching" -- copying from the remote repository -- the new revisions from trunk, then either "rebasing" or "merging" to include them in the branch you want to update.

Although rebasing and merging may end up with the same code, the process and "side effects" differ:

• Rebasing lifts your commits off of the common base your branch shared with trunk, then inserts the new trunk revisions, then applies your commits on top of those. Merging leaves your commits where they are, and adds the trunk revisions alongside them.
• Rebasing ends with a linear sequence of commits -- each commit has a single child. Merging ends up with a directed acyclic graph of commits -- some commits will have multiple child commits, or multiple parents.
• If you encounter conflicts while rebasing, you will fix up each of your commits to deal with the conflicts, so when you're done, each of your commits will still be self-contained, free of conflicts, and consistent with the trunk commits that are its ancestors. Conflicts during a merge are handled together at the end of the merge, in a separate "merge commit" that consists of all the new commits that came from trunk, along with any edits you make to deal with merge conflicts. Your commits will still contain their conflicts -- those were fixed externally.
• There is also a difference in how your commits will appear in the log, and in particular, what order they will be in once they are accepted into trunk. Rebased commits will be shown after the new trunk commits -- this is the same order in which users of the code will receive the changes if they are updating their deployments. Merged commits are shown at the time they were written, not at the time they became publicly available in trunk.

For these reasons, it is preferable to use rebase rather than merge to include trunk changes into your local repository.

Regardless of whether you rebase or merge, neither of these is a "safe" operation -- you may make a mistake during the process, or encounter errors or conflicts. So before starting, a simple way to be sure you can go back to your unmodified branch is to make a copy of the branch.

The following summary of commands assumes you are working on branch mychanges and want to save your work on branch mychanges_backup. It also assumes you have a master branch that mirrors branch master in trunk, and that you have a remote called upstream that points to the trunk repository. This setup makes it less convenient to use the git pull --rebase command, as that (now) requires that the remote and local branch have the same name. Here we show an alternative that does a pull into your local master branch, then a rebase from your updated master branch into your mychanges branch.

# Check whether you have uncommitted changes
git status

# Commit any new and changed files
git add ...
git commit ...

# Make a backup branch
git checkout -b mychanges_backup

# Update your master branch (this is assumed to have only trunk commits on it)
git checkout master
git pull upstream master

# Switch back to the branch you want to update
git checkout mychanges

# Update your mychanges branch from your (updated) local master
git rebase master

# If you get a conflict on one of your commits, the rebase will stop and leave files with the
# conflicts marked.  See the following section about how to resolve it.  After you have edited
# any files that need to be changed, git add them and optionally commit them, then continue the
# rebase.
# Edit as per next section...
git add ...
git rebase --continue

# If, at any point before the end of the rebase, something goes wrong, or you're just not sure
# what you're doing:
git rebase --abort

Resolving Merge Conflicts

If you encounter conflicts during the rebase, the conflicts will be tagged in the files with:

<<<<<<< HEAD
# Commits from Trunk
=======
# Commits from <mychanges>
>>>>>>> "My Changes"

You can correct these conflict in your code text editor, then add them and continue the rebase:

git add -u
git rebase --continue

You can create the .THIS & .OTHER files using:

git show :2:file.txt > file.txt.THIS
git show :3:file.txt > file.txt.OTHER

You can use the changes to a file in the remote branch (theirs / OTHER):

git checkout --theirs filename.c
git add filename.c
git commit 

...or the changes to a file in your local branch (ours / THIS):

git checkout --ours filename.c
git add filename.c
git commit

...or the changes to ALL files in the remote branch (theirs / OTHER):

git checkout --theirs . # checkout our local version of all files
git add -u             # mark all conflicted files as merged
git commit             # commit the merge

Or the changes to ALL files in your local branch (ours / THIS)

git checkout --ours . # checkout our local version of all files
git add -u            # mark all conflicted files as merged
git commit            # commit the merge

Although on Windows this seems by default to produce UTF-16 files! ('UCS-2 Little Endian'). A script to make this process easier for Windows users is attached: merge.cmd

Squashing commits

Option 1:

git log
# note hash of commit that you wish to squash all commits from
git checkout <hash>
git branch -b original
git checkout master # Assuming you're working on master
git rebase -i original

Option 2:

# Stash your current work
git stash
# N = number of commits to squash
git reset HEAD@{N}
git add -A
git commit -am 'Your commit message for the one-big-commit'
# Push to your branch on GitHub, overwriting any previous code there
git push origin +master
# Restore your work in-progress
git stash pop

Undoing a commit

You may need to remove a commit you've made, either if you find you don't want it, or if it needs to be fixed. (Note if your commit has already been pushed to GitHub and accepted into the trunk, then you should use a new commit to take out unwanted changes. This section only applies if you're fixing up your repository before the changes are in trunk.)

If you want to fix up your latest commit, you can undo the commit, and unstage the files in it, by doing:

git reset HEAD~1

This will return your repository to its state before the git add commands that staged the files. Your changes will be in your working directory. HEAD~1 refers to the commit below the current tip of the branch.

If you want to uncommit N commits, but keep the code changes in your working directory:

git reset HEAD~N

If you want to get rid of your latest commit, and do not want to keep the code changes, you can do a "hard" reset.

git reset --hard HEAD~1

Likewise, if you want to discard the last N commits, and do not want to keep the code changes:

git reset --hard HEAD~N

If you are not entirely sure you want to discard the code changes, but need a branch without them, you can simply check out a new branch that ends at the last commit you want. E.g. to make a branch that ends N commits before the latest:

git checkout -b <new_branch_name> HEAD~N

See below for more on using multiple branches. To remove a commit from your repository on GitHub, use either the hard reset or new branch method above to reset your local repository to the commit you want on GitHub. GitHub will not let you overwrite a branch with a different history unless you tell it to with the --force option. If your local and remote branches have the same names, you only need to specify the branch name once, without the ":".

git push --force origin <local_branch_name>:<github_branch_name>

Code review on GitHub

Under construction

Code reviews help assure that code is correct before it gets accepted into trunk by having other people besides the author examine it, install it, test it, and comment on it. GitHub provides some very convenient features for conducting code reviews.

• The code author can push their changes to GitHub, which makes it easy for the reviewer to pull their changes into their own repository.
• GitHub allows commenting on specific lines in a file or in a diff between revisions. This makes it easy to show exactly what code is being commented on. It is not necessary to squash commits or make a pull request at this point, but the reviewer will need a way to see your changes as a whole.

To prepare for a code review:

• Optionally squash your commits.
• Push your changes to GitHub.
• If your change contains more than one commit (and you are not doing a pull request yet)

Getting your changes accepted into the trunk repository

Before submitting code to be included in the trunk repository, several things should be done:

1. Rebase from trunk and deal with any merge conflicts.
2. Squash all commits in your change into one commit. This removes any false steps and re-working, and packages the related work together.
3. Test after rebasing.
4. Push your changes to your own fork of Eden on GitHub.
5. Ask for a code review if appropriate.

Once you are ready to submit your work, make a pull request: Click Pull Requests on the right side of the GitHub page for your repo then the New Pull Request button. Note if you already have a pull request open, then changes you push to the branch that is the source of the pull request will be included in it automatically -- you do not need to close the pull request and make a new one.

Working with multiple branches

If you are working on several tasks, or want to make temporary changes for experimentation, or want to back up your work before doing a rebase or reset, you can do that by having multiple branches in your repository and switching between them. Each branch is a sequence of commits leading up to a specific head commit (which can be referred to as HEAD without confusion since git knows which branch you're referring to by which is checked out). It is very common for git users to switch between branches, and this is a fast operation in git.

To make a new branch equivalent to the current branch:

git checkout -b <new_branch_name>

To switch from one branch to another, just check out the other branch: ((( git checkout <other_branch> }}}

Note git will not let you switch to a different branch if you have uncommitted changes in tracked files. To get around this, the simplest thing to do is to commit your modified files in a temporary commit, then undo that commit later. You can also "stash" your work, which actually makes a commit on a special branch. Since it's easy to forget what's in the stash, unless you're only stashing briefly, a real commit with an informative message is probably better.

To list all the branches in the repository:

git branch -v

The -v (for verbose) tells git to show not just the branch names, but also the head commit's revision number and message.

To rename the current branch:

git branch -m <new_branch_name>

To rename a branch that is not checked out:

git branch -m <old_branch_name> <new_branch_name>

To delete a branch:

git branch -d <branch_to_delete>

Under some circumstances, git will complain when you try to rename or delete a branch. If you are sure you want to do it, use -M instead of -m for rename, or -D instead of -d for delete.

If you will be working on a specific branch for a particular event (e.g. a deployment or hackathon), then you can add that branch to your local repository:

git remote add <branch_name> https://github.com/flavour/<branch_name>.git
git fetch <branch_name>

Now you have both the master branch from the trunk repository, and the specific branch <branch_name>.

You can push branches other than master to your GitHub repository.

git push -u origin <branch_name>

Note it is good to specify the branch name to push, else git will try to push all of them. The -u tells git to make your local branch "track" the remote branch. This is only needed on the first push to that branch.

If you need a specific revision of the current branch, your can check out that revision on a new branch:

git checkout -b <revision> <branch_name>

(git requires a name for a branch if you want to preserve commits.)

Creating a second working directory and repository

If you wish to have multiple directories containing different branches, you can run these as separate web2py applications. (Note this is not a standard manner of working when using git, and is generally only needed if you are working on multiple projects at the same time, and need to preserve their databases and other state.)

• Create a fresh clone:

  cd web2py/applications
  git clone git@github.com:flavour/eden.git <new_repository>
  

• Setup a new repository on GitHub -- see: ​https://github.com/repositories/new
• Add remotes.

  git remote add upstream git@github.com:flavour/eden.git
  git remote add origin git@github.com:<my_git_username>/<new_repository>.git
  

• Push to GitHub:

  cd <new_repository>
  git push -u origin master
  

Using Patches

If you have made changes to one branch & wish to backport them to another branch, then this is done most easily using a Patch:

• ​http://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git

Story Branches

The ​Story Branch pattern allows pushing interim work to GitHub for review & also allowing contributing quick fixes to Trunk without disturbing your longer-term work. (The linked story branch description uses a Ruby on Rails (RoR) project as an example, so you'll see RoR commands in the post, such as rake or rake db:migrate. In most cases where an RoR command is shown, an appropriate Eden substitute will be to update from Trunk and run tests.)

git checkout -b <mychanges>

git commit -a
...
git commit -a

# Quick review of code (no test code left in, etc)
git diff master...HEAD

​https://docs.google.com/drawings/d/1Vhvm1EmqWOVNZkZsiJ0MLLpdTl6H_ya263Tdb2HK1N0/edit

Cherry-picking

If a Pull Request contains multiple commits & you only wish to merge one of them in, then this can be done as follows:

# Create a temporary local branch
git checkout -b cherry_temp master
# Pull all their changes into this
git pull https://github.com/their_username/eden.git their_branch
# Switch to your master branch
git checkout master
# Select the commit you wish to pull-in
git cherry-pick abc0123
# Verify new commit in-place
git log
# Cleanup
git branch -D cherry_temp
# Push
git push origin master

Developer Configuration

Tell git your name and email, which it uses to identify you as the author of commits. And tell it your account on GitHub.

git config --global user.name "Your Name"
git config --global user.email your@email.com
git config --global github.user yourgithubusername
git config --global github.token yourtoken

These are examples of other settings you can change. The values shown are not necessarily ones you'll want to use. For instance, the choice of diff and merge tools is contentious -- use your favorites.

git config --global core.editor "'C:/Program Files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"
git config --global merge.tool diffmerge
git config --global diff.tool diffmerge
git config --global difftool.diffmerge.cmd 'C:/Program Files (x86)/Git/etc/wrappers/git-diff-diffmerge-wrapper.sh' "$LOCAL" "$REMOTE"
git config --global difftool.prompt false
git config --global merge.tool diffmerge
git config --global mergetool.diffmerge.cmd 'C:/Program Files (x86)/Git/etc/wrappers/git-merge-diffmerge-wrapper.sh' "$PWD/$LOCAL" "$PWD/$BASE" "$PWD/$REMOTE" "$PWD/$MERGED"
git config --global mergetool.prompt false
git config --global mergetool.trustExitCode false
git config --global mergetool.keepBackup false

An alternative to diffmerge is Perforce's free P4V:

• ​http://danlimerick.wordpress.com/2011/06/19/git-for-window-tip-use-p4merge-as-mergetool/

Windows

Install ​Console & ​Diffmerge & ​configure Powershell.

Developer Tools

.gitattributes runs version.py to update the VERSION file on commit

Possibly we could add a pre-commit hook to run tests, e.g. based on ​http://tech.yipit.com/2011/11/16/183772396/ or ​http://blog.penzilla.net/2011/09/git-pre-commit-hook-for-python-now-with.html

Notifications

You can subscribe to commits via RSS, e.g. to subscribe to Trunk, use:

• ​https://github.com/flavour/eden/commits/master.atom

If you wish to receive email notification of commits, then you can use an Email to RSS service like:

• ​http://www.feedmyinbox.com
• ​http://blogtrottr.com

Resources

• Community Book: ​http://book.git-scm.com
• Tips & Tricks: ​http://gitready.com

History

Sahana Eden started using Bzr & ​LaunchPad but has now switched to Git & GitHub.

Why?

• The current repository is huge & bloated, taking too long to download.
• Even simple changes take an age to Pull down to servers
• Web2py no longer supports Bzr/LaunchPad, so we want to be able to maintain a common tool to work with both repositories.
• New developers are likely to be more familiar with Git/GitHub

How?

The preferred approach is to have a clean repository, in order to get the maximum benefit from the migration...although it is technically possible to ​migrate the commit history. Commit History will instead be maintained on the LaunchPad site as an archive

Migrating existing Branches

Merge branch with trunk using normal Bzr tools

Assuming that you have already ​forked Eden Trunk, then you will need to create a new repository manually for additional branches, as you cannot ​fork your own project on GitHub:

• ​https://github.com/repositories/new

  cd web2py/applications
  git clone git@github.com:flavour/eden.git <mybranch>2
  

• Move .git & .gitignore from <mybranch>2 to <mybranch>
• Delete the rest of <mybranch>2
• Edit .git/config:

  [remote "origin"]
      fetch = +refs/heads/*:refs/remotes/origin/*
      url = git@github.com:<mygitusername>/<mybranch>.git
  

• Create the initial repo for this new branch

  cd <mybranch>
  git add .
  git commit -am "Initial Git version of My branch"
  git push -u origin master
  git remote add upstream git://github.com/flavour/eden.git