1.1   Objectives

This short guide's purpose is to show you how to export a project from another Version Control System -- or even from another Git repository -- such that its history is represented as cleanly and linearly as possible.

Perhaps this project my be part of an old corporate project that you just got approval to release as open source and, although you would like it to retain as much history information as possible when you release it, you still has a need (or obligation) to strip from it all and any sensitive corporate information it has while releasing it. Maybe they are in the form of sensitive log files that found their way into the repository, or personal information (e-mails, usernames) that are in the commit log messages.

So, it's not a matter of just removing, renaming, copying or moving files around and committing -- as those files would still show up in history, revealing the information you wanted to protect and taking unnecessary repository space -- but of doing some serious cleaning and re-structuring in the source code, its history and associated meta-data -- whatever that is. This short guide is also about that.

2.1   Importing from the previous version control system

So, the first thing you must do is import your project from the Version Control System it is currently residing into a Git repository.

If it is a subversion repository, git-svn will do just fine. If you are using something else, say, perforce or CVS, similar tools exist to convert your project and its history to Git. You may need to do a intermediary conversion, say, from CVS to subversion and from subversion to Git.

For simplicity, last assume you have a project in a subversion repository. Let's also assume that the URL for this repository root is svn+ssh://svn.example.tld/secure/repositories/meh_project/ and that your project (or the files you want to export) is located in aux/super_dupper_code. The following command would fetch this project and its history from subversion into a new Git repository

git svn clone --no-metadata \
 svn+ssh://svn.example.tld/secure/repositories/meh_project/aux/super_dupper_code

First, notice that we are not converting the whole repository to Git: we are limiting as much as possible what we are importing from subversion by grabbing just the code inside the super_duper_code directory. If, for some reason you had to import the whole repository into Git, do not worry, we will explain how to "prune" it later.

The import may bring some extra files that you may want to remove say, because they are lame, for some legal reason or because they contain sensitive information that it is not OK to share with the whole world. We will completely remove them and their history from the Git repository later, hopefully leaving no trace of them whatsoever.

Since we have no interest in exporting any changes we make back to its original subversion repository, we are using the --no-metadata option here. It will also get rid of some extra git-svn-id: lines that git-svn adds at the end of every commit. Had we not used the --no-metadata option, we would need to edit the commit messages to remove them. We will also show how to modify commit meta-data (commit messages, commit authors etc) later.

2.2   Cloning a repository

"We did not even got started and we are already cloning my repository? What gives?", you may ask.

Most of the steps we will give you in the following sections will alter your Git repository in semi-destructive ways, making heavy use of git-filter-branch. I say semi-destructive because although git-filter-branch almost always makes a copy of your repository's previous state, getting back to this state may be complicated or, depending on the kind of modification performed by git-filter-branch, impossible.

Additionally, it is of our interest to get rid of any "previous state" we get and properly cloning a repository does the trick.

So, to avoid regrets and problems, let's first make a proper backup or clone of your git repository:

git clone --no-hardlinks /XYZ /ABC

Using --no-hardlinks makes Git create a clone by really coping the files and not by using hard-links. This way the original repository won't share files and metadata with its clone. See the man page if you have no idea of what I am talking about.

Another way to get the same effect is by using a file://path/to/your/git/repo URL, as documented in the section "Checklist for shrinking a repository" from git filter-branch manpage:

git clone  file://full/path/to/XYZ /ABC

With your backup done, let's move to destruct and reconstruct your Git history.

3.1   Extract a single directory

Suppose you had to bring more files from your precious VCS than you originally wanted. Say, you imported a whole CVS repository into Git and all you wanted was a project that lives inside a particular subdirectory. In this case, instead of removing all the other files and directories, it would be simpler (and saner) to extract the target subdirectory from the whole mess.

Let's suppose your target subdirectory path is projects/parsing/htmlparser. The following commands would detach it this from your repository, leaving nothing but it and its history:

git filter-branch --subdirectory-filter projects/parsing/htmlparser HEAD -- --all
git reset --hard
git gc --aggressive
git prune

Notice that the first command ends in -- --all. That's right: two dashes space dash-dash-all. That will force Git to rewrite the history for all branches and tags you have.

Now your repository consists only of the contents of projects/parsing/htmlparser and its history. Nothing more, nothing less. Well, you may have mentioned other files in your commit messages but they will not be there.

3.2   Remove files and directory from history for real

So, by now we limited our history to the enclosing subdirectory holding all the files we wanted. But there may still be some extra files that you may not want to export because they are lame, for some legal reason or because they contain sensitive information that it is not OK to share with the whole world. Let's erase them from our repository and from its history altogether.

To remove a file or a directory named path/to/SensitiveLogs from your repository, run:

git filter-branch --index-filter \
  "git rm -r -f --cached --ignore-unmatch path/to/SensitiveLogs" \
  --prune-empty HEAD -- --all

Remove all files and directory you don't want exported using the command above.

4.1   Fix committer information

Let's start this section with the committer information: its name and e-mail address.

Once again, we will use git-filter-branch to edit our commit history. This time, though, I will show two ways to accomplishing the same task.

The first is somewhat more elaborated as it shows how one can programmatically alter the committer information. Say, for instance, that except for a given committer, all other committers' meta-data are OK. So you just want to alter commits related to this guy. Let's say that this guy was you using a now invalid e-mail address. All you have to do is alter only those commits where that old and invalid e-mail is used. Here is how:

git filter-branch --commit-filter '
        if [ "$GIT_COMMITTER_NAME" = "tmacam" ];
        then
                GIT_AUTHOR_NAME=`git config --get user.name`;
                # or ...="Your (full) Name";
                GIT_AUTHOR_EMAIL=`git config --get user.email`;
                # or ...="your.email@example.tld";
                GIT_COMMITTER_NAME=$GIT_AUTHOR_NAME;
                GIT_COMMITTER_EMAIL=$GIT_AUTHOR_EMAIL;
                git commit-tree "$@";
        else
                git commit-tree "$@";
        fi' HEAD

Notice that this command is assuming that you had already configured your identification information in git. If this is not your case, just replace those git config --get xxxxxxx commands for "Your name" and "<your.email@example.tld>".

Anyway, as you can see, with some Bash programming kung-fu you can create a pretty elaborated logic on how to replace or modify committers' meta-data.

If all you want is to replace all committer information for a single identity, the following one-liner would to the trick:

git filter-branch --env-filter '\
    GIT_AUTHOR_EMAIL="your.email@example.tld";\
    GIT_AUTHOR_NAME="Your (Full) Name";\
    export GIT_AUTHOR_EMAIL;\
    export GIT_AUTHOR_NAME;\
    export GIT_COMMITTER_EMAIL=${GIT_AUTHOR_EMAIL};\
    export GIT_COMMITTER_NAME=${GIT_AUTHOR_NAME};'

And that's it. All commits will be attributed to "Your (Full) Name <your.email@example.tld>".

4.2   Fix log messages

Now time to tidy up those commit log messages. Guess what we will use for this: git-filter-branch and its --msg-filter option. You can perform almost any kind of editing with this duo: add lines, remove lines, replace text. Just give it the name of a program that will alter the log messages and that's it. The sky is the limit.

So, here is a short example of a command that will remove all those nasty "git-svn-id:" lines that you got in your log messages just because if did not read what I wrote in the Importing from the previous version control system section.:

git filter-branch --msg-filter ' sed -e "/^git-svn-id:/d" '

5.1   Shrink your repository.

Now that your repository, its history, commits and their log messages are all clean, tidy and free from shameful or sensitive information, is time to do one last thing: shrink your repository.

See, as I said before in the Cloning a repository section, git-filter-branch does store some copies of the state of the repository as it goes modifying it. Now that we got here, we don't need or want those copies. Time to get rid of them.

Go back to the Cloning a repository section and create another clone of your repository using the procedures explained there. This should give you a clean and neat clone to export/upload.

5.2   Final check

Use a tool like GitX or gitk to analyse your history and look for any missing or pending problem. Are there any empty branches you want to remove? Do the commit messages look good? Does your project has any tag or branch that should not be exported or that makes no sense in being exported? Remove them.

Fix those issues and shrink your repository once again. Yeah, your heard me right: go clean your repo once again!

Good boy.

5.3   Export it.

Well, time to export Hooray! But export to where?

Well, there are countless options -- you could setup your own git environment or use something like GitHub. I strongly recommend you taking the latter. Just head to GitHub's page, setup an account and click on the "New repository" button. Fill the presented form and follow the steps presented there. And that's it Your code now lives in a public Git repository and is there for the whole world to see. Hope you are proud of if -- I really do.

6.1   Some missing things and TODOs

I merely covered the steps I usually perform when I move code to GitHub from old subversion and CVS repositories of mine that used to hold stuff from my masters and PhD -- so, there your got it, lame code

This means that there are tons of stuff I don't cover here. For instance:

• How to add a copyright notice to all header files, from their first commit and make them persist across all changes?
• How to do the opposite: remove comments or copyright notices from files and make this removal persist across changes to the files?
• Edit the contents of some particular commit message.

And so many other issues I don't have to deal with since I own the code I am releasing. Or because I am lazy to fix everything.

Your mileage may vary

1.1   Assumptions and requirements

• No WebDav or SSH support is needed nor used for serving Git repositories.

  Once again, the focus is on HTTP access using git-http-backend. As for SSH, guides describing how to setup a similar environment for SSH-accessible repositories can be found easily on the web.

• Repositories will be password protected and available for both reading and writting.

  As we will explain latter, in the Setup git-http-backend for your repositories section, we will have to password-protect our repositories in order to be able to git push to them through HTTP.

• We will stick to a DRY (Don't Repeat Yourself) philosophy.

  Thus, we want configuration options to be repeated in as few places as possible. We will use environment variables in Apache configuration files to do this. If this makes you uncomfortable, well, you can always manually spread configuration options all over the place. Your call.

• The web server being used is Apache.

  Well, that is what Dreamhost allows me to use so that is going to be the focus of this guide. While it should not be that difficult to port the settings here to something suitable for another web server, describing how to do it is out of the scope of this guide.

• We are able to run CGI scripts.

  DreamHosts puts some restrictions on how a CGI script can be executed and the environment where it runs. We will abide to those restrictions.

• ScriptAlias is not allowed by the web server.

  The instructions given in git-http-backend manpage will not work as they use ScriptAlias. The idea is to use common CGI scripts and mod_rewrite instead, roughly following the ideas presented in http://wiki.dreamhost.com/Git#Smart_HTTP .

• SuExec is used to run CGI scripts.

  Notice that, as stated in DreamHost page on CGI, SuExec "wipes out all environment variables that don't start with HTTP_". All our env. vars. will have this prefix.

• Your private git repositories will be accessible in a subpath of your domain.

  The idea is that your private git repos will be available in an address such as http://www.example.tld/corporate-git/. Adapting the instructions bellow so you can serve them from the root of a domain of its own, say http://corporate-git.example.tld should be fairly simple.

1.2   About this document

This document and its companion files are initially hosted on http://github.com/tmacam/private-git-on-dreamhost.

The file README.rst is generated from README-real.rst. So, if you plan on doing any updates or fixes, README-real.rst is the file you ought to edit. Just run make afterwards in order to get README.rst updated as well. This is done because I wanted to use GitHub's automatic rendering of README files but I didn't want to just paste the contents of the companion files in this README.rst and risk getting the Guide and files out of sync. Unfortunately, GitHub does not allow the use of RestructuredText's include directive, so I had to fake it -- and here is the reason why we have README-real.rst.

This guide is distributed under the Creative Common BY-SA license while companion files are distributed under a MIT License.

2.1   Install Git

Well, this should probably be a non-issue since git comes pre-installed on most Dreamhost machines. To verify it:

$ git --version
git version 1.7.1.1

As you see, the box that serves my domain in dreamhost has git version 1.7.1.1 installed. Anything greater than 1.6.6 shall do.

If you don't have git installed in you box, have an old version or if for some other reason your need to compile git, follow Craig's instructions in Hosting Git Repositories on Dreamhost.

2.2   Create the directory where your repositories will live

It should reside somewhere not accessible from the web or directly served by the web server. We will tell Apache and git-http-backend how to properly and securely serve those repositories latter. For now, we want them protected from third parties.

Say we decided to store them in ~/private_repos/. We will refer to this directly by GIT_REPOS_ROOT in the rest of this guide. Create this directory and protect it against filesystem access from others:

export GIT_REPOS_ROOT="~/private_repos/"
mkdir ${GIT_REPOS_ROOT}
chmod 711 ${GIT_REPOS_ROOT}

2.3   Setup the bare repository creation script

We will use the script newgit.sh, presented bellow, to create new repositories [1] [2] . Remember to modify the value of the GIT_REPOS_ROOT variable in it to match our setup:

#!/bin/bash
 
# this script is based on code from the following blog post
# http://arvinderkang.com/2010/08/25/hosting-git-repositories-on-dreamhost/
# and http://gist.github.com/73622
 
 
set -e
 
 
# Please, configure a default GIT_REPOS_ROOT to match your config
#GIT_REPOS_ROOT="~/private_repos/"
 
DEFAULT_DESCRIPTION='no description :('
 
 
# describe how the script works
usage()
{
  echo "Usage: $0 [ -h ] [ -r directory] [ -d description ] [ -n projectname ]"
  echo ""
  echo "If no projectname is given, the name of the parent folder will be used as project name."
  echo ""
  echo "  -r directory   : (root) directory holding your git repositories"
  echo "  -d description : description for gitweb"
  echo "  -h             : print this screen"
  echo "  -n name        : name of the project (should end in .git)"
  echo ""
}
 
DESCRIPTION=${DEFAULT_DESCRIPTION}
 
# evaluate the options passed on the command line
while getopts r:d:n:h option
do
  case "${option}"
  in
    r) GIT_REPOS_ROOT=${OPTARG};;
    d) DESCRIPTION=${OPTARG};;
    n) REPONAME=${OPTARG};;
    h) usage
      exit 1;;
  esac
done
 
# check if repositories directory is given and is accessible
if [ -z $GIT_REPOS_ROOT  ]; then
    usage
    exit 1
fi
if ! [ -d $GIT_REPOS_ROOT  ]; then
    echo "ERROR: '${GIT_REPOS_ROOT}' is not a directory"
    echo ""
    usage
    exit 1
fi
 
 
# check if name of repository is given. if not, use folder name
if [ -z $REPONAME ]; then
  REPONAME=$(basename $PWD)
fi
 
# Add .git at and if needed
if ! ( echo $REPONAME | grep -q '\.git$'); then
  REPONAME="${REPONAME}.git"
fi
 
 
#
# Ready to go
#
 
 
REP_DIR="${GIT_REPOS_ROOT}/${REPONAME}"
mkdir ${REP_DIR}
pushd ${REP_DIR}
git --bare init
git --bare update-server-info
cp hooks/post-update.sample hooks/post-update
chmod a+x hooks/post-update
echo $DESCRIPTION > description
# This mark the repository as exportable.
# For more info refer to git-http-backend manpage
touch git-daemon-export-ok
popd
exit 0

Move or copy this file to an appropriate path (say, your home directory would be fine) and turn it into an executable:

chmod u+x ~/newgit.sh

2.4   Apache Setup

Now, let's configure Apache to securely serve those repositories.

cd ~/www.example.tld
mkdir corporate-git
cd corporate-git
export GIT_WEB_DIR=`pwd` # we will use it in latter steps
touch .htaccess
chmod 644 .htaccess

Options +Indexes
 
# GIT BEGIN ###########################################################
 
SetEnv HTTP_GIT_PROJECT_ROOT /home/user/private_repos/
SetEnv HTTP_GITWEB_CONFIG /home/user/private_repos/gitweb_config.perl
 
 
RewriteEngine On
DirectoryIndex  gitweb_wrapper.cgi
# The following two rules can be used instead of DirectoryIndex
#RewriteRule ^$  gitweb_wrapper.cgi/ [L,E=SCRIPT_URL:/$1]
#RewriteRule ^([?].*)$ gitweb_wrapper.cgi/ [L,E=SCRIPT_URL:/$1]
 
# Everything else that is not a file is forwarded to git-http-backend
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^?].+)$ git-http-backend-private.cgi/$1
 
 
# GIT END ############################################################
 
# AUTHENTICATION BEGIN ###############################################
AuthType Digest
AuthName "Private Git Repository Access"
# UNCOMMENT THE LINE BELLOW FOR BETTER PERFORMANCE
# AuthDigestDomain /corporate-git/
AuthUserFile /home/user/private_repos/.htpasswd
Require valid-user
# AUTHENTICATION END  ################################################

$ (cd ${GIT_REPOS_ROOT}; pwd)
/home/user/private_repos/

#!/bin/sh
export GIT_HTTP_EXPORT_ALL=1
export GIT_PROJECT_ROOT=${HTTP_GIT_PROJECT_ROOT:?HTTP_GIT_PROJECT_ROOT env. variable not set. Aborting.}
/usr/lib/git-core/git-http-backend

chmod 755 git-http-backend-private.cgi

# AUTHENTICATION BEGIN ########################
AuthType Digest
AuthName "Private Git Repository Access"
AuthUserFile /home/user/private_repos/.htpasswd
Require valid-user
# AUTHENTICATION END  #########################

touch /home/user/private_repos/.htpasswd
htdigest /home/user/private_repos/.htpasswd "Private Git Repository Access" username

git --version

cd ~ # Yep, we will download it in our home directory
wget http://www.kernel.org/pub/software/scm/git/git-1.7.1.1.tar.gz

tar zxvf git-1.7.1.1.tar.gz
cd git-1.7.1.1
make prefix=/usr/bin gitweb/gitweb.cgi
rm gitweb/gitweb.perl # we won't need it

export GITWEB_INSTALL_DIR="~/gitweb"
cp -r gitweb ${GITWEB_INSTALL_DIR}

cp ${GITWEB_INSTALL_DIR}/*.{css,png,js} ${GIT_WEB_DIR}
# in this example, GIT_WEB_DIR points
# to ~/www.example.tld/corporate-git

#!/bin/bash
export GITWEB_CONFIG=${HTTP_GITWEB_CONFIG:?HTTP_GITWEB_CONFIG env. variable not set. Aborting.}
export GIT_PROJECT_ROOT=${HTTP_GIT_PROJECT_ROOT:?HTTP_GIT_PROJECT_ROOT env. variable not set. Aborting.}
 
${HOME}/gitweb/gitweb.cgi

chmod 755 gitweb_wrapper.cgi

# where is the git binary?
$GIT = "/usr/bin/git";
# where are our git project repositories?
$projectroot = $ENV{'GIT_PROJECT_ROOT'};
# what do we call our projects in the gitweb UI?
$home_link_str = "My Git Projects";
#  where are the files we need for gitweb to display?
@stylesheets = ("gitweb.css");
$logo = "git-logo.png";
$favicon = "/favicon.png";
# what do we call this site?
$site_name = "My Personal Git Repositories";

2.5   Troubleshooting

So, something is not working as expected?

#!/bin/sh
 
# disable filename globbing
set -f
 
echo "Content-type: text/plain; charset=iso-8859-1"
echo
 
echo CGI/1.0 test script report:
echo
 
echo argc is $#. argv is "$*".
echo
 
echo SERVER_SOFTWARE = $SERVER_SOFTWARE
echo SERVER_NAME = $SERVER_NAME
echo GATEWAY_INTERFACE = $GATEWAY_INTERFACE
echo SERVER_PROTOCOL = $SERVER_PROTOCOL
echo SERVER_PORT = $SERVER_PORT
echo REQUEST_METHOD = $REQUEST_METHOD
echo HTTP_ACCEPT = "$HTTP_ACCEPT"
echo PATH_INFO = "$PATH_INFO"
echo PATH_TRANSLATED = "$PATH_TRANSLATED"
echo SCRIPT_NAME = "$SCRIPT_NAME"
echo QUERY_STRING = "$QUERY_STRING"
echo REMOTE_HOST = $REMOTE_HOST
echo REMOTE_ADDR = $REMOTE_ADDR
echo REMOTE_USER = $REMOTE_USER
echo AUTH_TYPE = $AUTH_TYPE
echo CONTENT_TYPE = $CONTENT_TYPE
echo CONTENT_LENGTH = $CONTENT_LENGTH
echo ""
echo HTTP_GIT_PROJECT_ROOT = $HTTP_GIT_PROJECT_ROOT
echo HTTP_GITWEB_CONFIG = $HTTP_GITWEB_CONFIG
 
exit 0

[Mon Oct 25 18:30:28 2010] [error] [client 150.164.3.192] Service not enabled: 'receive-pack'

Digest: user username: password mismatch: /corporate-git/test.git/info/refs

3.1   Creating new bare repositories

In order to create a new repository, say toyproject.git, all you have to do is ssh into your Dreamhost account and:

~/newgit.sh -r ${GIT_REPOS_ROOT} -d "My first private repository" -n toyproject

That's it: your created and empty repository in you repository collection. You can clone it if you want.

3.2   Cloning an empty repository

So, you got a new pristine and empty repository. Let's clone it, shall we?:

$ git clone http://username@www.example.tld/corporate-git/toyproject.git
Initialized empty Git repository in /private/tmp/teste/.git/
Password:
warning: You appear to have cloned an empty repository.

But what if you already have a local repository and all you want is push it and its history to the server?

3.3   Pushing to a new empty repository

What you usually do is creating a local repository, adding file to it and committing this repository history to the new, empty and pristine repository in your web server:

mkdir toyproject
cd toyproject
git init
touch README
git add README
git commit -m 'first commit'
git remote add origin http://username@www.example.tld/corporate-git/toyproject.git
git push origin master

If you have an existing Git Repo, that's the procedure:

cd existing_toyproject_git_repo
git remote add origin http://username@www.example.tld/corporate-git/toyproject.git
git push origin master

The above workflow follows what is presented in http://help.github.com/creating-a-repo/.