RTC2GIT – Rational Team Concert to GIT migration (part 1)

Introduction

Rational Team Concert (RTC) is a great, extensive and rather complicated configuration management (CM) system. Now after several years, it is time for us to move on to GIT. Git is simpler to use, more straight forward in its approach and cheaper. We have many project areas in RTC, each with many streams with many components, each stream with a long history of changes, baselines and snapshots, some of them over 5 years of history. To switch to Git, we want to migrate the history from RTC to GIT and we need a tool for that. In this article (serie) I will explain how we did it.

On the internet I found 2 solutions:

1. rtc2git tool based on the RTC CLI (Command Line Interface), implemented in Python. See Github. There is also a version in Java.
2. rtc2git tool based on the RTC REST API (Application Programming Interface), implemented in Bash scripts. See BitBucket

The CLI-based tool is buggy because the RTC CLI is buggy. This is a known issue at IBM. Istvan Verhas implemented an API-based tool in bash, see this article. I could not find his bash scripts, but in general I find bash scripts to be rather cryptic and unreadable, especially if it contains a lot of awk and sed commands. So, using an example of an RTC changeset exporter in Python, I decided to build my own RTC2GIT tool in Python.

At first I was a bit reluctant to dive into Python since I had not prior experience with Python. I do have some experience with Perl (over 20 years ago). To be honest, Python turns out to be a lot easier to learn than Perl, as long as you set your editor to replace tabs by spaces – Python is quite critical concerning indentations. Mixing tabs and spaces results in errors which are hardly visible for the human eye.

The real challenge was that requirements for the RTC2GIT tool evolved over time. Starting with a simple migration of a single component from a single stream, it quickly turned into a complex of requirements, e.g. multiple components, multiple streams, loadrules, incremental migration (continued development in RTC after migration to the history to Git) and adding branches (RTC streams branched off from another stream to branches in Git). It turned out to be too much for a single article, so I decided to write it down in separate articles. This is the first, introductory article where you will get a general overview.

General concepts

As I wrote in the introduction, I go for an RTC2GIT tool using the RTC REST API and written in Python. Basically, it consists of 2 tools:

1. RTC_export.py – exporting history from RTC to a folder/file structure on a local disk (data/ folder)
2. GIT_import.py – importing history from the local disk (data/ folder) into a Git repo

By separating export and import I am able to perform several trials for the export from RTC within redoing the import, and I can do import runs on all or a selection of the data without redoing the export. Note that an export or import may take several hours for large histories. I have done many export runs on selected components with a small history (so export is quick), and many imports on a (small) selection of exported data.

The data structure created by the RTC export looks like this:

data/rtc2git/changesets/<streamName>/<UUID>/meta.json
data/rtc2git/changesets/<streamName>/<UUID>/<path>/<file>
data/rtc2git/baselines/<streamName>/<UUID>/meta.json
data/rtc2git/snapshots/<streamName>/<UUID>/meta.json

The <UUID> is the unique identifier of the changesets, baseline or snapshot in RTC. The meta.json file contains meta-information such as the datetime stamp, deliver message, workitem, change type (e.g. “added”, “modified”, “renamed” or “deleted) and the <path>/<file> of the file or folder that has been changed by the changeset. Using the files on the local disk, the GIT import script is able to reconstruct the changeset into commits on the Git repository, which looks like:

rtc2git/.git/
rtc2git/<path>/<file>

As you know, the .git folder contains the Git repository and <path>/<file> represents the folders and files of the Git workspace. The Git workspace is (re)constructed using the exported changesets from RTC and then committed to the (local) Git repository. More details about GIT import in follow-up articles.

In the next article, I will dive into details about RTC export script.