RTC2GIT – Rational Team Concert to GIT migration (part 2)

RTC export (RTC_export.py)

As I explained in the previous article (part 1), I use the RTC REST API instead of the RTC CLI to export the history from RTC because the RTC CLI is not reliable. However, the official (documented) REST API is too limited. The RTC Webgui also uses the REST API, but extends the REST API with an unofficial, undocumented REST API calls. These (official and unofficial) GET requests return JSON responses containing the information we are looking for. By analyzing the browser messages, you know the exact GET request (https) and the response (JSON) communication.

Reverse engineer GET request/response of the RTC Webgui

To figure out how the headers of the GET requests that the RTC Webgui uses look like, we open the page in the Webgui that shows the information. For example, login to RTC and open the page that shows all streams in Firefox, then press CTRL-SHIFT-E (Web Developer > Network), lookup and select the GET request with JSON output of the stream information (in this case it is typically the last GET with JSON response), find the URL in the Headers section and the JSON response in the Response section.

The advantage of JSON is that Python dictionaries use about the same syntax. This way, you can reverse engineer the REST API used by the the RTC Webgui and translate this into GET/JSON calls in Python quite easily.

#----------------------------------------------------------------------
# Extract information from RTC
class RTC:
    def __init__(self):
        self.session = requests.Session()
        self.login()

    #----------------------------------------------------------------------
    # Login to RTC
    def login(self):
        response = self.session.post("%s/auth/j_security_check" % RTC_URI, data={'j_username': RTC_USERNAME, 'j_password': RTC_PASSWORD})
        assert 'net.jazz.web.app.authfailed' not in response.text, 'Failed to login'

    #----------------------------------------------------------------------
    # Get all project areas from RTC
    def get_all_project_areas(self):
        url = '%s/service/com.ibm.team.process.internal.service.web.IProcessWebUIService/allProjectAreas' % RTC_URI
        response = self.session.get(url, headers={'accept': 'text/json'})
        response = json.loads(response.text)['soapenv:Body']['response']['returnValue']['values']
        return [{'name':v['name'], 'summary':v['summary'], 'ID':v['itemId']} for v in response if not v['archived']]

This example shows the get_all_project_areas function uses the internal web services and translates the JSON response into a dictionary structure. Only the ‘value’ data from the JSON response is relevant. The return value of the function is created by a Python “generator” construct to build a list (array) of relevant values from the response (list). Being new to Python, I find this a very powerful and beautiful mechanism.

Exporting RTC history

Now that we know how to extract data from RTC using the REST API used by the RTC Webgui, we can go into the export logic. First of all, we need to export from RTC now, what we need to imported into a Git repository later. There are 2 types of components in a stream:

  1. Components with history
  2. Components without history

Components without history are just a reference to a baseline of the component from another stream. Without history, there is no point in exporting the components. The history of the components with history need to be exported. These specific components are defined in components_of_interests. And the streams from which the components are exported are defined in streams_of_interest. In my case, I only export from a single stream, to be imported into a single Git repository later.

For example, exporting the history of 4 components from 2 streams, requires the following definition:

streams_of_interest = [
    'streamA',
    'streamB'
    ]
components_of_interest = {
    'streamA': [
        'component1',
        'component2'
        ]
    'streamB': [
        'component3',
        'component4'
         ]
    }

So the outer loops go through the streams of interest and the components of interest:

rtc = RTC()
for stream in rtc.get_all_streams():
    if stream['name'] not in streams_of_interest:
        continue
    for component in rtc.get_components(stream):
        if component['name'] not in components_of_interest[stream['name']]:
            continue
        print("Retrieving history for %s/%s" % (stream['name'], component['name']))
        for changesets in rtc.get_changesets(stream, component):
            for changeset in changesets:
               ...
               changeset['files'] = rtc.get_changeset_detail(stream, component, changeset)
               for file in [f for f in changeset['files'] if 'itemId' in f]:
                   ...
                   rtc.get_file(file['itemId'], file['afterStateId'], file_path)
               ...
               target = os.path.join(output_path, 'changesets', stream['name'], changeset['ID'])
               os.makedirs(target, exist_ok=True)
               with open(os.path.join(target, 'meta.json'), 'w') as fp:
                   json.dump(changeset, fp, indent=4)

Of each component, all changesets are exported. Per changset, I need to know which files have changed. The content of the files is saved (so it can be imported in Git later on), and the meta-data of the changeset is saved in a meta.json file.

In a similar way, snapshots and baselines are exported, except that they don’t need a double loop, as far as I can tell, and they don’t need to save files.

    for snapshot in rtc.get_snapshots(stream):
        print("Retrieving baselines for snapshot %s/%s" % (stream['name'], snapshot['name']))
        ...
        for baseline in rtc.get_baselines(snapshot):
            if baseline['componentName'] not in components_of_interest[stream['name']]:
                continue
            ...
            target = os.path.join(output_path, 'baselines', stream['name'], baseline['ID'])
            os.makedirs(target, exist_ok=True)
            with open(os.path.join(target, 'meta.json'), 'w') as fp:
                json.dump(baseline, fp, indent=4)
        ...
        target = os.path.join(output_path, 'snapshots', stream['name'], snapshot['ID'])
        os.makedirs(target, exist_ok=True)
        with open(os.path.join(target, 'meta.json'), 'w') as fp:
            json.dump(snapshot, fp, indent=4)

A stream consists of a set of components. A snapshot is a set of baselines for the components of the stream. Since we are only exporting the components_of_interest, the script only saves the baselines for those components. Unfortunately, I did not find a way to show all baselines of the components, only the baselines present in a snapshot of the stream.

Important to know is that a snapshot contains new component baselines only for the components that have changed since the previous snapshot. For other (not changed) components, the snapshot contains the same baselines as in the previous snapshot. This means that going through the snapshots, I will find the same baselines over and over again and only a few new ones for the components of interest.

Export details

Maximum batch size

In the middle of the changeset retrieval, you see this double ‘for’ loop.

    for changesets in rtc.get_changests(stream, component):
        for changeset in changesets:

The reason for the double loop is that RTC only returns the changesets in batches of maximum 100 changesets. The inner loop processes the changesets of a batch; the outer loop retrieves the next batch. The get_changesets function uses a neat generator mechanism in Python (yield) to produce these successive batches:

#----------------------------------------------------------------------
# Get change sets from a stream in RTC
def get_changesets(self, stream, component):
    # This function will yield changeset arrays with the batch size
    # The RTC REST api does not support values higher than 100
    batch_size = 100
    last = None

    while True:
        ...
        params = {'n': batch_size, 'path': 'workspaceId/%s/componentId/%s' % (stream['ID'], component['ID'])}
        if last != None:
            params['last'] = last['ID']

        url = "%s/service/com.ibm.team.scm.common.internal.rest.IScmRestService2/historyPlus" % RTC_URI
        response = self.session.get(url, params=params, headers={'accept': 'text/json'})
        response = json.loads(response.text)['soapenv:Body']['response']['returnValue']['value']['changeSets']

        changesets = response
        if last != None:
            # The first changeset is the same as the last changeset from the previous batch
            changesets = changesets[1:]

        changesets = [{
            'objecttype':'changeset',
            'date': changeset['changeSetDTO']['dateModified'],
            'label': [reason['label'] for reason in changeset['changeSetDTO']['reasons']] if 'reasons' in changeset['changeSetDTO'] else [],
            'message': changeset['changeSetDTO']['comment'],
            'author': '%s <%s>' % (changeset['changeSetDTO']['author']['name'], changeset['changeSetDTO']['author']['emailAddress']),
            'modified': changeset['changeSetDTO']['dateModified'],
            'added': changeset['changeSetDTO']['dateAdded'],
            'ID': changeset['changeSetDTO']['itemId'],
            'component': component['name'],
            'componentID': component['ID'],
        } for changeset in changesets]

        yield changesets

        # Stop if less than batch size are returned
        if len(response) < batch_size:
            break

        last = changesets[-1]

Yield acts similar to return (i.e. return a batch of max 100 changesets) but instead of ending the function (as return would do) processing continues with the while True loop to retrieve the next batch of changesets. This infinite loop is ended by the break. On the outside, the get_changesets function will return an array (list) of return values of the successive yield instructions, i.e. all batches of max 100 changsets. This is called a “generator” in Python.

Each exported changeset consists of the following fields, which is stored in a meta.json file in the folder identified by the changeset UUID.

  • 'objecttype' – changeset, baseline or snapshot
  • 'date' – datetime stamp of the changeset, typically latest modification date
  • 'label' – reason of the change, typically the workitem linked to the changeset
  • 'message' – comment provided with the changeset by the developer
  • 'author' – name and email address of the developer
  • 'modified' – date of the latest modification of the changeset
  • 'added' – creation date of the changeset
  • 'ID' – UUID of the changeset
  • 'component' – component modified by the changeset
  • 'componentID' – UUID of the component
  • 'files' – field filled by the get_changeset_detail function
    • 'path' – relative path of the file
    • 'type' – type of change, e.g. added, modified, renamed, deleted
    • 'itemType' – type of object, e.g. File, Folder, SymbolicLink
    • 'beforepath' – relative path of the file before a rename or move
    • 'size'– size of the file
    • 'itemId' – UUID of the file
    • 'afterStateId' – UUID of the version of the file

Non-chronological history

Another problem I ran into is that the history (of a component) is not chronological. For example, I found a sequence of changesets of a component dated in 2019 with a changeset from 2018 in between. So, I wondered how that is possible.

The reason for the non-chronological history is that the date of a changeset is de latest modification date of the changeset, not the delivery date into the stream. If a changeset is latest modified in 2018, but delivered to the stream in 2019, it sits in between the changesets created/modified and delivered in 2019.

Although the history itself is not chronological, RTC (i.e. the REST API) returns the history in the correct sequential order: newest changeset first, oldest changeset last. This is the same sequence as shown in the RTC Webgui. To keep track of the correct sequential order of the changesets, I add a sequence number to the changesets. With it, the Git import can reconstruct the right sequence order for the import.

                changeset['seqnr'] = seqnr
                seqnr += 1

Incremental export

Now suppose you have run the export for a number of hours and there is a hickup in the communication with the RTC server. The script stops. When you restart the script, you don’t want it to start over from the beginning and redo what it already did. But how do you know where it left off?

Well, you don’t! The script does go over all changesets from the beginnen, except that files already exported are not exported again. Instead it reads the meta.json file and updates the sequence number.

                target = os.path.join(output_path, 'changesets', stream['name'], changeset['ID'])
                json_file = os.path.join(target, 'meta.json')
                if os.path.exists(target):
                    with open(json_file) as fp:
                        changeset = json.load(fp)
                    print("\tAlready exported changeset", changeset['ID'], changeset['date'], "%s/%s" % (stream['name'], component['name']))

                changeset['seqnr'] = seqnr
                seqnr += 1

If the changeset was not exported yet, the files of the changeset are downloaded and saved. Then the meta.json file is written. That way it is assured that the changesets get the correct sequence number.

                if not os.path.exists(target):
                    os.makedirs(target, exist_ok=True)

                    changeset['files'] = rtc.get_changeset_detail(stream, component, changeset)
                    for file in [f for f in changeset['files'] if 'itemId' in f]:
                        file['seqnr'] = seqnr
                        seqnr += 1

                        file_path = os.path.join(target, component['name'], file['path'].replace("/", "\\"))
                        # Prefix the path with \\?\ to allow paths longer than 255 characters
                        file_path = "\\\\?\\%s" % file_path
                        os.makedirs(os.path.dirname(file_path), exist_ok=True)
                        rtc.get_file(file['itemId'], file['afterStateId'], file_path)

                os.makedirs(target, exist_ok=True)
                with open(os.path.join(target, 'meta.json'), 'w') as fp:
                    json.dump(changeset, fp, indent=4)

Conclusion

Now that we have the history from all components of interested exported from the stream of interest, we can go to importing it in Git. In principle, this involved replaying the changesets from the exported history and applying the baselines. But in practice, there are a few challenges to overcome. That is something for the next article.

Posted in configuration management, git, migration, professional, rtc, tools | Leave a comment

RTC2GIT – Rational Team Concert to GIT migration (part 1)

Introduction

Rational Team Concert (RTC) is a great, extensive and rather complicated configuration management (CM) system. Now after several years, it is time for us to move on to GIT. Git is simpler to use, more straight forward in its approach and cheaper. We have many project areas in RTC, each with many streams with many components, each stream with a long history of changes, baselines and snapshots, some of them over 5 years of history. To switch to Git, we want to migrate the history from RTC to GIT and we need a tool for that. In this article (serie) I will explain how we did it.

On the internet I found 2 solutions:

  1. rtc2git tool based on the RTC CLI (Command Line Interface), implemented in Python. See Github. There is also a version in Java.
  2. rtc2git tool based on the RTC REST API (Application Programming Interface), implemented in Bash scripts. See BitBucket

The CLI-based tool is buggy because the RTC CLI is buggy. This is a known issue at IBM. Istvan Verhas implemented an API-based tool in bash, see this article. I could not find his bash scripts, but in general I find bash scripts to be rather cryptic and unreadable, especially if it contains a lot of awk and sed commands. So, using an example of an RTC changeset exporter in Python, I decided to build my own RTC2GIT tool in Python.

At first I was a bit reluctant to dive into Python since I had not prior experience with Python. I do have some experience with Perl (over 20 years ago). To be honest, Python turns out to be a lot easier to learn than Perl, as long as you set your editor to replace tabs by spaces – Python is quite critical concerning indentations. Mixing tabs and spaces results in errors which are hardly visible for the human eye.

The real challenge was that requirements for the RTC2GIT tool evolved over time. Starting with a simple migration of a single component from a single stream, it quickly turned into a complex of requirements, e.g. multiple components, multiple streams, loadrules, incremental migration (continued development in RTC after migration to the history to Git) and adding branches (RTC streams branched off from another stream to branches in Git). It turned out to be too much for a single article, so I decided to write it down in separate articles. This is the first, introductory article where you will get a general overview.

General concepts

As I wrote in the introduction, I go for an RTC2GIT tool using the RTC REST API and written in Python. Basically, it consists of 2 tools:

  1. RTC_export.py – exporting history from RTC to a folder/file structure on a local disk (data/ folder)
  2. GIT_import.py – importing history from the local disk (data/ folder) into a Git repo

By separating export and import I am able to perform several trials for the export from RTC within redoing the import, and I can do import runs on all or a selection of the data without redoing the export. Note that an export or import may take several hours for large histories. I have done many export runs on selected components with a small history (so export is quick), and many imports on a (small) selection of exported data.

The data structure created by the RTC export looks like this:

data/rtc2git/changesets/<streamName>/<UUID>/meta.json
data/rtc2git/changesets/<streamName>/<UUID>/<path>/<file>
data/rtc2git/baselines/<streamName>/<UUID>/meta.json
data/rtc2git/snapshots/<streamName>/<UUID>/meta.json

The <UUID> is the unique identifier of the changesets, baseline or snapshot in RTC. The meta.json file contains meta-information such as the datetime stamp, deliver message, workitem, change type (e.g. “added”, “modified”, “renamed” or “deleted) and the <path>/<file> of the file or folder that has been changed by the changeset. Using the files on the local disk, the GIT import script is able to reconstruct the changeset into commits on the Git repository, which looks like:

rtc2git/.git/
rtc2git/<path>/<file>

As you know, the .git folder contains the Git repository and <path>/<file> represents the folders and files of the Git workspace. The Git workspace is (re)constructed using the exported changesets from RTC and then committed to the (local) Git repository. More details about GIT import in follow-up articles.

In the next article, I will dive into details about RTC export script.

Posted in configuration management, git, migration, professional, rtc, tools | 2 Comments

Configuration Management with SAFe

Introduction

Configuration Management is crucial for the success of an enterprise, whether it applies an agile, lean or a more traditional approach. Although CM is crucial, it has been given little attention in SAFe and other agile methodologies. In this article, I take an attempt to describe CM in concise form. Feel free to comment.

Lean-Agile Configuration Management

Lean-Agile Configuration Management is a important enabling function at all levels of the Scaled Agile Framework (SAFe). Traditionally, CM is based on the principles and practices from military traditions as captured in ISO standards [IEEE 828-2012] and CMMI [CMMI for Development, version 1.3], where the focus is on control of assets. This easily leads to blocking or delaying changes, sticking to the plan and contract agreements, and comprehensive documentation. This defies the value statements of the the Agile Manifesto. The focus of Lean-Agile Configuration Management is on enabling and supporting interactions, collaboration and change, while maintaining working software throughout the life cycles of development and operation and protecting the assets from loss or damage with a minimum of waste.

A common misunderstanding is that modern CM tools can solve the CM challenges of the organization. It is true that CM tools have improved significantly since the invention of configuration management. But although good CM tools are a prerequisite for good CM practices, in a complex software and systems development environment, good (CM) tools do not warrant good practices.

Details

In SAFe, Lean-Agile Configuration Management is a responsibility of the System Team. The System Team assists with building and using the Agile development environment. As the System Team is one of the Agile Teams, the System Team is part of an Agile Release Train (ART). For large enterprises, the System Team may be separated into an infrastructure team at enterprise level while the development teams of the ARTs take on the responsibility for use.

Lean-Agile configuration management consists of two primary responsibilities:

  • Manage assets
  • Manage change

The following sections describe the practices to implement these responsibilities.

Manage assets

For managing assets we use the input-process-output model [IPO]. Every process step requires input assets to produce output assets, which in turn are used as input assets in another process step. If the output is not being use or has no value directly or indirectly for the benefit of the customer, it is waste. Similar, input that is not used in the process is waste.
Lean-Agile Configuration Management manages both input and output assets.

IPO.PNG

Basic asset management practices are:

  • Identification – is a practice to assign an name or number to an asset. Every asset has a unique identification, for example a file name for a source file or a number of a release. If the same name or number is used within the same development environment, a location path to the source location helps on making the identification unique.
  • Version control – is a practice for keeping track of who made changes to the system, when, what was changed and possibly other metadata attributes, e.g. status. A version control system assures that subsequent versions of the assets are identified, stored, retrieval and protected against loss (deletion) and damage (change). Typically, versions of an asset are identified by a version number.
  • Structure and location – is a practice of grouping assets into a logical hierarchical relationship. File-based assets, e.g. source code or documents, can be organized in a folder structure within the repository. Object-based assets stored in a database, e.g. requirements, test cases or Bills-of-Material (BOM), use a grouping mechanism to structure the atomic assets into composite assets. Non-hierarchical structures can be defined by using ‘labels’ or ‘tags’.
    Using the structure, the location, version and name of an asset or a version can be identified, for example by repository name, version number, folder path and file name, or by Universal Resource Identifier (URI).
  • Security – is the access control practice to assure that only authorized users are able to access and/or modify the assets. Access permissions can be assigned to individual users, or to user groups. Typically access permissions are create, read, update and delete (CRUD).

More advanced asset management practices are:

  • Collaboration and sharing – assets or versions of assets are shared with other users. The security for those collaborating users may be different compared to the access permissions of the primary users. For example, a reviewer may require temporary access (for the duration of the review) to one particular version of an assets, without access to any other.
    Another form of collaboration is co-authoring. Co-authoring allows multiple authors to edit the asset simultaneously and interchangeably.
  • Releases and baselines – are defined collections of (a version of) assets. A baseline is a collection of assets serving as a starting point or reference for changes. A release is a collection of assets serving as a final point for a delivery.
  • Branching – is applied as an advanced form of versioning and structure to support parallel development. A branch represents a sequence of versions of the same asset in a different context. For example, a feature branch contains the changes for the particular feature; another feature branch contains the changes for another feature. By merging the feature branches, functionality of different features is combined.
  • Workflow or status control – is a practice to assign a status value (metadata) to an asset to identify the progress through a workflow. For collections of assets (e.g. a release or baseline) or for composite assets (e.g. a component in a bill-of-material), the status of the atomic assets may be different from the status of the composite asset. For example, a user manual may have status “final” while a deployment package containing the user manual has status “in review”.

So far, there is no difference between Agile CM and traditional CM.

Manage changes of assets

In traditional CM we identify two type of work items to manage change:

  • Change Request (CR)
  • Problem Report (PR), a.k.a. bug

In Lean-Agile CM, we have no need for change requests. Changes of plans are managed through backlog items, in the form of Epics or Enablers, or lower level work items such as Stories.

Problem Reports or bugs are not (yet) defined in SAFe, possibly because it is assumed that Built-In Quality sufficiently assures that no bugs leak through to the customer. In practice, defects do leak through which are reported back to the development organization where they are managed as PRs or bugs. We cannot plan PRs in the iteration planning or the PI planning upfront because the specific PRs cannot be predicted.

In Lean-Agile CM, we manage the following work item types:

  • Backlog items – for example Epics, Enables, Features, or Stories. Backlog items are managed through the normal backlog management practices.
  • Defect items – for example Problem Reports, bugs, or issues.

Lean-Agile CM practices for managing change are:

  • Backlog management – handling the backlog items through the normal agile planning processes as defined on the various levels in SAFe, such as portfolio, program or team level.
  • Change Control Board (CCB) – is authorized to take decisions on priority of defect issues. The defect items with high priority may overrule the iteration planning; lower priority items are planned in the iteration planning meeting.

When a high-priority defect item must be handled for the current increment or iteration, it is not an option to put it on the backlog and plan the item in the next program increment planning or iteration planning. A common practice for handling high-priority issues is an ‘expedite’ lane or ’emergency’ lane. Any issue on the expedite lane shall be handled immediately by the release train and any resource is at the disposal for the expedite lane.
Other defect items are put on the program or team backlog, where they will be handled in priority order by the teams like any backlog item.

CCB.PNG

Priority defect items from the program CCB that need to be solved in the current program increment, must be escalated to the CCBs of the affected teams to assure that they are planned and handled with the appropriate priority in the teams. For that reason, the product owners of the Team CCBs are represented in the Program CCB.

The difference between Agile CM and traditional CM concerning managing changes is that in Agile CM the goal is to respond to change as fast as possible while in traditional CM the goal is to negotiate the impact on the plan, conditions and commitments before accepting a change.

Discussions

Configuration Manager role

With traditional configuration management, the responsibility for CM activities lies with the Configuration Manager. No assets are added to the repository and no changes are applied to any item in the repository without the consent of the configuration manager.

In a lean-Agile CM context, many of the CM activities are automated or delegate to the users,  decision authority for CM is shared and delegated to the agile teams, and the CM tools have powerful security features. Responsibility for managing assets and managing changes to assets CM lies with the Scrum Master (SM), the Release Train Engineer (RTE) or the Solution Train Engineer (STE).

Backup & restore

Backup and restore are no longer subject for configuration management. The modern IT systems have evolved into reliable systems where loss and damage due to system failure has been mitigated through redundancy and data-protection measures.

Undo an operations

There is a misconception about configuration management, in particular version control, is that its purpose is to protect developers from themselves by allowing them to revert changes they have made. To support developer on this, there are two approaches:

  1. Workspace or stages
  2. Progressive reversion

A workspace is a copy of (a part of) the CM repository that is private to the developer, and not under control of configuration management. Until the changes in the workspace are committed to the CM repository, they remain private to the workspace. Most CM tools support workspaces and allow reverting (undoing) changes in the workspace. But when the changes are committed to the CM repository, they cannot be undone anymore.

To undo these changes anyway, we need to make a new change that reverts the change. In fact, the reverting change is an additional change; hence progressive reversion.

Configuration auditing

The purpose of configuration auditing is to verify compliance with specific requirements or standards. Since compliance to requirements or standards is a quality attribute, it is no longer a configuration management responsibility to audit it.

In Lean, it is preferable to prevent that quality issues during development rather than correcting them afterwards. Compliance to requirements and standards should be considered as part of the Built-in Quality.

Posted in agile, configuration management | Tagged | Leave a comment

Agile Manifesto in het Nederlands

Hoewel er al een vertaling bestaat van het Agile Manifesto naar het Nederlands (zie  hier), geef ik de voorkeur aan mijn eigen vertaling. Het zijn niet altijd letterlijke vertalingen vanuit het Engels omdat ik probeer (voor mezelf) de essenties te treffen.

Principes achter het Agile Manifest

Wij laten zien dat software ontwikkeling beter kan door dat te doen en het anderen te leren. Hierdoor waarderen we:

Mensen en interacties boven processen en hulpmiddelen

Werkende software boven volledige documentatie

Samenwerking met de klant boven contractonderhandelingen

Inspelen op veranderingen boven het volgen van een plan

Hoewel de punten aan de rechterkant wel waarde hebben, zien we meer waarde in de punten aan de linkerkant.

Twaalf principes voor Agile Software

Principes achter het Agile Manifest

Wij volgen deze principes:

  1. Onze hoogste prioriteit is een tevreden klant, door snel en vaak software op te leveren die waarde voor hem toevoegt.
  2. Verwelkom wijzigingen van de behoeftes, zelfs tot laat in het ontwikkelingproces. Door Agile processen, betekenen die wijzigingen een concurrentievoordeel voor de klant.
  3. Lever regelmatig werkende software op, elke paar weken of maanden, liefst in kortere tijdsintervallen.
  4. Mensen uit de business en ontwikkelaars moeten dagelijks samenwerken, gedurende het hele project.
  5. Vorm teams van gemotiveerde mensen. Geef hen de werkomgeving en ondersteuning die ze nodig hebben, en vertrouw erop dat ze hun werk goed  doen.
  6. De beste manier om te communiceren met of in een team is door elkaar in de ogen te kunnen zien.
  7. Werkende software is de voornaamste maat van voortgang.
  8. Agile processen bevorderen een voortdurende ontwikkeling. De sponsoren, ontwikkelaars en gebruikers moeten het tempo wel kunnen blijven volhouden.
  9. Aanhoudende aandacht voor hoogwaardige technieken en gedegen constructies versterken de agility.
  10. Eenvoud – de kunst van het weglaten – is essentieel.
  11.  Zelfsturende teams leiden tot de beste ideeën, ontwerpen, en resultaten.
  12. Regelmatig evalueert het team hoe het beter kan en past zijn gedrag daarop aan.

 

Posted in agile, software development | Leave a comment

SAFe 4.0 for small release trains

Introduction

SAFe 4.0 states that an Agile Release Train consists of 50 to 125 people. Larger teams should be split up in multiple release trains and smaller teams should be organized as individual agile teams, not as an agile release train. A second statement in SAFe 4.0 is that the Value Stream level is applicable for very large organizations of many hundreds or thousands of people, with multiple value streams and multiple agile release trains. This article covers two situations that are corner cases of SAFe 4.0: (1) small release trains of less than 50 people, and (2) managing multiple small release trains.

Situation

Suppose a product development organization is about 100 people, developing multiple independent commercial (off-the-shelf) products. One product is a mobile app that will be released to the app store every 2 weeks. The second product is a website that can be released every day, every hour or even continuously. And other products are high-tech devices that can be released no more than 2 or 3 times per year.
Now the questions is: should these 100 people be combined into a single agile release train (ART) with multiple cadences, in multiple ARTs some of which are too small for a release train (< 50 people), in one large-enough ART (>50 people) + multiple agile teams, or in agile teams without a release train. And the big question is, how to keep everything aligned if you have small ARTs, big-enough ARTs and loose agile teams?

To make the situation even more complicated. Suppose the app, website and devices form the components of an integrated system that is also commercialized. In fact, the integrated system is the USP for the individual products. And to go one more step further, other vendors may partner up to have their products and systems integrated into this system.
Now again, the question is: how should these 100 people be organized?

Solutions

Option 1: a single ART of 100 people

From a SAFe 4.0 perspective, having a single ART for 100 people is preferred. It is in the range of 50-125 people. However, from a practical point of view it is quite artificial to combine the app teams with the web teams and the devices teams.They are independent teams, with their own planning, their own technology, their own resources. Combining for instance the PI planning session and collocating them isn’t really improving the effectiveness if they work independently.

On the other hand, although being independent products they may be able to interconnect into a system. It is typical for the Internet-of-Things (IoT) that everything is interconnected to become a big system, but that does not necessarily mean that everything should be put in the same release train. With IoT you don’t know upfront which systems will actually be interacting.

Option 2: no ART, many agile teams

The other extreme is to have no release train at all, only singular Scrum teams. Each team has its Product Owner (PO) responsible for the Voice-of-the-Customer. But there may be multiple related Scrum teams. For example in app development, there may be an Android, iOS and a Windows team and they need to be aligned on feature and planning level. Also for devices, there may be very similar device that need to be aligned on feature and planning level.

So if all teams are autonomous Scrum teams, the question is how to organize the alignment between teams that need alignment. Scrum-of-Scrums is an option but that does not take into account the PO and backlog alignment between teams. Of course, you can try to invent an approach, but… hasn’t SAFe already done that?

Option 3: 1 ART for devices, agile teams for the rest

Assuming the the device development teams comprise of over 50 people, we could make an ART of the devices teams. It provides the program coordination, alignment and synchronization. However, what to do with the rest. The teams that belong together as less than 50 people, so according to SAFe 4.0 is can’t be a release train.

So in this situation, we have an ART and a set of Scrum teams, some of them belonging together. And since they belong together, we need some kind of alignment and synchronization between them. Since they are too small for a release train, we end up with 2 approaches for alignment and coordination: SAFe 4.0 ART and some kind of Scrum-of-Scrum like organization.

Option 4: 1 ART for devices, 1 ART for app, 1 ART for web

That leaves us with the ultimate solution of combining all teams that belong together in an ART, yet separating the teams that are independent into different ARTs. The devices development may be large enough to fit the SAFe 4.0 criteria; the other ARTs are smaller than 50 people. And it is a small step to assume that also the devices teams are less than 50 people.

SAFe argues that for agile teams smaller than 50 people the overhead of an ART exceed the benefits. Yet given that the other options cause a lot of confusion that is bad for effectiveness of the teams, I think making multiple small ARTs is the best option.

Dependent independence and IoT

Now that sound strange. But with the IoT technology, everything becomes interconnected including independent ‘things’. The app must be aware of the services that the devices provide, the devices may become interdependent by responding to the reaction of other devices. For example,  weather sensors detect an increase of rain and wind, a weather website predicts a thunderstorm moving towards your house, the app warns you that your sun screens are down and you may decide to open your sun screens with your app to prevent them from being damaged by the thunderstorm.

In that scenario, the supposed independent products all of a sudden become dependent because they have to know the services that other products provide. The app should have knowledge of weather reports, sun screens and other devices. But does that imply that the development teams should be part of the same ART?

Yes, no, that depends! Suppose the wind sensors and the sun screens are built by our company, but the rain sensors are not. Originally, the wind sensor was controlling the sun screens directly: when the wind is above a particular threshold, the sun screens are opened. That could be a reason to combine the rain sensor team with the sun screens team. But when the rain sensor reports to a weather reporting website independently of the sun screens there is no need to combine them.

SAFe for small ARTs

Defining mini-ARTs

Finally, I have arrived where I want to be. There may be reasons for an organization to define ARTs that are less than 50 people. Some people think that when following SAFe, all practices are mandatory. But that is not true! You are allowed to deviate from anything that SAFe defines; what is not allowed is claim that those deviations are part of the SAFe model. They are not because Scaled Agile Inc. does not define it that way and they have copyright on the SAFe model and trademark on the term ‘SAFe’.

You are allowed to create an Agile Release Train of less than 50 people and you should if that helps your organization. But you may need to reconsider the “overhead” of an ART. For example, some roles may not need to be full-time jobs, some events may be shortened. I would not recommend to skip events (e.g. PI Planning session or the I&A workshops), because SAFe is already quite minimum it the number of formal ceremonies – although at first sign many people may disagree.

Managing mini-ARTs

Although the ARTs may be independent from each other on the development activities and functionality, there may be good business reasons to establish alignment and synchronization between the ARTs. For example, the marketing strategy may require that launches of products and services to the market must be aligned into a common event for a specific commercial window, e.g. 4 weeks before Christmas. Retailers won’t be happy if they want to decorate their shops for Christmas and the products are services are launched at different dates.

SAFe 4.0 defines the Value Stream level in between Portfolio level and Program level. The Value Stream level is intended for “very big and complex organizations”. But as we have seen above, an organization may be medium size and still become rather complex with multiple mini-ARTs. Now the question is: is the Value Stream level suitable for managing multiple mini-ARTs?

No! If you look closely at the Value Stream level, it is primarily suitable for managing big value streams with big features (called “capabilities”) that are implemented in multiple ARTs because the total team is too big to fit a single ART (because a single ART would become unmanageable). The Value Stream level is not suited for managing multiple mini-ARTs that are together small enough for a single ART but that cannot be combined into a single ART.

But with a small  change, the Value Stream level can be made suitable. Since the mini-ARTs are independent release trains, with distinct functionality and architecture runways, there is not point in defining Capabilities for functionality that stretches over multiple ARTs. But you can define capabilities (epics, features, stories) for the common, aligned work, such as marketing preparations (photos, videos, advertisements, retailer decorations, etc.). And pre- and post-planning activities for the PI Planning event could be used to align multiple (mini-)ARTs with the common value stream work.

Conclusion

SAFe claims that an Agile Release Train (ART) should be between 50 and 125 people. In this articule, I have shown that mini-ARTs of less than 50 people may be applicable for medium size organizations developing multiple independent products within the same value stream. For managing the individual mini-ARTs, the concepts of the SAFe Program level are nicely applicable. Arguably, this leads to extra overhead but coordinating the agile teams without using mini-ARTs would lead to more overhead.

Although the ARTs may be independent on features, it may be required to align particular activities across the release trains that have no impact on the functionality of the individual ARTs, for example common marketing activities. The SAFe Value Stream level may be suitable for managing the common activities and assuring that the impact on the work within the ARTs is incorporated in the normal release train activities.

 

Posted in agile, complex systems, large projects, professional, systems, Uncategorized | Tagged | Leave a comment

“Olifantenpaadjes” or how to change habits


One of the most difficult things in change management is change people’s habits. Habits are like reflexes: act without thinking why and how to do it. Habits save a lot of time and energy. How difficult would walking or driving a car be if we had to think about what to do, why you do that and how you should do it.

When I go to work, I always take the same route. Boring… some people say. But it allows me to think about other things without thinking about where to go. Sometimes I try an alternative route to avoid traffic jams. And if the traffic jam happens every day, the alternative becomes my fixed route, even when traffic is low and there are no traffic jams. Just out of habit.

This is how “olifantenpaadjes” (Dutch) are born. Literally they translate to “Elephant paths”, but in English they are called “Desire paths” or “Game trails”. Usually desire paths are the result of a shortcut; cutting a corner is shorter and faster. But sometimes it is longer, e.g. going around a fence over the grass. The same with habits. Habits are usually efficient ways of doing something, but often it is an easier way not necessarily more efficient or better.

In change management, changing a way of working is like changing the roads. Deployment of the change involves instructing and training people to followed different roads and routes. But as old habits die hard, people will continue to follow the old routes which then become “elephant paths”. The reason I don’t like the term “desire paths” is because people don’t desire to follow the path, they just do out of habit following the same way as the other elephants go. One elephant going the “right” way won’t change the habit of the herd.

So what can you do? You can punish people for not following the roads (or rules), by fining them, taking disciplinary measures or put it in their evaluations. Or you can instruct and train them until their habits have changed. But since old habits die hard, people will continue to fall back to their familiar “elephant paths”.

A better way would be to make use of the natural inclination of people to take the “elephant paths”. Instead of paving the roads where you want people to go, you should build obstacles around the areas you don’t want people to go and allow them to go anywhere else. People will go around the obstacles in the easiest possible way to get to their destination. New elephant paths will grow that way and old ones will be forgotten. Then you can pave those new elephant roads and remove the paving of obsolete paths. You don’t need signs, rules or guidance to stay on the road! People will just do it.

So how does this change change management? First of all, change management should not focus on defining and deploying processes. People do not follow processes, they follow “elephant paths”. So instead, change managers should make it easy to realize the dream of success. Secondly, they should pave the “elephant paths” to make it even more easier to follow them, and clean up the obsolete paths. And finally, change managers should place obstacles at strategic places in the processes. This may be when the elephant path does where you’d rather not have it – so people start going an alternative path – or at places where you absolutely don’t want people to go regardless of a path leading there or not (e.g. restricting access to confidential information).

Posted in agile, improvement, modeling, people | Leave a comment

SAFe 3.0 versus SAFe 4.0

When SAFe 4.0 was release was my first question: which one is better, 3.0 or 4.0. In my opinion the 3-level model of SAFe 4.0 is definitely an improvement compared to SAFe 3.0.

  • The triangle RTE – System Architect – Product Manager is a much clearer representation of main roles at program level
  • The program backlog is managed by a Kanban. In Jira, we have struggled with using a Scrum board or a Kanban board, but finally the Kanban board seems to work best on program level.
  • Scrum and Kanban op team level. The team backlog is managed by Scrum or Kanban. Scrum teams can have timeboxes (sprints) of 2 or 3 weeks weeks. But some teams have a more continuous process that does not fit in fixed timeboxes, so working with Kanban is more convenient.
  • Incorporation of SAFe values and principles in the poster. Many critics of SAFe claim that SAFe is a single prescribed process, while SAFe proponents claim it is an open framework leaving room for fitting in your own processes and practices. With the SAFe values and principles more prominently depicted, it is emphasized that you have to think yourself about how to implement SAFe, within the framework.
  • Customer. In 3.0, the customer was implicit while in 4.0 it is made more explicit. Still there is much room for interpretation about what the customer actually is.

Things I don’t like about SAFe 4.0 (and to some extent SAFe 3.0):

  • Enterprise Architect. A better name would be “Business Architect”. For an IT organization, it is okay to have an enterprise architect because the products and systems being developed are for the business processes of the enterprise itself. But for a production company, the business is selling products and systems to the customer markets.
  • Product Management. A better name would be “Product Manager”. Product management is a function within the enterprise responsible for managing the products in production, marketing, sales, logistics, procurement, etc. A product manager is role that understands the features of the products or systems, and as such is a partner to the RTE and System Architect.
  • Enablers. Enablers are just a different kind of epic, feature or story. They are written, prioritized and follow the same rules as their respective Epics, Features and Stories. So why not call them Epics, Features and Stories?
  • Quality and problem solving. SAFe defines that you need to build in quality as you go, but practically you cannot – ever – build a perfect system without problems. So somehow, while working on a new iteration you need to cope with problems that emerge in past releases. I am missing that in SAFe 3.0 and still missing it in 4.0.

Now the question arises: if you have built your agile organization around SAFe 3.0, do you need a (major) transformation to adopt SAFe 4.0? No! If you implemented SAFe according to the spirit (and not according to the letter), you have made your own interpretation of each and every element of SAFe that fits your organization. This interpretation is still valid with 4.0 and you don’t need to change anything (to adopt SAFe 4.0).

Of course, with SAFe 4.0 you may have new insights that may lead to a desire to change. In fact, you should change something to improve. If there is nothing left to improve, please let me.

Posted in agile, process | Leave a comment