Implementing SAFe with Jira (part 3)

In part 2 we have looked at estimating and planning of a release of the Agile Release Train. This is mainly on SAFe portfolio and program level. In this part, I will focus more on the challenges of estimating and planning on program and team level. Since Jira does not offer much cross-project support, we will be using the structure plugin. So first, we must get our structure boards right.

Structure plugin (revisited)

Last time, I explained that you can make a structure board with the structure plugin, and populate and locate issues on the structure board with synchronizers, in particular:

  • Filter synchronizer – use a filter query
  • Agile synchronizer (JIRA Agile (Greenhopper) – using Epic links and Subtasks
  • Links synchronizer (Issue Links synchronizer) – using “Implement” links

The filter synchronizer can be configured to remove issues that are outside the filter query results. For example, if you filter on ‘project = PSYS AND status != Closed’ the filter synchronizer removes issues when they are put in Closed status. But, it will also remove issues for other projects than PSYS. And given that the links synchronizer will typically add issues from other Jira projects (e.g. the portfolio project or product development projects), a resync of the filter synchronizer will remove issues added by the links synchronizer. We don’t want that!

My first instinct was to configure the filter synchronizer not to remove issues, only add them. That way, issues added by the agile and links synchronizers remain on the structure board. It worked at first, but then issues of the product teams started popping up on the system structure board. And obviously, they were not removed. So, why did they pop up and how do I get rid of them?

They popped up by “clone and move”. Suppose we want to clone a system epic on program level to a product epic for a particular project on team level. Step 1 is to clone the system epic. The result is that 2 identical system epics are on the system structure board. Step 2 is move the (cloned) system epic to the product project. Jira actually renames the issue id (key). The structure plugin shows the renamed issue id on the structure board because the filter synchronizer is not configured to remove it. So, now the question is, how to get rid of it on the structure board.

  • Manual way 1: select the issue and press the ‘x’ button on the structure board.
  • Manual way 2: select all issues and press the ‘x’ button on the structure board, which makes the board empty. Then resync the filter synchronizer and then the agile and links synchronizers to add, sort and locate issues on the structure board
  • Automatic way: extend the filter of the filter synchronizer and configure it to remove issues you don’t want

In Jira it is not possible to define a query to find all issues that the Agile and Links synchronizers add. The query language (JQL) is just too limited for that. But the structure plugin does add the function structure(<structure>,<query>) to find issues on a structure board. Now if the filter synchronizer adds PSYS issues and the agile and links synchronizers only add sub-issues underneath, the top issues are always PSYS issues. For any top that is not of the PSYS project should be removed. Result:

project = PSYS OR structure(“System structure”,”ancestor in [project = PSYS]”)

In English: all PSYS issues and all sub-issues on the structure board underneath a PSYS issue. If you want multiple projects on the structure board you can use project in (PSYSA,PSYSB,PSYSC) instead.

Jira boards for SAFe

Let’s have a look back: what boards do we have for SAFe implementation in Jira?

  • Portfolio level
    • Portfolio Kanban board – manage the workflow of the portfolio backlog items
    • Portfolio Structure board – visualize the hierarchy from portfolio to products
  • Program level
    • System Kanban board – manage the workflow of the system backlog items
    • System Structure board – visualize the hierarchy from system to products
  • Team level
    • Product Scrum board – manage the workflow in sprints of product backlog items
    • Product Structure board – visualize the hierarchy from product to portfolio

On the portfolio structure board and the system structure board, it is only a downwards hierarchy to the product. On the product structure boards, it is an upward hierarchy to the portfolio, which requires a different filter for the synchronizer:

project = PRDA OR structure(“Product A structure”,”descendantsOrIssue in [project = PRDA]”)

Of course the links synchronizer needs to be configured differently to locate issues in an upwards hierarchy, i.e. adding parents instead of sub-issues.

Orphans

Orphans are backlog items on the system or product backlog that are not related directly or indirectly to an item on the portfolio backlog. Portfolio backlog items represent value to the business. Since an orphan is not related to a portfolio item, it does not add value to the business and therefore working on an orphan is wasted effort. Of course, if links are missing they may seem like orphans but they are not.

To be able to eliminate waste (which is a Lean principle), we strive for linking all backlog items, to a portfolio backlog item. This includes defects; more on defects later. For example, a story (on team/product level) may be linked to an product epic (through an epic link), which is linked to a product epic of another team (through a “implement” link), which is linked to a system epic (through a “implement” link) which is linked to a business epic (through a “implement” link) on the portfolio backlog. So working on the story adds value to the portfolio backlog item and thus it is not waste.

Finding orphans is quite simple: all backlog items without an ancestor (i.e. parent, grandparent, …) on portfolio level is an orphan. As a query, assuming PBUS is the portfolio project in Jira:

structure(“Product A structure”,”ancestorOrIssue not in [project = PBUS]”)

Estimating and planning

Now that we have structure boards cleaned up of orphans, we can finally start estimating. One of the nice things about the structure plugin is that it can accumulate estimates: original estimate, remaining estimate and logged work. For each estimate it shows 2 numbers: excluding and including the estimate of the underlying item. So when the structure hierarchy is complete, you can nicely see the total of original estimate, remaining estimate and logged work, assuming the numbers are filled in right.

However, in practice the hierarchy is never complete. There are portfolio items which have not yet been refined into system items, and there are system items not yet refined or partially refined into product items. Here is the dilemma:

  • Complete the hierarchy before estimating a portfolio item
  • Estimate the portfolio item before completing the hierarchy

Completing the hierarchy takes a lot of time and effort. You need to do an impact analysis and technical (architectural) decision to decide the affected system and product parts. You should do that only for backlog items early on the roadmap. However, if you estimate before making the hierarchy complete, the estimate may be unreliable. So what to do?

Simple answer: estimates on any level (portfolio, program or team level) should take into account the velocity at that level only. The story point estimate is a size estimate for a backlog item relative to other backlog items on the same level. Don’t take underlying estimates or velocities into account! Based on the (recent) past, calculate the velocity (story points per time unit). For the upcoming iteration (e.g. program increment on program level, or sprint on team level), set the planned velocity equal to the velocity of the last iteration. Then adjust:

  • If the velocity over the past iterations shows a trend (e.g. increasing), adjust the planned velocity in the extrapolated direction
  • If there is a significant change in the development staffing (e.g. new people,  reduction, outsourcing), technology (e.g. new platform, new OS, new protocols) or another factor that may affect the velocity, estimate the impact on the velocity and adjust the planned velocity accordingly
  • If there are temporary circumstances (e.g. holidays, absence of key people), estimate the impact on the velocity for backlog items high on the backlog and adjust the planned velocity accordingly. Consider delaying the backlog items that are hampered by the temporary circumstances

With a planned velocity and story point estimates (on portfolio, program or team level), you can determine how may of the (top) backlog items you can plan for the next iteration.

Why do you need the hierarchical structure anyway? You don’t need it for estimation! You do need it for planning. On team/product level, the sorting order of the product backlogs should satisfy the sorting order on program/system level. And on program/system level the sorting order of the system backlog should satisfy the sorting order on portfolio level.

The backlog hierarchy also allows cross-checking the sanity of the estimates for velocity and story points. For example, if the program velocity is planned to be reduced by 10% due to holidays but actually the average team velocity drops by 30% during the holidays, you know that you should increase the expected velocity drop for the next holiday season. Similarly, if a backlog item is 2x the size of another backlog item on program level, but the underlying product backlog items are 4x the size, you need to analyze which type of backlog items are over or underestimated and adjust the estimates accordingly.

Conclusion

In part 1 and part 2, we discussed the 3-layered structure and how the structure plugin supports it. In part 3 (this part), we discussed orphans in the structure and estimating and planning across the portfolio, program and team levels. Important to remember is that story point estimations and velocity planning is independent from the hierarchy. The hierarchy is merely used to assure the consistency between the portfolio, program and team level in sorting order (planning), story points (estimating) and velocity (planning). I am not sure yet about part 4, so if you have ideas of subjects you would find interesting, feel free to let me know.

Posted in agile, complex systems, large projects, project management, software development, systems, tools | Tagged , | 11 Comments

Een acuut hartinfarct – hoe is dat?

Op 28 september 2015 om kwart voor twaalf ‘s nachts kreeg ik plotseling pijn. We waren net naar bed gegaan en ik wilde net gaan slapen. Terwijl ik op mijn zij was gaan liggen om lekker te gaan slapen kwam de pijn opzetten. Maagzuur, of de slokdarm, dacht ik. De pijn werd snel sterker, midden op mijn borstbeen. Stomme maag! Even gaan liggen, gaat zo wel weer over… niks, werd alleen erger. Mijn vrouw belde toch maar even de huisartsenpost om te overleggen. Allerlei vragen of tinteling in de vingers, uitstraling naar armen of de kaak, duizeligheid, desorientatie… niks. De pijn werd alleen maar heftiger, en ik kon alleen nog ineengekrompen op de rand van mijn bed zitten. Laat die pijn stoppen!! Dan ineens: waarschijnlijk een hartinfarct, ambulance komt er aan. Nee, hartinfarct, ik toch niet!? Ik heb nooit wat!
Na enkele minuten komt de ambulance. Hulpverleners komen binnen, maken vlot een hartfilmpje en ja hoor: infarct, links. Ik krijg 4 pillen. Op sokken en slipper, in pyjama met korte mouwen en korte broek, lopend van de trap, door de koude nacht naar de ambulance. Pufje nitro, muntsmaak. Rustig naar het ziekenhuis, geen blauw want het is vlak bij en de opgeroepen dienstdoende arts is er nog niet.
In het ziekenhuis naar de cardio ruimte; 20 a 30 minuten geschat. Vervelend prikje in de pols, ader verwijden wat een beetje prikt, catheter er in, wat schraperig gevoel in mijn arm. Rechts ok, links zit dicht. Rontgen buis gaat van links naar rechts, terug, boven, onder, vlak langs mijn neus. Lampje aan, uit. Allerlei groottes van stents hoor ik noemen; nee, doe toch maar die andere. Dit doet even wat meer pijn, en ja hoor. Mijn benen en armen trillen; sorry hoor, ontspannen lukt me echt niet Meer. Het is zo gebeurd, geef hem maar even wat… nee, doe maar wat meer. Het helpt, maar een klein beetje. Om 3 uur ‘s nachts is het klaar. Moe, uitgeput, maar klaarhelder. Ik ben naar een kamer gebracht. Daar lig je dan. De pijn is gelukkig weg. En nu? Mijn vrouw wordt er bij gehaald. Ze heeft ook midden in de nacht, van half 1 tot 3 uur in spanning zitten wachten. Half uur, heel uur, ja nu zijn ze bijna klaar… en dan nog een uur! Helemaal hyper door de adrenaline. Het gaat goed, de pijn is weg. Je kunt gerust naar huis. Ik zie je morgen weer. Dag. Kus.
Daar lig je dan. Sokken nog aan, in je pyjama, onder een dekentje. Klaarwakker! Af en toe komt er een witte jas langs. Proberen te slapen en geen slaap. Ik merk dat ik gewoon kan bewegen, zonder pijn. Heerlijk! Alleen het infuus en al de draden zijn hinderlijk. Linker zij, recher zij, op de rug, allemaal geen probleem met wat wurmen. Half 4… vier uur…kwart over vier… half zes… toch even weggedommeld. Tegen een uur of 8 wordt mijn bed naar een andere ruimte gebracht. Moe en helder. Zal wel een zware dag worden. Maar de pijn in mijn borstbeen is terug. Minder erg als voorheen, maar toch. Belletje voor de zuster. Ja, dat komt voor. Wordt wel minder. Maar het werd niet minder; het werd meer. Niet zo erg als gisteravond, maar toch.
Pas laat in de middag spreek ik de cardioloog. Hij vertelt dat ze een 2 stents hebben geprobeerd te zetten op een splitsing van aderen in het hart. Dat lukte niet, dus hebben ze besloten de stent op de splitsing te zetten waardoor een van de aftakkingen werd afgesloten (en afsterft). Afsluiten van die aftakking is eigenlijk hetzelfde als een afsluiting door een infarct, vandaar de pijn op de borst. Hadden ze dat wat eerder verteld dan had ik me niet zo’n zorgen hoeven maken. Na een ruime dag trok de pijn ook weg.
De volgende dag komt de fysiotherapeut langs voor een inschatting van wat ik nodig zal hebben aan revalidatie. Ik voel me prima, en kan volgende week wel weer aan de slag, denk ik. Nou, dat is nog niet zo zeker; ga maar even mee naar de trap. Uit bed, door de gang langs de batterij hartmonitoren, naar de trap. Dat gaat prima, zie je wel. Een paar treedjes op de trap… 4 treden, buiten adem. Sh*t! Dat had ik niet verwacht. Toch maar revalidatie afspreken dan.

De weken er na bleef ik me prima voelen. Wel een “golden 5” aan medicijnen (bloedverdunner, betablokker, bloeddrukverlager, cholesterolverlager, stollingsremmer). Een maand niet autorijden. Balen natuurlijk. Na die maand naar een reanimatie cursus die al gepland stond. Iedereen nieuwsgierig natuurlijk, en ik doe goed mee, maar mijn condities (om te reanimeren) is wel bagger. Maar goed, wel gemotiveerd om dit te leren natuurlijk. Burgerhulpverlener word ik toch maar even niet in deze situatie, maar ik heb er wel ernstig over gedacht.
Na drie maanden weer part-time aan het werk. Iedereen bezorgd, en ook maar even rustig aan gedaan. Trappenlopen gaat nu weer prima, beter dan voor het infarct toen ik nog geen 24 treden kon doen zonder buiten adem te raken. Nog een ruime maand later weer full-time aan het werk. Weer helemaal de oude, maar dan fitter…

Posted in life | 2 Comments

Implementing SAFe with Jira (part 2)

This is part 2 about implementing SAFe with Jira. In part 1 I explained how we implemented the portfolio, program and team backlogs as the 3-layered structure of SAFe in Jira. You may recall that we renamed them to business, system and product levels. In part 2 we will dig more deeply in connecting the layers.

Structure plugin

Since Jira (core) is not able to visualize linked relationships, we use the Structure plugin from ALMworks. On a structure board, you can add Jira issues, you can sort them and put them in a hierarchy (i.e. an issue nested under its parent issue), manually by drag-and-drop or automatically by synchronizers. A synchronizer interprets the data in Jira (e.g. a filter, Epic link or issue links) and puts the issues in the structure at the right place. Synchronizers work bi-directional. For example, by adding or removing a link the synchronizer places the issue on the right place in the structure, and by changing the place of an issue in the structure, the synchronizers adds or removes links in Jira.

So if we have a business epic -> system epic -> product epic -> product story -> subtask relationship (through Epic links and Issue links of the “Implement” link type), the Structure board visualizes the hierarchy structure as nested issues. By adding the columns on the Structure board (through views), you can see, check and even change the value of a Jira issue. For example, when a release on team level is later than the release on program level, you can easily see this by looking at the FixVersion/s column.

One of the features of the structure plugin is its time tracking capabilities. The plugin is able to aggregate the planned and actual effort (original estimate, remaining estimate, logged work) over items in a structure. For example, if an epic contains 4 stories each estimated at 3 days work, and the epic itself requires an extra 1 day work, the estimate of the epic is 13 days work (4 * 3 days + 1 day).

There is a lot more to say about using the structure plugin and its synchronizers. I may come to that later.

Agile Release Trains (ARTs), Program Increments and Sprints

An Agile Release Train (ART) is planned and organized in iterations called Program Increments (PIs). We define 2 PIs per year, corresponding with the commercial windows of product marketing to the market. In fact, we don’t even use the term “Program Increment” and call them “windows”. Each PI (window) consists of 3 releases, and each release consists of 4 sprints. In Jira, PIs (windows) and releases are defined in the FixVersion field and sprints are defined in the Sprint field.

  • Business backlog items are planned in windows: W1/2015, W2/2015, W1/2016, …
  • System backlog items are planned in releases: 1.1, 1.2, 1.3, 1.4, …
  • Product backlog items are planned in releases (1.1, 1.2, …) and assigned to sprints

As you can see development and releasing are both on cadence. SAFe says “develop in cadence, release on demand”, but since we come from a traditional stage-gating world, releasing in cadence is a old habit. We may change that in the future.

Portfolio/business level and program/system level have no sprints; they use Kanban, which represents a continuous flow of input (new items) and output (closed items). On team/product level we do use sprints with Scrum. Typically, sprints are 2 week time-boxes. The name or numbering of each sprint is dependent on the team; sprint 6 for one team may not coincide with sprint 6 of another team. But by using separate projects in Jira for portfolio, system and for each product team, each project may have their own set of version values and sprints in Jira. This requires some alignment between system and product level, because we want the release numbers to be the same for all projects, at least for now.

Of course, our world is not ideal. We have teams with a different iteration lengths and we even have teams that do not even follow Scrum, e.g. hardware development. That is okay as long as they are able to deliver for and integrate with the release.

The Structure board shows the hierarchical relationships from portfolio to system to product level. The fixversion column shows the windows and releases and the sprint column shows the sprints, so we can spot inconsistencies easily. The estimated effort column shows the aggregated effort and the story points column shows the aggregated story points. This helps us decide about the release content. And the progress bar shows the aggregated progress of a portfolio item and all its descendants. That way, we can visually check the consistency and progress.

Agile release train (ART) planning

Before a release starts, we have an ART planning session, which we call the “release train workshop”. The portfolio backlog in Jira containing the  business epics on the strategic roadmap for the coming years, specifies the business epics for the coming window. Since we have 4 releases per window, we don’t need to do everything in a single release. But the question is: how much can we do in this release?

The first step is to divide the portfolio items (epics in the Portfolio project in Jira) into multiple system items (epics in the System project in Jira). Typical system epics are a minimum viable product (MVP) of the business epic and successive increments.

After putting them in Jira, they need to be estimated: how much capacity does it require from each of the teams for each of the system epics? In Jira, you only have one estimate field per epic, so that is insufficient to capture the individual estimates of each team. So instead of trying to extend the system epics with extra custom fields, we make epics in the Jira projects for the product teams and link the product epics to the system epic with an “implement” issue link. Through the structure plugin, the product epics show up underneath the system epics and the estimate column (or story point column) shows the aggregated effort (or size) of the product epics for the system epic.

Unfortunately, Jira (or better, the structure plugin) does not allow us to extract the aggregated data to reports or even to queries.

Having these estimates brings us to the next question: will it fit the release? Accumulating the estimates of the system epics for the release may give the impression that the total capacity of the teams is able to cope with the total amount of work for the release. But fortunately, it is not a realistic approach either. If one team is overloaded, the release cannot be completed in time anyway. So a better metric would be to look at the total estimate for each team for the release. And since the estimates for a team are all in the same Jira project, it is easy to create a report that shows the totals.

Knowing the total estimate per team allows us to decide whether the release content fits. Of course, we are running into the situation that it does not fit! In those cases, the MVP definitions may even be reduced further and the estimates need to be redone. So in practice, the information is not actually entered in Jira until there is sufficient consensus about the feasibility of the release content and the estimates. Jira is not a planning tool and does not allow the dynamics of what-if scenarios; I wish.

Conclusion

In part 1 I described how we have implemented the 3-layered backlog structure in Jira. In this part 2 I have made the first step of making use of this structure to deal with release content and estimation. Part 3 will take us to the next step. If you have ideas what next step you would like me to focus on, feel free to let me know.

Posted in agile, complex systems, large projects, project management, systems, tools | Tagged , , | 11 Comments

Implementing SAFe with Jira (part 1)

In this article I am not going to explain what the Scale Agile Framework (SAFe) is and what Jira is. I assume you know, and if you need more information go to http://www.scaledagileframework.com and http://www.atlassian.com.

But before we start, there are a few things that are important to know:

  • Jira is not designed for agile development. In essence it is an issue tracking tool. You need plugins to added support for Scrum/Kanban. Currently, there is no plugin for SAFe support.
  • SAFe is not a prescriptive process model. It is a framework. You can implement it anyway you like, leaving out what does not work for you and adding thing that you need.

My experience with SAFe and Jira is based on the implementation of SAFe (with Jira) with a customer I worked with in a development and innovation organization of a large, multinational technology company for commercial products for consumer and professional markets. It concerned development of software (embedded and applications), hardware (controlled by the embedded software) and other disciplines (e.g. mechanics, optics). In this article, I will try to guide your through our thought process so that you might understand why we made our choice.

Implementing the 3-layered model of SAFe

SAFe (v3.0) defines a model with 3 layers:

  1. Portfolio layer – for managing the business epics
  2. Program layer – for managing the release trains
  3. Team layer – for managing and executing the development activities

For us this was confusing. Teams exist at all levels, so we renamed “team” level to “product” level. This may suggest that teams directly deliver commercial products to the market like an ordinary Scrum team, which some teams actually do. However, all teams deliver to system integration to build integrated systems for the agile release trains (ARTs).  Therefore, we renamed “program” level to “system” level. And finally, we call “portfolio” level “business” level, although we also use the term “portfolio”.

So we have the 3 layers:

  1. Business level – for managing the business epics and architectural epics
  2. System level – for managing the release trains
  3. Product level – for managing and executing the development activities

On each level we have a backlog as SAFe defines: portfolio backlog, system backlog and product backlog. Each backlog has backlog items. SAFe defines 3 types of backlog items:

  1. Epics – on portfolio level
  2. Features – on program level
  3. Stories – on team level

In later parts of the article, I will extend these backlog items types for defect management.

To get agile support in Jira, you need the Agile plugin. This plugin support Scrum and Kanban boards, and several reporting extensions such as burndown charts. Scrum defines a “product backlog” which nicely corresponds with the product backlog we defined at team/product level. The Scrum board (in plan mode) has epics and stories (and potentially other issue types). Stories (and other issue types on the Scrum board) can be linked to the epics, and these links nicely show up on the Scrum board as colored links.

However:

  • Jira Scrum boards do not support features according to SAFe
  • Epics on the Jira Scrum boards are not corresponding with business epics according to SAFe

The development teams were already used to epics and stories on team level in their product backlogs according to the Scrum approach with Jira. According to SAFe, multiple teams (on team level) may be working on the same epics and features (on program and portfolio level). So we had to find a solution for implementing business epics and features. We saw a number of alternative solutions:

  1. Define extra issue types on portfolio level (e.g. Themes), put epics on program level and stories on team level (see Sander Brienen [2013]),
  2. Define a project for the business epics with a Kanban board on portfolio level, and projects for the Scrum teams on team level (see Agile plugin documentation).
  3. Define a project with a Kanban board on portfolio level, a project with a Kanban board at program/system level and projects for Scrum teams on team/product level (see Agile plugin documentation).

We chose for solution #3 because it allows us to use the same approach supported by the Agile plugin (e.g. the Scrum and Kanban boards) on all levels. Every level has epics and stories (and other issue types) linked to epics, every level has a backlog reflected on a Scrum or Kanban board, every level has the standard reporting capabilities, etcetera. Users don’t need to learn new ways to work with Jira when they use the backlogs of other layers or teams.

Given our translation of portfolio/program/team levels into business/system/product levels, this comprised into:

  1. Business project – contains business and architectural epics (and stories) with a portfolio backlog workflow and a Kanban board
  2. System project – containing system epics (and stories and other issue types, e.g. defects) with a system backlog workflow and a Kanban board
  3. Product projects (short for Product development projects) – containing product epics and stories (and other issue types, e.g. defects) with a product backlog workflow and Scrum boards

Implementing relationships between backlogs

Now, within a Jira project we can use Scrum or Kanban boards to view and manage the relationship between epics and stories (and other issue type). For example, on the Scrum board in plan mode you can drop a story (or an issue of another issue type) on an epic to set the Epic link. As a result the Scrum board shows a colored link on the story, and you can do all kind of nice reports as supported by the Agile plugin.

However, we need to relate the product epics (on the product backlog)  with the system epics, and the system epics (on the system/program backlog) with the business epics or architectural epics (on the portfolio backlog). We do this with an “implement” link, which is a bidirectional link (“is implemented by” / “implements”). (See also Sander Brienen [2013]).

So we use 2 link types:

  1. Epic link – for agile relationships within a Scrum team
  2. Issue link of link type “implement” – for backlog relationships across SAFe layers

The Epic link relationships can be viewed and managed through the Scrum boards provided by the Agile plugin. But Scrum boards cannot visualize or manage issue links. For that you need to open the issue and view/manage the issue links in the issue itself. For example, when you open an epic you see a block with “Issue Links”  and a block with “Issues in Epic”. But to visualize and potentially manage the relationship of product stories linked to a product epic, linked to a system epic, linked to a business epic, you need the Structure plugin. The Structure plugin is a powerful plugin, but also a nasty beast. I will tell you about it another time.

Conclusion

Having backlogs and backlog relationships in Jira in place is only the beginning of the implementation of SAFe with Jira, but it is an important foundation. One of the challenges is of course how to work with the information and with Jira to make the release trains and the development teams effective. There is a lot more to tell, but I’ll have to wait until part 2.

Posted in agile, complex systems, large projects, project management, software development, systems, tools | Tagged , , | 5 Comments

Nieuwe vleugel bij het CKE – Fazioli

Gisteren moest ik op de muziekschool zijn om een vioolleerling te begeleiden op piano. Het was de laatste repetitie voor het examen volgende week. Terwijl we wat stonden te kletsen kwam haar vioollerares naar me toe. Ze had de sleutel van de concertzaal bij zich, met daaraan een grote sleutel. Ok, dacht ik, we spelen op de vleugel. Hier, zei ze veelbetekenend, een verrassing, en ze overhandigde de sleutels aan me. Nog altijd niks vermoedend ging ik voorop naar de concertzaal.
Daar, midden op het podium, stond een vleugel met een gekapitoneerde afdekhoes er over. Fazioli stond er op. Hmmm een nieuwe, dacht ik, maar waar is de Steinway? Onder de afdekhoes, zwart met een prachtig rode binnenzijde, verscheen een schitterende zwarte vleugel van concertformaat. Ik denk wel ruim twee en halve meter lang.

Ik ben er achter gaan zitten. Mooie toetsen, niet wit maar een licht romige kleur. De eerste voorzichtige tonen klinken mooi. Maar ja, van een nieuwe vleugel verwacht je dat ook. Maar het leek wel meer dan mooi. Dan maar een paar maten van de begeleiding. Mooi! Heel mooi! Fijne aanslag. Dan maar een stukje Reflet dans l’eau van Debussy. Als boter zo zacht kwam de warme klank uit het instrument. De noten gleden uit mijn vingers als vanzelf, zo gemakkelijk, zo egaal. Moeiteloos dansten mijn vingers over de toetsen, die precies stroef genoeg waren naar mijn smaak. Geen enkele oneffenheid in de aanslag, een perfecte klank, mooie romige klank en nog veel mooiere bassen die zwaar grommend en mysterieus aanzwelden. Dit is een hemels instrument. Alles gaat zo makkelijk, zo licht maar niet te licht, volkomen egaal en perfect. In combinatie met de op dat moment vrijwel lege concertzaal was het alsof ik in een droom verzeild was geraakt.

En de Steinway? Die was ik al lang vergeten. Die mag dan de reputatie hebben om de beste ter wereld te zijn, ik heb die altijd als veel te direct en onvergevingsgezind gevonden. Geen instrument voor begeleidingen maar voor solopartijen. Deze Fazioli is daar perfect voor. Past precies bij mijn smaak en mijn spel. En ja, de repetitie was werkelijk een genot om te doen op “mijn” Fazioli. Mag ik ‘m echt houden, vroeg ik nog, maar dat zat er helaas niet in.

Posted in music | Tagged | Leave a comment

How to double your money?

In this article I am not talking about doubling your revenue or profits, I am going to focussing on doubling your costs. Why? Because nothing is for free: if you want to earn more, you need to pay more. And because you have direct control over your costs, not over your earnings. Like in darts or archery: you have control only until you release the arrow.

Suppose you are managing a small racing team. You are spending 1 million on the car and 5 million on the driver. If you double the money on the car to make a better car you will win more races, and if you double the money on hiring a better driver you will win more races too. Which one would you choose? I would choose doubling the money on the car. It will cost you 1 million extra, not 5 million.
Now suppose you are running a production factory. You are spending 1 million on machines and robots and 5 million on workers. If you can increase your production, where would you double your money? Again, I would double the amount of money on machines and robots. In fact, most car manufacturers are spending over 90% on equipment.

Well it is not always that obvious. System and software development is a knowledge industry. People are the most valuable assets, but also the most expensive. It is no exception when 10% of the costs are spent on tools, such as licenses, servers and support and maintenance, while 90% is going to the engineers, architects, testers and project managers. No development manager will approve doubling the costs on tooling. It is considered as “overhead” costs and overhead should be minimized, not increased. But he is willing to spend even more money on better training, process improvement initiatives, metrics programs and new methodologies. Workers in projects are allowed to spend a considerable amount of their time on implementation and deployment of better processes and tools, time they do not spend on actually developing the system or software.

Why is that? Why is a factory manager willing to spend over 90% on equipment to maximize productivity of the factory, while a development manager is not willing to go beyond 10%. If people cost 10 times more than tools, an increase of 10% on the people would be equivalent of an increase of 100% (doubling the money) on tools. Which one would ultimately be more effective for the organization?
I argue that if you spend 90% of your development costs on tools, tool integration and automation, and 10% on the best possible teams that you can buy, where the people can fully focus on the intellectual work that cannot be done by tools, the organization would be far more effective in speed and delivering quality than any organization with the reverse ratio.

Of course, you should not double the costs of tools and equipment blindly. But in system and software development there seems to be a taboo on doubling the “overhead” costs even if that would be more effective than spending even more money on improving the capabilities of people.

Posted in integration, people, software development, systems | Tagged | Leave a comment

Would you spend 1 FTE more to save 2 FTE?

Suppose you have a software or system development organization of 400 people, working in 4 development programs with each 5 projects and 20 people per project, on average. You are using a heterogeneous tooling landscape of different tools from different vendors to support the projects, such as a requirements management tool (e.g. Doors), an architecture modeling & analysis tool (e.g. Enterprise Architect), integrated IDE editing-build-test tools (e.g. Eclipse, Visual Studio), a test management tool (e.g. HP ALM), source control tool (e.g. Git, SVN, RTC), defect control tool (e.g. Jira, Bugzilla, Trac), Scrum support tool (e.g. Jira Agile, Rally, RTC, TFS).

Of course, the tooling infrastructure did not drop out of thin air. It is the result of an evolutionary growth, where also practices evolved over time. Some tools are used in isolation but at a certain moment the need for integration of tools emerges. Let’s go through some of those integrations:

  • Defect control – Source control
    We need this integration to understand which source code was changed to solve a particular defect. Oppositely, it helps understand why a piece of code was modified.
  • Test management – Defect Control
    We need this integration to understand which defect was submitted because a test run failed, or more accurately because a particular step of a test case in a test run fails. Oppositely, we need to understand which test cases to re-execute when the defect is solved.
  • Requirement management – Test management
    When a test case passed, we need to know which requirements are OK. And oppositely, if a test case fails, we need to know which requirements are not OK. In addition, if we want to release our product, we need to understand how much of the requirements are OK – the essential requirements must be OK 100% and less essential requirements may be less than 100% OK. So we need to integrate requirements with tests
  • Agile Planning – Requirement management
    Most projects have limited capacity to develop the product or systems. We need to plan which requirements will be delivered in the next releases. That requires integration between requirements and plans.
  • Agile Monitoring – Test management
    A burn-down chart visualizes the amount of stories that are ready over the course of a sprint. A story is ready when it passes the acceptance criteria which typically involves testing. We need the integration between the stories counted as ready in the burn-down chart and the status of the test. Typically, this is a 3-way integration from the agile planning/tracking to requirements to tests.
  • Agile Planning – Defect Control
    To understand that you can finish the delivery of a release, you need to understand the amount of open work (e.g. stories) and the amount of open rework (defects).
  • Source control  – Requirement management & Architecture modeling
    To implement a piece of code, you need to understand what it is expected to do. In traditional organizations you would have a specification document; in modern organizations the specifications are stored as requirements and architecture models in the corresponding tools. Oppositely, if you want to change the requirements or the architecture, you need to understand which source code is affected.
  • Reporting – any tool
    Many tools have limited reporting capabilities and when there are reporting capabilities it reports are often only accessible within the tool itself or as a PDF or Excel file. Integration with a reporting or data warehouse tool is often a neglected integration.

Now suppose you want to implement those tool integrations. Some integrations are supported by a plugin, adapter, integrator or synchronizer of the vendor of the tools. By the way, if you have tool A from vendor AA and tool B from vendor BB, it is not always obvious which vendor provides the integrator and how far that vendor is willing to support integration issues with the other tool (free of charge as part of the license agreement). So it takes effort to find the integrator, to install it, to configure it, to test it, to troubleshoot it and to solve problems of the integration. If there is no vendor with an out-of-the-box integrator, you might make your own scripts or application to integrate the tools. That takes effort to learn the APIs of the tools, to discuss and define the integration needs, to implement it, to test it, to troubleshoot it and to solve problems.

And then the vendors come with a newer version of the tools, that you need because bugs are solved and extra functionality is needed in the projects. When upgrading a tool, you may need a new version of the integrators, you need to reinstall it, to reconfigure it, to retest it, to troubleshoot it and to solve problems. And if the integrator belongs to vendor BB while you are upgrading tool A, you might run into issues with vendor BB to support it.
And in case of your own integrations, you need to learn the updated APIs for the new tool version (hopefully, no functions are deprecated that you use), to redefine the integration needs (to take advantage of the new capabilities of the new tool version), to adjust the implementation, to retest it, to troubleshoot it and to solve problems.

And then you want to add a new tool and/or replace an existing tool. You need to reconsider all your integrations with those tools, (re)install or (re)define and (re)implement them, (re)configure them, (re)test them, troubleshoot issues and solve problems.

To make things worse, you need to maintain resources with technical knowledge and expertise with the integrations and keep that knowledge up-to-date.If someone leave, he/she must be replaced. For integrations that you made in-house, you need to maintain that knowledge in-house; for vendor-based integrations you may depend on the vendor or external service provider partners, but still the organization specific knowledge needs to be maintained in-house. For open-source tools it may even be more difficult.

And to make things even more worse, software engineers have a tendency to invent their own tools and integrations in the margins of their work. What starts by retyping information from one tool into another tool, often ends in a “handy” little script that extracts information and import that in another tool. Run it in a crontab on a local computer and your done, until another projects wants the same thing. Before you know, your machine is overloaded with processes that are not yours, but that you need to support anyway in case there are issues.

“How much resources does maintenance and development of tool integrations cost?”

As I started, we assumed an organization of 400 people. What do you think, would you be able to support, develop and maintain all tool integrations (including maintaining technical knowledge of the APIs and the integrations) for about 10 tools for 400 people with 1 person or 1 FTE? And how much time do projects spend in the margins to work on integrations? Would 0.1% be realistic, which is 2 hours/year per person on average? And how much overhead would the projects lose because an integration does not work properly, is missing or down, and synchronizations need to be recovered? Would 0.2% be realistic, which is only 4 hours per year per person? Would that be a realistic representation of reality?
Anyway, in that case the total costs would be 1 + (0.001 * 400) + (0.002 * 400) = 2.2 FTE which is about $250k per year. My estimate would be that this is highly underestimated!

Now if I can do the same amount of tool integrations and more, keep it up-to-date with the latest version of those tools, better tested and of higher quality, for the cost of 1 FTE (about $120k per year), which solution would you prefer? If a tool integration is missing, I could add it for another $50k with no yearly cost increase, it would still be cheaper than the underestimated current reality.

“Impossible? Unrealistic? Are you mad?”

This is a absolutely realistic and possible. The only problem is that I have never met a development manager who is willing to increase the operational overhead costs by $120,000 per year on tool integrations. Why not? Because less effectiveness in projects are hidden costs counted as “normal” development costs and because a tools support team of – say – 4 people is already considered “overhead” costs.

“We ‘d rather hire an extra engineer than spend an extra $120k per year on tool integrations that we already have or don’t need!”

Posted in configuration management, integration, tools | 3 Comments