Law and Software

I am on a “dormant” mail-alias for software migration. Software migration is a polite way of referring to the process of ripping out an old API from your software application, and replacing it with something more modern. Like when you flip a house for resale and give it a new kitchen and a fresh coat of paint. This nightmare process pwns you when your user interface toolkit (like Motif) finally bites the dust. You either throw away your software and start again, or migrate to another toolkit (like Qt)  that is likely heading the same way – but your competitors are all using the “new” one, so it must be ok.

I’ve watched a few requests for help come down the wire recently. And no-one is reading them. Except me. The alias is dormant because the people who did this work are now looking for new jobs. Last week, I decided “enough of this.” I don’t have a dog in this game anymore, and I no longer have ethical issues about responding to messages that were ignored, or answered in a way I wouldn’t. So I replied.

The email was what I would have written in the old days when I worked in this field. And there’s nothing proprietary in it – I end the message with:

This is much what I used to say to customers, since it’s not about proprietary techniques – just about planning, and being able to make many many code changes automatically without introducing the slightest human error. I use these same techniques now in the legal area – I recently prepared a statute for the web and there were over 500k edits required. The trick isn’t to automate, it’s to do things systematically in a way that you can review systematically as well.

Here is the full text of the email.  Unless software migration is your current life nightmare, it won’t mean all that much to you. But if you are reading out of curiosity, the audience is usually someone who has a valuable software asset that is maybe 10 or 15 years old, and has invested a huge amount in maintaining that software, making it very robust and error-free. Throwing it away and starting again is a ridiculous idea, but it is starting to look like the right choice because the alternative seems impossible. Reprinting it here means that I will just be able to point others to this blog post.

Most of it is true for any API you are locked into and need to “migrate away from”

***

Hi ****

I don’t get to talk about software migration much these days – this email should be fun.

What I found interesting in migration work was that – even though all applications are completely different in what they do – they are all “the same” when it comes to a migration.

You actually have two issues. And every successful migration I did was because customers were able to separate these two issues out. Occasionally people were not able to, because of management pressures in their own company – and I would (sadly) decline their work.

First issue is migrating away from Motif. Second is migrating “to” the UI of your choice. It may be Qt. Back five years ago it was the platform of choice. For that second issue, the real question is where the market will be in 5 years time. Migrations are intense activities. In 5 years you want to be in mid-cycle with a stable product and not facing a new migration because the toolkit chosen is tired or EOLed.

Key to success is the first step – “migrating away from motif” – I would begin each migration with an audit of the software – I actually did one recently for as a favor to an old customer/friend and it isn’t hard. I’ll try and explain. I used to have standard presentations for this – hopefully I’ll be clear here.

The biggest risk to any migration is in errors that slip in during the process. Even if your engineers are so careful that they only introduce 1 error in 1000 changes that they make, you *will* end up with a significant number of high cost bugs to fix at then end of the road when the application is almost converted. The goal of “migrating away” is to quarantine all of the UI work – so that all of the application code you have maintained and made robust over the years is completely safe. No errors because you don’t touch it.

Your UI will be somewhere between 8 and 20% of your application. The low figure is when it is all generated from a GUI builder. However, in almost all applications, there isn’t a clean separation between UI and application. Callbacks rarely call into the application with completely abstract and portable interfaces. They use the Motif and X11 data structures. Rendering routines similarly can be a real mix of app code and UI. When you do an audit you will get an idea of how far through your application your UI has spread. Typical is that 60% of all files have some UI code in them – even if it is a single header file and one function that uses a UI data structure.

This issue is completely separate from the actual conversion, and it can be done before any final decision is made. You do this abstraction/separation work as a tidy-up exercise, and end up with a very healthy piece of software where the UI is as quarantined as possible. And you do it now because you can test each block of changes you make. You can tidy 20 files, recompile and run through your test suite to catch the occasional slip that would cost a fortune to find in the migrated app when you just don’t know which error of combination of circumstances caused it.

Very often people would be retargeting their app to a new platform at the same time – compounding all the ways this can go wrong. For example they would be going from solaris to windows. And there, you have a two stage “clean up” – the UI housekeeping followed by platform work – where I would get them to move away from the heavily customized Solaris OS to one of the simpler vanilla linux distros (or to freebsd if they were heading towards the mac). But they would do the clean up in two stages or it would get out of hand.

Everything in a successful migration is about pushing all the “risk” to the beginning, where you still have a working test rig and can do the work as “just more maintenance” and before you break loose and can no longer do regression tests.

Audit sounds complicated but it’s very easy for Motif. Any file that is going to have UI code has to have a header file. so you can search all your files for include lines that contain X11/ Xt/ and Xm/ and get the list of “infected” files. Then you can search for the various function and data structure prefixes to get a sense of how many calls to the API or uses of a datastructure are done in any one of these files.

60% of files sounds a lot – but there is a long tail. I would expect 25-35% of your files to have substantial direct interaction with the GUI and then a “long tail” of files that may have only a single line that uses the toolkits.

I can’t emphasize strongly enough how important this audit will be for you. It starts you down the road of quantifying quite accurately how much this is all going to cost. Once you have it, you decide whether you want to separate GUI and application as if the GUI was a thin client – where there is going to be a data abstraction overhead – or whether you define “interfaces” to allow the same code objects to wear two “hats”

That’s a long email. But it should help you think around how to prepare for your migration. Whether or not to use Qt is a whole different discussion – and you don’t have to make that decision before the software “hygene” check. Only once ever did someone give me a source distribution that was perfectly quarantined. And if the result of your audit is that only 12% of all your files have any UI code in them, that should really encourage you. And if not, that is just normal.

One last word about “automated tools” – many things can be automated safely – converting menus, buttons, labels etc. And that makes the tools look good. There are two big issues – one is form layout, the other is rendering. Motif uses the “form” with its attachments. Most all modern UI’s use some form of tabular or grid layout. The conversion from relative attachments to weighted grids is non-trivial. Tools will do simple conversions – but anything complicated will generate nasty code, and it would be wiser to plan for someone to relayout by hand (i.e using a design tool – not by hand) your dialog layouts, once you have done the automatic stuff.

Rendering in X11 can happen anywhere – because of the “display server” client/server model – everything gets sent down the wire to the display, and it is asynchronous. That’s what you give up when you migrate, so you have to make sure all your rendering happens at discrete times in “paint routines” – that may require some serious code restructuring.

I used to think this stuff was hard. Then I went to law school 🙂

I can’t think of much else to say at the moment. This is much what I used to say to customers, since it’s not about proprietary techniques – just about planning, and being able to make many many code changes automatically without introducing the slightest human error. I use these same techniques now in the legal area – I recently prepared a statute for the web and there were over 500k edits required. The trick isn’t to automate, it’s to do things systematically in a way that you can review systematically as well.

best