Assumptions and Unpleasant Surprises

You and I are having some kind of discussion about something we'll be doing. I make my points; you make yours. Eventually, we reach some kind of understanding, and we walk away and get to work. Later on, we realize that something was missing, or didn't turn out in the way we thought it would. What happened, and more importantly, what do we do about this kind of dynamics?

I think what happens is that in all discussions there are lots of assumptions, some of them explicit, some of them implicit; some of them shared, others not. Then, those assumptions have ramifications, some of them known, some of them unknown (at the time of the discussion.) The net result is that, at the end, we were not as much in the same page as we thought we were.

Eventually, and especially when the ramifications result in unpleasant surprises, there are almost immediate calls to "think things through" so "it doesn't happen again." True enough. However, no matter how much discussion, investigation and specification you do, the problem doesn't seem to go away and keeps recurring.

I've realized that this is a kind of "fractal" problem: you start with a situation that you don't like, so you apply some kind of strategy to solve it, and the situation indeed improves, but you end up more or less with the same problem again, just at a different scale. On top of that, the law of diminishing returns eventually kicks in and the cost of more discussion, investigation and specification goes beyond practical levels, some times very quickly.

I think now that focusing on making assumptions more explicit and "thinking through" their ramifications, as worthy a goal as it is, is the wrong answer. Yes, it has to be done to the best of our abilities and as far as it is reasonable and practical, but putting mechanisms in place to make the surprises less unpleasant, or actually more bearable, is a better strategy in the long run.

In the context of Agile development there are several practices that I think help with this: encouraging more fluid communication helps from the assumption side, iterative development helps on the surprise side. Each iteration is an opportunity for the team to evaluate the state of their work, discover surprises early and deal with them.

C++ inline classes considered harmful for TDD/Refactoring

One of the things that make practices like Test Driven Development and Refactoring so powerful is their focus on flow and rapid feedback: small changes done at very frequent, short intervals allow for rapid verification of assumptions and lots of mid-course corrections.

This is something important to keep in mind, because in the heat of the coding battle other decisions can be made in order to address other issues that at the same time (maybe inadvertently) hinder the effectiveness of TDD and refactoring to keep the flow moving. One example of this is the decision to implement C++ classes inline, completely in header files.

Compiled languages like C++ impose a natural roadblock to rapid feedback: the compile and link cycle. Each time we make a change and we want to see it in action, we must go through this cycle, which unavoidably creates an interruption in flow while we wait for the compiler and linker to do their job. The longer it takes to compile and link, the longer the lag time between making a change and observing its effects.

If you're doing refactoring and TDD properly, you'll find yourself doing a sequence of small, little changes. After each one, and depending on the nature of the change and the stage into the refactoring, you're supposed to either just compile the code, or compile and run all the tests. C++ inline classes negatively affect this process in two ways:

If you need to run your tests, that means compiling and linking all your code, so if the code you're changing is in a header file that is included directly or indirectly in many other compilation units, then each of those small changes can trigger a massive recompilation, even if the change affects only function implementation details.

In other cases, after a refactoring step all you need to do is just to compile the class you're changing. Usually this happens when all you're doing is to lean on the compiler to make sure things are still wired up in the correct way, so all you need is to compile a file or two. If the class you're changing resides only in a header file, there is no single compilation unit (.cpp file) that you can use for that. You'll need to find a .cpp file that includes it, but that will most likely include other files, which will make the compile time longer.

It may not even be possible to just compile that class, though. Imagine that you're refactoring a class hierarchy. You've made changes in the base class, and you want to compile only that before moving on to the subclasses. But then you find that there's no single .cpp file that includes the base class' definition; there are other files that include it, but they also include subclasses, which still haven't been refactored and still do not compile. The end result is that you need to make changes all across the class hierarchy before you can compile anything.

The most obvious effect of all this is overall productivity (more time is spent waiting for the compiler) and interruptions in flow (which may affect focus and concentration.) But there's also other problem: because we hate waiting for the compiler, we'll try to avoid it, creating the tendency to make changes in bigger chunks, run the tests less often, or skip some tests instead of running them all, all the time.

For all these reasons, I think the benefits of TDD and refactoring when it comes to flow and shortening of feed back loops clearly outweighs the "overhead" of separating class declaration from definitions.

As Scott Meyers points out in Effective C++, Item #33, "use inlining judiciously." What's your take?

Splitting Stories and Independence

Brian Marick has a good example on how to break a big story in smaller chunks. There, he uses an incremental approach, adding small pieces of functionality to a web page.

My first thought was that this violates the "I" part of Mike Cohn's INVEST principle, since the resulting stories are not "Independent": they represent a sequence of enhancements, each one building on top of the previous one, so they are meant to be selected and built in a very specific order. You can't have Story 3 without building Stories 1 and 2.

I've always had the feeling that there is a minimum size for a story under which the "Independent" principle just can't be applied; that as we split stories to make them implementable in a couple of days (and in that way, more "estimatable" and manageable) we risk hitting a level of granularity in which functional dependencies are too visible, making the stories not "swappable" any more.

The way Brian does the split, though, is very business-value driven. Moreover, he admonishes us of creating "technical stories", doing the split along purely technical lines.

This made me realize that the "independent" requirement can have other interpretation: allow the user to go as far as he/she wants. Even though with this set of stories the user looses his/her ability to pick and choose which pieces of functionality to develop, there's always the option to "go only that far" and stop adding features at any time.

Seen from a higher perspective, maybe what happens is that we start with big, "fat" stories that are really "independent" (in the sense that they can be added or dropped from the release plan as a whole) and as the splitting progresses, we cross that granularity level in which the resulting stories form an implementation sequence. As long as those sequences contain "incrementally independent" steps (and providing that all the different "sequences" are really "independent" from each other) we are OK.

Bookshelved: "The Pragmatic Programmer"

Last night, I finished reading "The Pragmatic Programmer: from Journeyman to Master", by Andrew Hunt and David Thomas, the last book in the 2006/2007 season of the Toronto Design Patterns Study Group.

This is a book about attitude:

"We feel that there is no point in developing software unless you care about doing it well."

That's how the book opens, right on the preface. From that point forward, you know that you're reading a very opinionated piece.

Of course, the statement above can be interpreted as a truism, just stating the obvious. The meaning of "doing things well", however, is not always that obvious, so to dispel doubts, the authors use the rest of the book to explain what "well" means for them: a mixture of pragmatism, technical excellence, professional responsibility and common sense.

As it is stated at the beginning, this is "a book about doing." However, don't expect to find concrete, step by step instructions on how to do anything. This is a book about philosophy; about the mindset that it's appropriate for approaching software development problems, ranging from programming and design issues, all the way to teams, projects and individual professional practices.

That being said, the book includes whole sections devoted to general software design principles, like the importance of keeping things that are not related from affecting each other ("orthogonality"), the effects of coupling and the techniques to deal with it ("It's just a view", "Law of Demeter"), design by contract and reversibility, etc.

But what I enjoyed the most were the sections dedicated to personal practices. The book relies heavily on the metaphor of the craftsman, journeymen and masters, tools and workbenches, to convey the idea that software development is an art that needs to be cultivated and grown in time. Along the book, you'll find tips like "Don't live with broken windows", "Be a catalyst for change", "Don't repeat yourself", "Don't program by coincidence" and "Sign your work" which all sound like the words of wisdom of the master talking to the apprentice.

In all, it was a thought-provoking reading; the type of book that you need to come back to from time to time. Very recommended.

It's all about lists

After the initial collection and organization, the next step in implementing GTD is separating all your actionable items in lists, each of them grouping items by some common characteristic.

At the bare minimum, you need three lists:

• the "Next Actions" list, which contains your actionable items;
• the "Someday/Maybe" list, containing things you have explicitly decided not to do now;
• the "Projects" list, containing the projects you're working on (with "project" defined as anything that takes more than one next action to complete.) Essentially, this list contains entries that feed the "Next Actions" list with new actionable items.
  

However, usually more lists will be required to better organize the things you need to do. GTD's recommendation is to use "contexts" to generate lists, grouping tasks by the place, tools or activity that are required to execute the task. Examples of contexts are HOME, OFFICE, PHONE, COMPUTER, ERRANDS, etc.

I decided to keep two sets of lists: one for things to be done at the office, kept in Outlook, and other for things to be done somewhere else, kept in my PDA. The decision had mostly to do with portability of the work: when I'm not in the office, I don't need to be reminded of those items that are strictly work-related.

To organize the office lists, I created several task folders in Outlook, under my Inbox. I found this particular type of folder useful because you can create task items manually, by quickly entering some text in an edit box, or from e-mail messages or documents in your Inbox, by dragging-and-dropping them to the folder. Then, each item's status can be flagged as "In Progress", "Completed" or "Not Started", and the folder's view grouped by this field, giving you an idea of the level of overall progress in the list.

For consistency, I created a similar folder structure in my personal Yahoo! Mail inbox, but for my personal tasks (or any other activity that is to be done outside the office) I decided to use mainly my PDA's TODO application, using task categories to form the lists:

• @HOME, for tasks to do around the house;
• @COMPUTER, for personal tasks that require a computer;
• @CALLS, for phone calls, e-mails to send, etc.
• @ERRANDS, for tasks that require going somewhere;
• @WAITING FOR, for things I've delegated to others and that should come back;
  
• @LATER, for things I've decided to postpone

There's also @INBOX, which is a task list I use to create new items that need to go through the organization process later.

Interestingly enough, there's no project list here, because I keep them all in my personal wiki, where there's a wiki page per project. Each page has a brief statement of the expected outcome of the project, the list of items or ideas I've come up with so far, and any reference material that can be kept in electronic form (like Internet links, for example.)

This has been my first week trying this out, and I'm still streamlining the system, but so far the results are very positive. Looking at my "next actions" list is still overwhelming, since there are lots of items still there, but at the same time it's reassuring to know where everything is. Let's see how this goes.....

Bootstrapping GTD, session 2

After the initial collection exercise, the next step in implementing GTD is to go over the pile of collected "stuff" to decide whether is something that requires further action, needs to be archived for future reference or trashed altogether.

If the item is actionable, then the rule is that if it can be done in 2 minutes or less, it should be dealt with immediately. Otherwise, it goes to a new pile: the pending items stack:





For the items to be archived, GTD suggests using a file cabinet, in which store this material in folders (properly labelled, with the labeller, of course!) in strict alphabetical order.

Some people think that in this day and age everything can be stored digitally, or found somewhere in the web, so keeping a file cabinet is not required. Actually, and after taking an inventory, I found that I keep a variety of reference information digitally: bookmarks in del.icio.us, calendar and contacts in Yahoo!, pictures in Flickr, and notes and e-documents in a personal wiki at my home machine. Still, some paper based reference material is worth keeping, so I needed to get some sort of cabinet to store them. After some looking around, I found this cart (at Ikea, where else? :-) which I can store away in a closet:





The good thing about this second day of GTD prep work is that, at the end of the day, you have done a dozen small things from your pile, stored lots of material out of sight, and trashed a good other amount of stuff which you won't have to worry about any more.

It was enough work for a Sunday, and time to go see the Canada Day fireworks.

Getting ready to GTD

I have decided to give the "Getting Things Done" personal organization method a try.

The first step in this method is to collect all your "stuff", for which the book suggests blocking out a full day (6~8 hours) to go over all your cabinets, drawers, shelves, notes and papers, computer files, etc. collecting items that you think "are not in the way they should be" and dumping them in your "in basket." Then, after the "physical sweep", comes the "mind sweep": collect anything that might be still residing in you "psychic RAM."

The end result should be a big pile of "stuff", which in my case looks like this:





As part of the collection process, I gathered over 100 items, and yes, it took that many hours as the book recommends (including lunch break and a brief nap; it's Saturday after all!)

The next step in the process is organizing all this pile into lists of actionable items, deferred items and archived items. But that's going to be tomorrow.