I Know What You're Thinking

Back in grad school, I had a professor who told me about an interdisciplinary summer retreat he had been invited to years ago.  There were economists, political scientists (like him), sociologists, educational theorists, philosophers, and many other smart people.  Though they talked about a wide range of topics, he found it most interesting that, by the end, he could predict the kinds of questions people were going to ask before they asked them.

Economists would wonder about incentives, transaction costs, and barriers to entry.  Sociologists re-framed things in terms of demographics and norms.  Educational theorists were concerned about developmental impact.  Philosophers and English professors would bring the conversation back to 'the text.'  Political scientists would analyze power relationships and procedural constraints.  And so on.

This story has stuck with me for years, especially as I experienced the same thing myself in a similar summer seminar.  On the one hand, it's no surprise that professionalization teaches you a conceptual system with a specialized vocabulary, and it shows the value of doing so.  On the other hand, it's all a bit depressing, as if what we study and do for a living bends us to the wheel of a necessarily partial worldview.

Since leaving the ivory tower and returning to software full-time, I've noticed the exact same phenomenon.  In a technology firm, people from each team will have a perspective that's hard to shake.  Developers and architects will be concerned with system coherence and stability.  Testers live in a world of exceptional cases, where everyone is trying to divide by zero and servers are falling like flies.  Product analysts think in terms of costs, revenue, and rates of return.  Accountants want to know how they can report on it.  Support teams are concerned with how much manual intervention is required.  And project managers want to know how many person days it will all take.  And so on.

(This is all very rough, but you get the idea.  It takes a lot of work to truly appreciate any of these perspectives.)

It's important to recognize these points of view when working with other teams.  If you give a technical reason why it will take a long time to develop something, non-developers will only hear white noise.  They'll probably just think you don't want to do it.  But if you can talk about it using a set of terms they understand, such as existing products and features or past time estimates, you're much more likely to have a productive conversation.

This may seem like common sense, but I think it's one of the central challenges of getting things done in a complex organization: you have to be an expert in one knowledge domain while simultaneously being able to communicate with other experts in other knowledge domains.  All projects involve some give and take.  The less agile your process, the more important it is to make sure you have good communication from the start.  I've seen many projects fail because no one really understood what anyone else wanted.

Branching and Merging

Nothing endures but change.
--Heraclitus

Heraclitus is usually remembered for saying that you can't step into the same stream twice, because the only thing real is change, not rivers or even mountains. Since most code is in about as much flux as the mighty Mississippi, I've often preferred the metaphor of streams to that of branches when thinking about version control. But really there's no perfect metaphor. What best describes a constantly growing and shrinking, branching and merging code base with information going in two directions and which can be frozen at any point?

I've always thought version control was one of the more interesting aspects of software development. Many people don't have a very good grasp of version control. It's another one of those key things they don't teach in school. But, truth be told, there's always something more to learn. Recently I discovered that there are as many ways to organize the branching structure of a repository as there are repositories. I had become accustomed to tying branches and tags (or streams and snapshots) to releases. A great MSDN article describes some of the more popular models.

First, of course, is the branch per release model. It's the model most used by software companies that have to support older versions of code while developing new ones. If you have some customers on 9.3 and some on 2.4, you have to be able to recreate older environments for debugging purposes. You can try to force all your customers to upgrade at the same time, but, given the different testing requirements they all might have, that's pretty unrealistic.


Most interesting of the new models I learned about was the branch per environment model. This makes a lot of sense if you're supporting in-house application code. It allows you to track what's going on in your different environments at any time, and it helps to keep your main (development) stream uncluttered. It can still get a bit messy when you have a lot of different development projects going on at the same time and with uncertain release dates. If your development is not fairly linear, this model may not be easy to implement.


To deal with multiple asynchronous development projects, you could do a branch per task. It's interesting to note that in this model branches have short lifespans, unlike in the previous two models, where they live in perpetuity. If 1.3 happens to go out before 1.2, you just have to merge 1.3 into the trunk and update the 1.2 code. I could see this being used in a relatively simple in-house shop, but definitely not at a software company.


Another option is having a branch per component. In this case, your core architecture is stable and your components do not bleed together. This might work if you have a waterfall-type SDLC in its initial stages and you want to isolate various web services, for example. If you want to avoid integration hell, however, this might not be the way to go.


Finally, you might have a branch per technology if you support multiple platforms. You would have your core code, which you could merge into all your different phone, gaming, or OS platforms. How well this strategy would work probably depends on the amount of overlap possible in the code of the different platforms.


There is no one best way to organize your repository. These models are just that--models. A combination of two or three might work the best for you. For example, you might have a branch per feature in your development stream, but then have fairly simple streams for your other environments. You could have branches for each technology, and then branches for each component within those branches. The possibilities are endless, but an overly complex system is probably more trouble than it's worth. Too few, and you'll have a difficult time developing. Too many and you'll spend all your time integrating. Find a happy medium between and a palm tree and a hedge labyrinth.

Links:
-The MSDN article

Pragmatic Programming

I'm a big fan of Andy Hunt and David Thomas's Pragmatic Programmer (2000) and the series of books they publish at the Pragmatic Bookshelf. You might think this is because I wrote my dissertation on John Dewey, a famous member of a group of philosophers called the American Pragmatists. To be sure, philosophical pragmatism has a lot in common with what we commonly mean by the word "pragmatic." For instance, the Pragmatists thought traditional philosophical questions were abstracted from the problems they were really meant to solve. Philosophers asked "What is the beautiful?" without working with artists who were trying to make more beautiful art. They asked "What is the good?" apart from the moral dilemmas of actual people. And they tried to define "What is truth?" in abstraction from rival truth claims.

In short, philosophical pragmatism eschews head-in-the-clouds questions and focuses on solving problems. Any reflection that is not grounded in solving a problem risks being disconnected from practical consequences. It might even make us worse at solving problems by narrowing our thinking by theories that are untested by experience.

Though there are dangers to pure theory, I find most technical books (and blogs) to slide too far to the other side of the spectrum between theory and practice. These books are often written in the form: "If you want to do X, do Y." If you want to build an index on a table, here is the command. If you want to inherit one class from another, use this syntax. If you want to create a web service, you must create this kind of a connection. Such books are often so focused on the trees that you never see the forest. They are pragmatic in the sense that they don't give you a bunch of high-falutin' theory that you'll never use, but they aren't really pragmatic in the sense of helping you solve real-world problems.

Why is that? Technical books are often useless because they assume you know what the problem is, and anyone involved in software development for long knows that this is rarely the case. If I know that I need to create an index on a table, I can read a tutorial on creating indexes. But maybe what I really need is to write better queries. Or perhaps I need to normalize my table architecture and make more use of clustered indexes rather than throwing non-clustered indexes at performance problems. I might be using a relational database when I should be using a document store. It could be that my company's code review process needs to be improved so that issues like this are handled by database programmers and not our support staff. Perhaps performance isn't even what I should be worrying about, and the biggest problem facing my company is writing more robust code. Once you start to delve into even the simplest of problems, you can open a Pandora's box of other questions that change your understanding of the matter at hand. The reason I like the pragmatic programming books is that they go beyond the nut-and-bolts approach and provide us with concepts to help us better prioritize and deal with the problems we actually face. In an ideal world, everyone would write perfect code. But in the real world, we are always in the process of improvement, and we need conceptual tools to determine what is most in need of improving.

One of the best recent examples I've seen of the kind of truly pragmatic conceptual tools I'm talking about is Brent Ozar's hierarchy of database needs. We all know that our databases could be more secure, robust, and responsive, but what should we focus on now? Well, if we aren't backing them up, we need to create a maintenance plan ASAP. Then we can worry about security, and so on. The point here is not that Ozar's hierarchy is perfect. As he himself says, the problem with doing backups isn't that we don't know we should be doing backups (I hope); the real problem is that the technology and business groups haven't worked together to prioritize what is most important. And a major cause of such a communication breakdown is the lack of a vocabulary to describe the problem the teams face.

I haven't seen a hierarchy of generalized software development needs, but Hunt and Thomas provide many tools to help you and your team develop your own. The base of the pyramid might be having a version control system, and then a ticketing system could come next. In the middle we'd want to make sure our code is orthogonal, or highly de-coupled. Finally at the top we could turn to documentation. Priorities will vary from place to place and from time to time, but without a language to talk about them, we cannot intelligently decide what to do next.