The Clouds

I am walking in the air, and speculating about the sun.
-- Socrates, in Aristophanes' The Clouds

Self-explanatory.

What's wrong with having your head in the clouds?  For starters, you might fall into a well.  That's what happened to the ancient Greek philosopher Thales while he was gazing at the heavens.  Aristophanes ridiculed Socrates in a satire called The Clouds for speculating about abstract nonsense without understanding anything about the here-and-now.

Slightly more recently, 'The Cloud' has arrived to save us from all earthly ills.  Got a problem scaling?  Move to the cloud.  Want to reduce cost?  Go to the cloud.  Disaster recovery?  Cloud.  High availability?  Cloud.  Yet, the cloud can't really be a cure-all.  It may be appropriate for your needs, but this has to be evaluated given an understanding of how your cloud implementation will work.  If your number one problem is performance or security, the cloud is probably not going to help.

There are many 'clouds' or high-level cure-alls available to developers today.  For example, I've been learning the Spring Framework, which is a Dependency Injection (DI) framework.  DI is a form of Inversion of Control (IoC), in which object coupling is not known at compile-time but rather is determined at compile-time.  Using a DI framework, you can build interfaces, inherit classes from them, and control application flow using configuration files and annotations (like @inject).  During development, you can stick to the pure work of thought--programming, that is--without having to worry so much about the nuts-and-bolts of connecting classes.

DI frameworks like Spring are just one more layer of abstraction (or one more cloud) that let us focus on business logic rather than implementation details.  And, just like other layers of abstraction, they're a double-edged sword.  Dependency injection isn't always a great idea, such as if your interfaces are likely to change often.  Dependency information has to be stored somewhere, after all.  Enterprise frameworks tend to give you flexibility for the price of code bloat and complexity.  In my time, I've seen programming become ever more complex, but, at the same time, ever more like plumbing.  Frameworks like Spring, Rails, and .Net's Entity Framework let you quickly build applications.  But they don't prevent you from doing very stupid things if you don't know how they really work.  ORM libraries are great, for example, until you have to scale.

For this reason, I've been brushing up on my data structures and algorithms.  It's easy to think such things don't matter.  Plenty of people get by without earning computer science degrees.  And who cares if a client-side algorithm is O(N^2) when the database is orders of magnitude slower?  You might concede that algorithmic theory is worth studying only to develop certain intuitions about processing time, memory size, problem size, and algorithmic complexity, which is certainly true.  But I think it's useful to revisit data structures and algorithms once in a while to stay grounded in reality.  If you're only doing high-level plumbing, you have your head in the clouds. 

Does the Internet Work?

Does the Internet work?  Is it a success?

There are a few ways to answer these questions.  It might seem obvious that it is a success, since the whole world is connected or about to be connected.  Global domination is the very definition of success, isn't it?  We could try other definitions, such as happiness.  Does the Internet make us happier?  This would be hard to answer in a blog post.

Another approach is to use the original goals of the architects of the Internet as a benchmark.  In 1987, David Clark wrote a retrospective paper describing these goals.  Clark was the Chief Protocol Architect for the Internet in 1981, and he remains an Internet visionary today.

Goals:

0. Develop an effective technique for multiplexed utilization of existing interconnected networks.
1. Internet communication must continue despite loss of networks or gateways.
2. The Internet must support multiple types of communications service.
3. The Internet architecture must accommodate a variety of networks.
4. The Internet architecture must permit distributed management of its resources.
5. The Internet architecture must be cost effective.
6. The Internet architecture must permit attachment with a low level of effort.
7. The resources used in the Internet architecture must be accountable.

Some reflections:

1. Though it's hard to remember today, the Internet was originally a military program.  It was designed to withstand nuclear attack.  This goal (1 and 4), coupled with the desire to interconnect existing networks and networking protocols (0, 2, 3, and 5), led to the design choices of 1) a packet-switched network of dumb routers with 2) the datagram as its base messaging unit. 
   
   A packet-switched network transmitting datagrams meets many of the goals of the Internet architects, but it has some serious trade-offs.  Because the Internet is stateless, error detection, retransmissions, and security must be handled at the end hosts.  The network just does the best it can to transmit datagrams.  There are no other perks.
   
   In many ways, this was a pragmatic decision: there was a large installed based of existing hardware, and it would have been difficult to convince people to start again with specs designed by the military.  The architects had no idea how widespread the Internet would become.  Still, it was a military application.  You would think that security, at least, would need to be baked into the network.
   
   Interestingly, decentralization led to many social, political, commercial, and economic consequences that typically are not high on the list of military priorities.  I have never worked in defense, but I understand that there is an emphasis on chains of command and accountability.  I can't think of anything less hierarchical or accountable than the Internet.
   
   
2. Speed is not a goal.  Isn't it funny that a communications protocol using telephony cannot transmit real-time voice communication?  The packet-switched network was designed to route around network failures.  There is no guarantee that packets arrive in any order or within any period of time.  For this reason, packets have a lot of overhead in the form of message headers.  The packet-switched datagram is a relatively slow form of communication.  It wasn't until broadband was widely available that VOIP became feasible. 
   
   Much traffic today is composed of message headers, retransmissions, and flow control information, all of which could be better handled by a stateful network.  A stateful network would be more like the phone system, in which dedicated lines handle streams of information.  These lines would have to be easily reroutable to handle interruptions to the system (such as nuclear attack), but packet-switching is not the only way to do this.
   
   
3. Accountability (i.e., monitoring) dropped off the list.  Since it was a military application, the Internet Architects wanted to be able to account for its uses and abuses.  However, it would have been very difficult to develop monitoring systems in a stateless network.  After all, the monitoring system AT&T uses for billing is about as complex as the telephonic network itself, and that is a stateful system.  For this reason, accountability was quickly dropped from the list of goals.
   
   Since there is no monitoring within the network itself, all accounting must be handled at the application layer.  TCP has to figure out what speed it should transmit by monitoring dropped packets.  Denial-of-service attacks have to be handled by firewalls.  Routers can simply declare what addresses they can handle.  And, of course, speed can't be a goal if there is no way to measure performance.

In short, the Internet is not a perfect system.  It meets many of its goals, but its architects faced a trade-off between stateful and stateless systems.  Stateful systems are good for speed, error-handling, security, and accountability, but a stateful Internet would require more expensive hardware and a reworking of the entire system.  A stateless system is cheap and simple, but it is slow and requires hosts to handle any advanced features.  At the end of his paper, Clark suggests that the packet-switched datagram was a compromise and that a 'soft state' system composed of streams may have been better.

Like the Internet architects, however, we have to be pragmatic.  The ubiquity of Network Address Translators (NATs) forces us to stick with TCP/IP.  Because NATs read and rewrite message headers, there has to be agreement on the protocols in use.  If we wanted to change protocols, we would have to throw out all our routers and modems.  Of course, new protocols could be created on top of TCP/UDP/ICMP, but these would have the same shortcomings as the layers below.

Where's the Logic Go?

Typically, the business logic resides in the middle, or domain, layer

Every application has at least two components: the design of technology platforms, called the application logic, and the processes that need to happen, called the business logic. In theory, the business logic is dependent of the application logic, since a business has rules, workflows, and transactions that have nothing to do with any programming languages or database systems. In practice, however, application logic can put constraints on business 'illogic.'

One of the key design choices in developing any application is deciding where the business logic should go. Database developers think it should go in the database, since keeping the code at the database level is often most performant. The problem with this is that SQL doesn't have many of the basic niceties of any Object-Oriented language. Furthermore, since stored procedures use proprietary SQL, they can prevent the migration of database code to another vendor.

OO developers think business logic should reside in the domain layer, since objects are best at representing the real world. Libraries and IDE's like Visual Studio make it very easy to get an OO application off the ground, and they help with maintainability. For many applications, however, the amount of code necessary to create an MVC model, for example, is not necessary and may even be prohibitively burdensome.

In reality, no application design should be used for all problems. Martin Fowler provides four models that couple domain and database access logic.

Transaction Script / Row Data Gateway - Domain code simply passes requests from the UI to the database. Database access is modeled at the record level.

Table Module / Table Data Gateway - Domain code is organized in objects corresponding to tables in the database. Database access is modeled at the table level.

Domain Model / Active Record - Domain code is organized according to business rules. Database access is modeled by CRUD objects.

Domain Model / Data Mapper - Domain code is organized according to business rules. Database access is modeled by a mapping object layer.

Fowler suggests that your choice of pairings should depend upon the complexity of your business logic. An application used for reporting can simply send requests to a database, but a complex sales order process should probably be mirrored by a domain model and a data mapper. A domain model will have a higher up-front cost, but it may pay off as the complexity of an application increases.

I think this general trade-off makes a lot of sense, and it helps me understand and categorize a number of applications I've seen. But, unless I am mistaken, pretty much any enterprise application is going to require a layer for business logic objects, a layer for data mapping, a layer for data access, a layer for the data itself, and, of course, the presentation layer. If it's possible to reduce the complexity of these layers, do so!

Software Gardening

The Pragmatic Programmer has a lot of nuggets. Here's one:

The most common metaphor for software development is building construction. Using construction as the guiding metaphor implies these steps:

1. An architect draws up blueprints.
2. Contractors dig the foundation, build the superstructure, wire and plumb, and apply finishing touches.
3. The tenants move in and live happily ever after, calling building maintenance to fix any problems.

Well, software doesn't quite work that way. Rather than construction, software is more like gardening--it is more organic than concrete. You plan many things in a garden according to an initial plan and conditions. Some thrive, others are destined to end up as compost. You may move plantings relative to each other to take advantage of the interplay of light and shadow, wind and rain. Overgrown plants get split or pruned, and colors that clash may get moved to more aesthetically pleasing locations. You pull weeds, and you fertilize plantings that are in need of some extra help. You constantly monitor the health of the garden, and make adjustments (to the soil, the plants, they layout) as needed.

I'm trying to think this metaphor through, as, until now, I had not questioned the metaphor of architecture. Intuitively, I have a sense for what Hunt and Thomas mean. After all, these guys started the Agile Alliance, which takes the 'waterfall' or linear model of software development as its main target. The four main principles of the Agile Manifesto are:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

I have done some organic gardening, so I have a little experience to draw upon to understand what Hunt and Thomas are getting at.

The first thing I learned gardening is that you cannot start from scratch. No matter how much pesticide you spray or soil you import, there are soil, drainange, water, sunlight, insect, animal, and other conditions that shape what will grow and what won't. The earth has been here a lot longer than your garden has. In the software world, you don't begin at the beginning either. You have a team's knowledge and relationships, IT infrastructure, available languages and platforms, customer expectations, and the market of existing competitors and third party vendors, to name a few. And, of course, the majority of work as a software developer is not development but refactoring existing code.

The second thing I learned is that things rarely work out the way you planned and that you have to adjust to these events. One year, a groundhog ate almost all our butternut squash. Another year there was a tomato blight. We built an electrified fence to cut our squash losses, and we made sure to not plant tomatoes in the same place next year. In software, you have to do what you can to contain unexpected events. This happens at a macro and a micro level. For instance, on a micro level, code should be robust enough to handle exceptional situations. At a macro level, it should be flexible enough to handle change. For instance, in order to best adapt to increasing demand, your system should designed to scale. If you might have 10 million new customers in a month, you must make sure that your database architecture can adapt to that demand.

The last main thing I learned is that to be a good gardener, you have to be able to learn. No one is born with a green thumb, but some people learn more quickly than others what works and what doesn't. It's a lot easier to start gardening if you know some people who already garden. And no matter how much reading you do, there is always more to learn, because there are always new things you have to deal with. The same is true for software development. Many programmers are 'book-learned' and can't apply their knowledge, or they are narrowly knowledgeable about a handful of languages or platforms. It amazes me when I hear people describe themselves as either back-end and front-end programmers, because a good programmer must be both. If you follow Hunt and Thomas's advice, you'll learn a language a year!

Of course, gardening doesn't sound quite as cool as architecture, but good architecture follows many of the same practices outlined here. For example, modernist architects are accused of ignoring site specificity and simply plopping down skyscrapers in the middle of thriving urban ecosystems. If you don't learn from previous projects, you'll get a bunch of haunted shells like those that make up Dubai's skyline. Buildings are much less malleable than code, but that is all the more reason not to use them as a model.

Links
-An interview with Andy Hunt and David Thomas about software gardening