Does the Internet Work?

Does the Internet work?  Is it a success?

There are a few ways to answer these questions.  It might seem obvious that it is a success, since the whole world is connected or about to be connected.  Global domination is the very definition of success, isn't it?  We could try other definitions, such as happiness.  Does the Internet make us happier?  This would be hard to answer in a blog post.

Another approach is to use the original goals of the architects of the Internet as a benchmark.  In 1987, David Clark wrote a retrospective paper describing these goals.  Clark was the Chief Protocol Architect for the Internet in 1981, and he remains an Internet visionary today.

Goals:

0. Develop an effective technique for multiplexed utilization of existing interconnected networks.
1. Internet communication must continue despite loss of networks or gateways.
2. The Internet must support multiple types of communications service.
3. The Internet architecture must accommodate a variety of networks.
4. The Internet architecture must permit distributed management of its resources.
5. The Internet architecture must be cost effective.
6. The Internet architecture must permit attachment with a low level of effort.
7. The resources used in the Internet architecture must be accountable.

Some reflections:

1. Though it's hard to remember today, the Internet was originally a military program.  It was designed to withstand nuclear attack.  This goal (1 and 4), coupled with the desire to interconnect existing networks and networking protocols (0, 2, 3, and 5), led to the design choices of 1) a packet-switched network of dumb routers with 2) the datagram as its base messaging unit. 
   
   A packet-switched network transmitting datagrams meets many of the goals of the Internet architects, but it has some serious trade-offs.  Because the Internet is stateless, error detection, retransmissions, and security must be handled at the end hosts.  The network just does the best it can to transmit datagrams.  There are no other perks.
   
   In many ways, this was a pragmatic decision: there was a large installed based of existing hardware, and it would have been difficult to convince people to start again with specs designed by the military.  The architects had no idea how widespread the Internet would become.  Still, it was a military application.  You would think that security, at least, would need to be baked into the network.
   
   Interestingly, decentralization led to many social, political, commercial, and economic consequences that typically are not high on the list of military priorities.  I have never worked in defense, but I understand that there is an emphasis on chains of command and accountability.  I can't think of anything less hierarchical or accountable than the Internet.
   
   
2. Speed is not a goal.  Isn't it funny that a communications protocol using telephony cannot transmit real-time voice communication?  The packet-switched network was designed to route around network failures.  There is no guarantee that packets arrive in any order or within any period of time.  For this reason, packets have a lot of overhead in the form of message headers.  The packet-switched datagram is a relatively slow form of communication.  It wasn't until broadband was widely available that VOIP became feasible. 
   
   Much traffic today is composed of message headers, retransmissions, and flow control information, all of which could be better handled by a stateful network.  A stateful network would be more like the phone system, in which dedicated lines handle streams of information.  These lines would have to be easily reroutable to handle interruptions to the system (such as nuclear attack), but packet-switching is not the only way to do this.
   
   
3. Accountability (i.e., monitoring) dropped off the list.  Since it was a military application, the Internet Architects wanted to be able to account for its uses and abuses.  However, it would have been very difficult to develop monitoring systems in a stateless network.  After all, the monitoring system AT&T uses for billing is about as complex as the telephonic network itself, and that is a stateful system.  For this reason, accountability was quickly dropped from the list of goals.
   
   Since there is no monitoring within the network itself, all accounting must be handled at the application layer.  TCP has to figure out what speed it should transmit by monitoring dropped packets.  Denial-of-service attacks have to be handled by firewalls.  Routers can simply declare what addresses they can handle.  And, of course, speed can't be a goal if there is no way to measure performance.

In short, the Internet is not a perfect system.  It meets many of its goals, but its architects faced a trade-off between stateful and stateless systems.  Stateful systems are good for speed, error-handling, security, and accountability, but a stateful Internet would require more expensive hardware and a reworking of the entire system.  A stateless system is cheap and simple, but it is slow and requires hosts to handle any advanced features.  At the end of his paper, Clark suggests that the packet-switched datagram was a compromise and that a 'soft state' system composed of streams may have been better.

Like the Internet architects, however, we have to be pragmatic.  The ubiquity of Network Address Translators (NATs) forces us to stick with TCP/IP.  Because NATs read and rewrite message headers, there has to be agreement on the protocols in use.  If we wanted to change protocols, we would have to throw out all our routers and modems.  Of course, new protocols could be created on top of TCP/UDP/ICMP, but these would have the same shortcomings as the layers below.

How the Internet Works

I never really had a proper introduction to networking until taking an online course.  I recommend the course, even for people with little experience in technology.  It's really not rocket science.  A network engineer, and former plumber, I know claims it's exactly like plumbing--"It's all pipes," he says.

Network engineers may be a bit like plumbers, but the architects of the Internet were cultivators of complex systems.  They needed to find a balance between being too rigid and thus hampering future innovation on the one hand, and being too open and thus causing Tower-of-Babel-like chaos.  These architects used at least two key conceptual tools to manage complexity: encapsulation and layering.  Software developers should be familiar with both terms.  For example, object-oriented programmers use encapsulation to abstract away the details of application code from their use, and they make use of layering to decouple functional domains.

This is all a fancy way of saying that the development of complex systems requires black boxes.  It's impossible for any one person to understand everything that is going on in a reasonably complex system, so we must be able to say, "Well, we're not going to worry about that right now."  Furthermore, if a change in one place requires a change in ten other places, you have a very fragile system.

The Internet has four layers.  In the highest layer, applications talk to each other without worrying about any of the messy details.  For example, your browser just requests a web page--it doesn't care how it gets there.  The transport protocol (probably TCP) establishes connections and ensures that data is reliably delivered in sequence.  It also manages congestion.  TCP doesn't care how the data segments are transmitted.  This is the job of the network layer, most likely IP.  IP is a simple delivery service that does its best to get a message from one location to another.  Routers direct traffic to the appropriate address based on routing tables, but there is no overall plan for the path packets should take.  Finally, the link or physical layer is composed of the actual cables or wireless hardware.  It relays bits, but it has no idea what the bits mean.

What's amazing about the Internet is that it hasn't changed much in forty years.  We've never had to start over from scratch.  We didn't have to stand by while the Masters of the Internet upgraded to version 2.0.  In fact, it works so well that we usually don't have to think about it.  Changes to the Internet, such as ARPAnet's upgrade to TCP Tahoe in 1986, happened pretty seamlessly.  The current switch to IPv6 is transparent.

The main reason for the Internet's resiliency is its use of layering and encapsulation.  Because each layer functions independently, its implementation is encapsulated or hidden from the layers above.  For example, there has been a proposal to implement IP via carrier pigeons.  The application and transport layer would still work the same, just a bit slower.

Layering and encapsulation are made possible by using a message-in-a-message technique, much like Russian stacking dolls.  Network traffic is comprised of messages.  TCP calls them segments, IP calls them datagrams, and Ethernet calls them frames, but they're all composed of a payload and a message header containing instructions for a particular protocol.  The payload may contain another message consisting of another header and payload, and so on, down to the physical layer.

For example, when you go to a website, you use the HTTP protocol (application layer).  Each HTTP response has a header and a message.  A response header returns a GET header with the web page as the message.  This response is broken up into the payloads of messages with TCP headers (transport layer).  These message headers have instructions ensuring safe delivery.  The TCP messages are the payloads of yet other messages, having headers with IP destination instructions.  Finally, these messages are the payload of yet more messages having headers containing ports and MAC addresses.

The message-in-a-message technique is simple yet powerful.  For example, VPN works by recursively layering messages.  You might have an HTTP request in a TCP segment in an IP packet in a secured TLS presentation message in a TCP segment in an IP packet in an Ethernet frame.  Requests over VPN are encapsulated within normal Internet traffic.

IPv6 works in a similar manner.  We've already started to run out of all 4.29 billion IPv4 addresses (the familiar 192.168.1.1) and we have begun using IPv6's 3.4×10³⁸ addresses (the not-yet-familiar 2a91:0db8:85a3:0000:0000:8a2e:0370:7334).  We don't need two Internets to support both network protocols thanks to a technique called tunneling.  Network Address Translators can transmit IPv6 datagrams over IPv4 routers by simply putting IPv6 datagrams inside IPv4 ones.  That is, IPv6 tunnels through existing IPv4 networks.

We don't know what the Internet will look like 100 years from now, but it will probably have new layers stacked on top of or recursively contained within existing layers.  It will evolve piecemeal, without major upgrades causing blackouts.  It may swap out TCP for some other, more efficient communications protocol, or it may replace the physical layer with faster wireless relays.  Because of layering and encapsulation, changes to any one functional layer will have no impact on any others.

The Physical and the Virtual

lol

The Internet is awesome--that's why everyone wants to be there. It's got cheap stuff, lolcats, all kinds of games you can play with your friends, news about even the most mundane things, and every conceivable way of 'talking' to other people without having to deal with them face to face. Did I mention that it's global and instantaneous too? Sold!

The Internet has shaped our lives so much that it's easy to become fed up with the real world. Why do I have to sit in traffic when I could telecommute? Why should I go to the store when I can buy online and have it the next day? Why should I learn anything when I can look it up on my phone? Why should I have to slog away at my job when kids are making millions selling apps? Why should I put up with a girlfriend when I can... well...

In short, why should I do anything that doesn't give me immediate satisfaction?

The shit flies when our speed-of-light connection is lost. I've been a cord cutter for a few years now. I watch less television, I don't pay the big cable companies quite so much, and I've learned to deal with the lag between air time and online availability. But, every so often, the cable goes out and I suddenly have no idea how to function. "What are we supposed to do?" I say.

Don't cut that line or we'll lose New Zealand!

It's times like these I remember that the virtual world can't be separated from the physical world. Even as a developer with a (albeit limited) knowledge of networking, it's so easy to forget the routers, cables, protocols, and standards that make it all work. We're abstracted from all that.

The story of the physical Internet is fascinating, and Andy Blum tells it well in his new book. Reading it, I remembered all the things I should have known already--like that the Internet is a network of networks, so there have to be places where these networks connect to each other. (Duh.) These stuffy closets and unmarked office buildings that Blum describes are where the Internet is, if it can be said to be anywhere. There's a whole other world that underlies our virtual 'reality.'

A subway map shows how to get between points above ground, not what happens below

During the summers of my youth I ran cables for the school district, among less glamorous tasks. It was a shock when I first popped up the drop ceiling at my high school and realized that there was a huge space between the tiled 'ceiling' and the floor above. This in-between space is full of conduits, plumbing, cables, HVAC systems, and meters that make our comfortable lives possible.

Being abstracted from the guts of the system is good. It lets us deal with the things we care about, like learning, while ignoring other things, like trying to keep at a comfortable temperature. Supermarkets abstract us from the food chain so we can focus on cooking. Virtualized desktops and servers abstract us from physical hardware so we can focus on computing. High level languages abstract us from their implementation so we can focus on coding.

But abstraction has a cost. When the AC breaks, the tomato crop is ruined by disease, the network is down, or the compiler does something really stupid, we're hosed. We can't learn, cook, compute, or program.

I don't think there's some big moral to this story. We obviously can't get rid of HVAC, supermarkets, or OO languages. (Right?) But maybe it's a good idea to understand these physical worlds underlying our virtual ones... before they break.

The Reef and the Market

Philosophy professors like asking students where their ideas come from. "I just think them," students are bound to retort. "Aha!" The professor pounces. "But where does the idea of 'I' come from?"  Silence.

Much of philosophy involves trying to explain where various ideas come from. Socrates and Aristotle understood the advance of thought as a process of dialogue which builds upon the ideas of the past. René Descartes argued that the idea of God was at the root of all true ideas. Karl Marx thought many of our ideas, such as religion, are ideology and a product of power relations. W.E.B. DuBois believed that we understand ourselves and others through the lens of race and that these ideas have a contingent history. William James thought our ideas were a product of 'what worked' for us and people like us in the past. Thomas Kuhn argued that scientific theories belong to a history of evolving paradigms.

Stephen Johnson is concerned with good ideas in his book, Where Do Good Ideas Come From? There are three images of innovation that orient his inquiry. Coral reefs, which make up 0.1% of the Earth's surface but have support 25% of all marine species. There is the city, which, as Geoffrey West has shown, increases in innovation in relation to population at a super-linear rate. And there is the web, which has decreased the time required for innovating and adopting new technologies from 20 years to 2.

By looking at innovation from a number of different scales, including at the level of brains, cities, and, ecosystems, Johnson comes up with a framework I summarize in the following way.  Good ideas are fostered by:

• networks that can change
• that have some stability
• that favor chance encounters
• that embrace error
• that support re-use
• that support building on other good ideas

More creativity per capita than any suburb

How can businesses foster innovation? Johnson shows that the majority of major inventions in the last two hundred years did not happen in R&D labs at major firms or in the garages of people who later struck it rich. They usually took place in colleges and universities, or organizations like CERN. This was surprising to me, given my experience in the academy, its silos of rival departments, and its distance from the real world. However, universities do allow people from very different backgrounds to work together and to circulate and build upon others' ideas freely. They allow for experiments to go wrong and let people research controversial things.

Many businesses today taut the importance of innovation, but few allow for failure, the open exchange of diverse ideas, change, or time for reflection. A recent survey of CEO's showed that they spend around 50 hours a week working but have little time to reflect, given the constant interruptions of BlackBerries. Google, on the other hand, requires employees to work on their own projects 20% of the time. Twitter built an open API and then built their services on top of that. Apple, while opaque to outsiders, has a very messy development process where everyone at each step of the development chain is involved with a new product at the very beginning.

Twitter or GM?

More personally, Johnson's book caused me to reflect on when I'm most creative. I'm best in my sleep or in writing after having a discussion with someone. I need time to let ideas simmer, but I'm lucky enough to have lots of smart people to discuss ideas with. Johnson concludes:

"Go for a walk; cultivate hunches; write everything down, but keep your folders messy; embrace serendipity; make generative mistakes; take on multiple hobbies; frequent coffeehouses and other liquid networks; follow the links; let others build or your ideas; borrow, recycle, reinvent.  Build a tangled bank."

Do Technologies Have Politics?

I've been exploring the idea that technology isn't just a collection of things which can be used interchangeably for either good or ill. Technologies shape the world and the range of choices we have. For example, in my last post, I suggested that real-time technologies may have effects for the ways we do finance. Our question today is: if technologies can shape financial decisions, can they shape politics as well?

Riot!

It's one thing to say that technologies are used for political ends and another to say that technology is political. The latter is Langdon Winner's claim. If politics is about power and authority, then at least some technologies are political if they embody, define, or exert the power of some people over others.

Technologies are obviously political when they are used as tools of control. It is often feared that new computer technologies will allow governments to spy on people in order to control illegal activities. But forms of control can take much subtler forms. For example, when Robert Moses designed many public works in New York City in the mid 20th century, he designed them so that people could not get to them via public transportation. The overpasses he constructed have a clearance of only 9 feet--far too low for buses and those who would ride them to get to Jones Beach. Similarly, many people don't realize that the grand boulevards of Paris were designed by Baron Haussmann in the mid 19th century as a form of riot control.  As a Penn State alum, I can tell you that narrow streets are much more conducive to riots than open fields.

These are examples of technologies being used to empower some and dis-empower others. But are some technologies inherently political? I think so. Marx and Engels called for factory workers to take over control of 'the means of production,' but Engels later argued that the very technology of industrial manufacturing requires a division between workers and elites. How can you run a factory without a boss?

Winner endorses solar power over nuclear energy because it can be decentralized and doesn't require the scientific and bureaucratic elites who make decisions without the knowledge of anyone else, as in the case of the Fukushima Daiichi disaster. For months, the Japanese government and TEPCO lied to people about the seriousness of the accident. Countless polls show that support for nuclear power is at an all-time low across the world, with Germany abandoning future plans for construction. Nuclear fallout is an issue, of course, but I think people are really worried about the forms of power that are inseperable from nuclear energy.

Visit Japan, future skate park capital of the world.

Technologies that shapes public opinion and collective decision-making are also inherently political. Before the invention of the printing press, it would have been impossible to have democracy on anything other than a very small scale. Though it required a set of elites who broadcast information to information consumers, and though, as Walter Lippmann worried, most people would never get the whole truth, the dynamic duo of newspapers and democracy were better than any alternative. Pamphlets made the American Revolution possible.

I, for one, welcome our new search engine overlords.

Today, internet technologies are already reshaping power. The decline of major 1-directional or broadcast media, including newspapers, is heralded by some as the dawn of a new, more democratic age. Like much breathless optimism about technology, such claims should be taken with a grain of salt. Most internet traffic is routed through about five sites, a problem Michael Hindman terms Googlearchy. Most influential political bloggers are white males who went to ivy league schools. But there are many precedents for thinking that new technologies can undermine and recreate power in radical ways. These will be fought by the old guard and embraced by the new.

It comes back again to: what kind of technology do we want?

Technology and Collective Problem-Solving

"Technology" signifies all the intelligent techniques by which the energies of nature and man are directed and used in satisfaction of human needs; it cannot be limited to a few outer and comparatively mechanical forms.
--John Dewey

In a previous post, I explained how many philosophers, including Heidegger and Marcuse, see a rift between ethical reflection and technology. They worry that the means-ends thinking at the heart of technology can cause us to ignore other kinds of reflection--especially about who we want to be, what we hold to be just, and how we can lead more meaningful lives.

There is obviously a difference between painting a picture and developing a manufacturing plant to make paints and brushes, but what's wrong with solving problems? Is it really so dangerous as philosophers--who aren't typically known for being technologists--seem to think? John Dewey says no, arguing that all inquiry has a technological component insofar as it is meant to solve problems. If moral inquiry helps us solve problems, it's as technological as lasers and airplanes are. Theories are just tools for solving problems.

Understanding technology as problem-solving may seem impossibly vague, but it's actually very powerful. Whenever considering a new gadget, theory, or way of doing things, Dewey suggests we ask: what is the problem this is meant to solve? Remarkably, many new products don't seem aimed at solving any problems, or at least not any serious ones.

What about the problem of collective decision-making? Humans have created two lasting technologies for this purpose: representative governments and markets. Governments are good at ensuring certain behaviors that its people think should be ensured. They define and enforce justice, including the means of determining what justice is. This wasn't always the case and took many years of trial and error. Life used to be filled with a lot more anxiety, because the world was so much more unpredictable, and the means of determining fairness were uncertain.

Althingi, where Icelanders have solved problems since 930 CE
Governments, however, can only solve certain problems. They're bad at picking market winners, for example, and they're slow to react to change. They are good at prohibiting certain behaviors, but it's hard for them to make citizens moral, healthy, intelligent, or cultured. As Cass Sunstein argues in his book Nudge, the best governments may be able to do is incentivize certain behaviors so that people will make the right choices on their own.

Markets, on the other hand, provide a highly responsive way of determining what people value and what should be produced. As Friedrich Hayek recognized, markets aggregate people's individual choices and values and thus collectivize intelligence in a very efficient manner. Markets will always have the input of more people than governments as well as higher levels of participation. And, since people often know what they want better than 'experts,' markets can be more rational than governments.

Unfortunately, many things cannot be quantified in dollar values, such as the environment, health, or justice. We can adjust markets so that they take hidden costs into account, as cap-and-trade systems do, but these work best when you have a metric that can be easily tied to cost. Another criticism of markets is that people do not always act rationally, as Daniel Kahneman and other behavioral economists have shown. Even if we know what we want, we can't be sure to act accordingly.

Given the limitations of governments and markets, Deweyans turn to small groups for salvation. There are many interesting examples of small-scale collective problems solving, such as the rebirth of Pittsburgh or river management in Mexico, but it's hard to see how such solutions will scale. As our interactions become ever more global, we need globalized methods of collective decision making.


For these reasons, Clay Shirky and other technologists point to the internet as a possible third way of making intelligent choices collectively. It's not enough to say that the internet connects people. The idea of the internet as a 'Global Village' has become a joke, as new technologies help us filter each other out like never before. What Shirky points to is the way the internet lowers barriers to participation. Shirky's poster child is Wikipedia, which, like most internet phenomena, displays a long tail of participation. Many people work together, though the vast majority only contribute a little.

Lowering barriers is great, but it is probably not enough if we are to find a third way to compete with governments and markets. Can new technologies help us better solve collective problems? The question becomes ever more pressing as big players like Google, Microsoft, and Facebook become ever bigger and structure the ways we interact more and more. Not being evil is not the same thing as providing venues for increasing collective intelligence. What other problems should we be trying to solve?