The Clouds

I am walking in the air, and speculating about the sun.
-- Socrates, in Aristophanes' The Clouds

Self-explanatory.

What's wrong with having your head in the clouds?  For starters, you might fall into a well.  That's what happened to the ancient Greek philosopher Thales while he was gazing at the heavens.  Aristophanes ridiculed Socrates in a satire called The Clouds for speculating about abstract nonsense without understanding anything about the here-and-now.

Slightly more recently, 'The Cloud' has arrived to save us from all earthly ills.  Got a problem scaling?  Move to the cloud.  Want to reduce cost?  Go to the cloud.  Disaster recovery?  Cloud.  High availability?  Cloud.  Yet, the cloud can't really be a cure-all.  It may be appropriate for your needs, but this has to be evaluated given an understanding of how your cloud implementation will work.  If your number one problem is performance or security, the cloud is probably not going to help.

There are many 'clouds' or high-level cure-alls available to developers today.  For example, I've been learning the Spring Framework, which is a Dependency Injection (DI) framework.  DI is a form of Inversion of Control (IoC), in which object coupling is not known at compile-time but rather is determined at compile-time.  Using a DI framework, you can build interfaces, inherit classes from them, and control application flow using configuration files and annotations (like @inject).  During development, you can stick to the pure work of thought--programming, that is--without having to worry so much about the nuts-and-bolts of connecting classes.

DI frameworks like Spring are just one more layer of abstraction (or one more cloud) that let us focus on business logic rather than implementation details.  And, just like other layers of abstraction, they're a double-edged sword.  Dependency injection isn't always a great idea, such as if your interfaces are likely to change often.  Dependency information has to be stored somewhere, after all.  Enterprise frameworks tend to give you flexibility for the price of code bloat and complexity.  In my time, I've seen programming become ever more complex, but, at the same time, ever more like plumbing.  Frameworks like Spring, Rails, and .Net's Entity Framework let you quickly build applications.  But they don't prevent you from doing very stupid things if you don't know how they really work.  ORM libraries are great, for example, until you have to scale.

For this reason, I've been brushing up on my data structures and algorithms.  It's easy to think such things don't matter.  Plenty of people get by without earning computer science degrees.  And who cares if a client-side algorithm is O(N^2) when the database is orders of magnitude slower?  You might concede that algorithmic theory is worth studying only to develop certain intuitions about processing time, memory size, problem size, and algorithmic complexity, which is certainly true.  But I think it's useful to revisit data structures and algorithms once in a while to stay grounded in reality.  If you're only doing high-level plumbing, you have your head in the clouds. 

HBase: A Column-Family Store

The next NoSQL database in my whirlwind tour is Apache HBase, a column-oriented database. Riak was a simple key-value database, which mapped an object it knew nothing about to a bucket/key combination. Its job was just to make sure the data was replicated consistently in a highly fault-tolerant manner. HBase schemas, on the other hand, are made up of tables, which are dictionaries of dictionaries. It gives you more granularity and schema control, but it's still dumb about the values of its key-value pairs.

For SQL developers, HBase terminology will not make much sense at first. The large objects are tables, which are made up of key-value pairs consisting of row keys and column families. The column families are, in turn, collections of key value pairs. A table is not a group of relations, nor are there any guarantees that one row will look much like another.

An HBase table

The value of this system is that you get another layer of granularity over a key-value store like Riak. You have a bunch of stuff (a table of rows), which has a couple broad characteristics (a row of column families). Listen, these NoSQL database systems are not all that different from each other. Which one is right for you will depend upon 1) your other applications, 2) your size needs, 3) your performance needs, and 4) your fault-tolerance needs.

For example, HBase cannot read until one client succeeds in writing to all replicated instances. Riak can be tweaked so that writes are successful when only one node has been written to, so it should be faster (though it's hard to find any data on these things--you'll have to do the POC yourself). HBase is best when you have a lot of data and need to run MapReduce. Riak is right if you need 99.999% uptime.

For me personally, HBase was a pain in the ass to set up. I'm not a Linux pro, and HBase isn't really meant to be run in a single instance on a crappy laptop, but this was a real turnoff. MongoDB, Riak, and PostgreSQL were all pieces of cake to set up. So why do it?

One reason is that HBase is part of an ecosystem of tried-and-tested enterprise applications, like the Hadoop Distributed File System and the query language Hive. Its users include Facebook, Twitter, and Yahoo! (Google uses its own BigTable.) There's a large community of users to go to with questions.

Still, HBase has the flavor of many Open Source systems that grew out of a matter of necessity of scaling fast on commodity hardware. I'll be interested to see what Microsoft does with Hadoop and HBase. Call me a Microsoft fan-boy, but I think their integration of technologies is often very good, and it takes time to work out the kinks in a new platform. The first iPhone was a piece of junk. It takes a few iterations for a technology to mature and solve the problems it's designed to solve.

Links:
-Check out this nice comparison of Cassandra, MongoDB, CouchDB, Redis, Riak, HBase, Membase, and Neo4j

Mapreduce and Key-Value Systems

I've been playing around with Riak (ree-ack), my first foray into NoSQL, and it's been a lot of fun. Riak is an open source implementation of Amazon's DynamoDB. It's a web-ready key-value store that scales easily. And it's a whole different world from RDBMS's. Basho has a great tutorial for getting started.

If you're a database guy like me, you're probably wondering what you can do with a key-value store. Using key-value tables is a SQL anti-pattern. Even if you get performance and scalability (which I'll discuss in an upcoming post), it's hard to see how you could do any interesting queries with a key-value system.

One thing to note is that a file system is basically a key-value store. The key is the file name and the value is the contents of the file. You can do searches on file systems, but you can't really do much querying. If your data needs are more like a file system than complex analytics, a key-value store might be right for you.

I also finally learned what the MapReduce algorithmic framework does. You may have heard of MapReduce (amongst many other new technologies) in connection with Hadoop. MapReduce will probably go down in history as one of Google's greatest contributions to computer science (along with the PageRank algorithm).

MapReduce is inspired by a functional language, LISP.  The idea is simple: take the algorithm to the data. In a highly distributed and scalable system, it's not feasible to move data to a processor in order to aggregate and slice it. You have to do any processing in-place. The way to do this is to map your keys to some kind of broader category and then run a reduce algorithm over the mapped data to pick out only the information you need.

For example, let's say your key-value store contains documents, and you want to count the number of documents beginning with the letter 'X' that contain the word 'tibialoconcupiscent'.  In this case, you would map all documents with a key like 'x*', such as 'xerophagy.pdf'.  Your reduction algorithm would simply return 1 for every case of the word and 0 otherwise. This reduction could be run on each server, a group of servers, and then all servers, returning a single (probably small) number.

This framework may seem limited, and it is. Key-value systems just aren't great for complex querying. One reason SQL has been so popular is that it allows for unlimited combinations of queries. You're limited in the ways you store data, and you have to wait for the system to write the data to disk in order to ensure consistency, but you can read the data in lots of interesting ways. That's why it's called a Structured Query Language.

(Riak does let you do some more interesting things because you can create links between different keys that define any kind of relationship between them. These links are just metadata. You can MapReduce across links, but the basic idea is the same.)

Now, where would I use Riak? There are a good number of production users already. I would think it would be best used in systems that don't require a lot of complex querying or complex data types. For instance, it could store webpages, messages, or other content. Unless you have a huge amount of data or require very fast writes, it's probably not necessary. But if you're looking to grow fast, it might be the right choice.

Cathedrals, Bazaars, and Data

According to Eric Raymond, with enough eyeballs, all bugs are shallow. This is the secret to the success of open source software. Linus Torvalds' primary innovation wasn't so much creating Linux as the development model surrounding it: get something out there, get people using it, and have them be your testers / contributors. That is, make users co-owners, not just consumers. Raymond compares open source software development to a bazaar, as opposed to a building a cathedral.

I've been using Linux and open source software for more than a decade, but I've always been a bit wary of open source databases. Do you really want a fail early/often approach when dealing with people's data? Oracle has made a killing off of Data Fear. After all, can you promise 99.999% uptime with an unproven system? Open source might be fine for startups, but is it fit for enterprise?

PostgreSQL seems to merit the hype. Even SQL Server DBAs like it. I don't have experience with it on the enterprise, but I'm impressed so far. I believe that it's a tested solution, and I really like its extensibility. Anyone can create any kind of extension, such as an Amazon Web Services interface or K Nearest Neighbor mapping.

As a .Net/SQL Server developer, however, I have some concerns.

First, is free really free? SQL Server Enterprise is $6,874 per core. Oracle Enterprise is $47,500 (though Standard Edition One is only $5,800). That's nothing to sneeze at. I don't know Oracle that well, but with SQL Server, you get a whole suite of technologies--ETL (SSIS), data warehousing (SSAS), and reporting (SSRS) in particular. You get a number of disaster recovery options, like 2012's AlwaysOn.

By default, PostgreSQL works from the command line. You can download PGAdmin, but Navicat costs money. Graphical ETL tools? I don't think so. Reporting? You'll probably have to spring for Crystal reports. Data Warehousing? I hear Pentaho is a pain. It would be hard to keep PostgreSQL free--at least in an enterprise environment.  If you're just doing CRUD for a web UI, you're probably ok--for a while.

The second thing that makes me nervous is the ridiculous number of open source technologies out there. The principle of open source is to 'let a thousand flowers bloom,' but in practice this means playing around with a whole lot of different products and figuring out how they can fit together. It seems like chaos. Microsoft is known for its Three Letter Acronyms (TLAs), but at least everything fits together pretty well.

Finally, because of the complexity of open source solutions, I can't imagine they'd be easy to maintain. Most companies now outsource common DBA and production support tasks. If you're running a PostgreSQL OLTP system with a MongoDB document store and Hadoop web logging, you'll have trouble getting Tata to maintain that for you. Even if you're not interested in outsourcing, it is useful to have standards and best practices across companies.

Of course, it's the innovative start-ups that are using new open source technologies to do new things.  This is one of the main factors that helps them be innovative. They don't want to be standardized. They want to be part of the bazaar of new tech IPOs, rather than the cathedrals of established companies--at least until they have enough users for the cathedrals to buy them.

Anyway, I'm having a lot of fun.  Check out my new favorite book.