Style guide style guide

After looking over Google's style guide for shell scripting, I feel like I need to write up what should be obvious: the style guide for style guides. It's a common failing of coder culture that we think it's best to try to write style guide documentation as if it were code, being as explicit as possible and catching exceptional conditions where possible.

Unfortunately, the audience is not a computer, it's a programmer, and programmers do many things that computers don't:

• Ignore large sections of prose
• Discard the parts they disagree with
• Become defensive

So, to avoid these problems, here are some simple guidelines along with examples from the Google document.

Critical skills for new computer science graduates

So, you've decided to go to college for computer science. Good for you. I'm here to tell you that most of the time, you won't care about most of what you've been taught. In fact, what you need to know might not be obvious at all. In many cases, it's the secondary tools, not the abstract knowledge, that's most important, and that is a rarely communicated view that many graduates I see have never been informed of.

Here's what you need to learn (much of which you probably won't unless you seek out the knowledge on your own) before you graduate and start looking for work. When I interview people, I expect you to know these fundamentals as a starting point, and I ask questions that build past these basics. If everything on this list isn't second nature to you, then you probably won't be able to deal with the kinds of things that I'm going to want you to figure out in an interview at all.

Why do we argue about text editors?

Google for "emacs vs. vi" and you'll see a torrent of pages that claim that one text editor is superior to another. Supporters of vi will claim that emacs is huge and bloated with unnecessary garbage that's unrelated to editing. Supporters of emacs will claim that vi is just a toy, capable only of the most basic text editing, and falling short whenever a complex task appears.

The waters have gotten muddy, of course. Nowadays, emacs is a fairly small application compared to behemoths like a browser or an IDE like Eclipse. Similarly, there are newer, beefier versions of vi such as vim that provide many of the features that emacs users claimed were essential to emacs's superiority over vi.

So, why do we argue about which is best? Wouldn't it make sense to learn all of the editors out there and then make an informed choice as to which to use in any given environment? Well... yes, but we don't argue about editors for rational reasons, and therefore (much like religion) we can't easily come to rational choices about their use.

I think the problem becomes much clearer if you compare a text editor to a spoken language. Speakers of American English will tell you that "colour" is spelled wrong, but this was a gradual change that occurred over the course of the 18th century, splitting the American and British spellings. Clearly, what we're dealing with is a spelling error that occurred in the United States and then became entrenched. So, why hold onto this error? Why not just normalize the language now, across both regions? Because language isn't used just to communicate. In fact, probably just as important is its use to help distinguish those who are from other subcultures, and thus not immediately trusted as "us".

The same is true for editors. We argue about them because they are the tools we use to communicate our thoughts. The parts of our brains that discern "us" from "them" on the basis of language aren't aware of the idea that we're using our editor to communicate with a non-sentient machine. Those parts of our brain just experience the routine process of turning thoughts into communication. So, when we consider another editor, we immediately recoil because, to part of our brains, that feels an awful lot like becoming a traitor to our people; abandoning the marker that shows which social group we belong to and picking up a new, alien language.

It doesn't matter that none of this actually applies, it's just a quirk of the way we think about communication.

So, the next time you run into someone who tells you that, "emacs sucks, vi rules!" or visa versa, just tell them that they need to spell that "emaucs"...

The Android App experiment has failed

My experiment was this: spend 3-4 months doing Android App development, and see if I could make enough profit to justify continuing down this road of freelance development, professionally. The answer is no. The hard fact of the matter is that in 1 month of active use of my first free app, it has made back 1/25th of the money that I sunk into advertising it. Also, in a little under a week, with $50 sunk into advertising, my first for-pay app has 3 installs at $1 each.

While I'm sure that I could continue to put out apps and would eventually see more revenue than this, it is increasingly unlikely that I will see revenue on which I could make a living wage.

Failure is part of life, and I've learned a new programming language and a fairly complex platform in the process, so I don't feel the last 3 months have been a waste, but now is the time to go back to work and start making some money again. I'll leave my apps out there and submit bug fixes from time to time. Who knows. Maybe at some point, there will be a surge of interest...

Android AsyncTask management

Warning: Coding ahead...

Releasing my own Android app has taught me an amazing amount about Java, Android and the state of commercial app libraries in general. I now have a deep and profound respect for what I considered "trivial" apps, just a few short months ago.

My first app, which still isn't in its final release form (though it's on the Android Market in beta form) had one pretty large problem: it loads images, and at times it would get stuck, forever downloading an image. The solution was to actively manage tasks, of course, but I'm new to Java, and I didn't know much about its task management facilities. I had assumed that it was fairly straightforward, and I would simply:

 task.cancel();

Alas, that's not it. as I quickly discovered, the AsyncTask class's cancel method is more of a hint. It tells the task that it should wrap up what it's doing, by setting a semaphore (or whatever the underlying mechanism is) and it's the job of the child task to check in with the isCanceled method and react accordingly.

This presented a great deal of complication for me. When the user presses the "back" button or clicks outside of the progress dialog to cancel a download, I don't want to leave a stranded download going, but the download code isn't mine, it's part of the BitmapFactory class. How can I tell it to stop what it's doing? The solution is fairly complicated, and I hope that I'll refine it or find there's a better way over time. Here's what I arrived at (after the break):

Android roadbumps

As some of my readers know, I'm writing Android apps. It's been slow. The Java learning curve is actually fairly steep (at least as compared with high level languages I've been working in like Perl, Python and so forth). Just to give you a taste of what you have to look forward to if you want to do some Android development, here's what I've been fighting with today.

First, I had a really silly problem. I had a list and I'd defined a callback called onItemSelect for it. I set a breakpoint in this method and ran the app under the emulator. Click. Click. Click... nothing. I kept going over the code and trying to figure out how this could happen. Click. Nothing. Damn!

So then I was just messing around in the emulator and accidentally hit the scroll wheel. Bang! My callback is invoked! After scratching my head for a second, I had a brain storm. I checked to see if there was an onItemClick callback, and indeed there is! What onItemSelect does is apparently handle selecting, but not clicking on an item. What I'm not sure of is how you could manage to select an item without clicking on it outside of the emulator... honestly, I can't figure out of this is even useful. Maybe if you used the D-pad to navigate a list...

Then, I was trying to write a class that handles a download for me. It needed to take two parameters that represent callbacks. Now, Java is kind of neurotic about insisting that there's no such thing as a function, so when you want to do something that's clearly functional like passing around a callback, you have to do it in terms of an object oriented behavior like sub-classing. I can't begin to explain how horribly wasteful this is in terms of coding and efficiency, but let's just get past that. Here's what I tried to write:

Class file A.java:

import B;
class A {
  public void registerCallbacks() {
    B.new(new B.OkCallback() {
      @Override
      public void callback(String result) {
        // do stuff with result
      }
    },
    new B.FailCallback() {
      @Override
      public void callback(String message) {
        // deal with failure represented by message
      }
    }).execute();
  }

And in B:

  class B {
    public void execute() {
      // do stuff
    }
    public abstract class OkCallback {
      public abstract void callback(String);
    }
    public abstract class FailedCallback {
      public abstract void callback(String);
    }

Ignoring the obvious duplication (since it wasn't that simple in my real code), there's only one error, but it's a really hard thing to find if you're new to Java: "static" is required on the definition of both abstract classes. I'm not entirely sure why you would ever define a non-static, abstract, nested class, but I guess there's some application for declaring child classes... still, it seems like this kind of runaway syntax is just absurd. To give you an example, let's look at a hybrid functional/OO language like python:

  from B import B

  def handleResult(result):

    # do something with result

  def handleFailure(message):

    # do something with failure message

  class A(object):

    def registerCallbacks(self):
      B.registerCallbacks(handleResult, handleFailure).execute()

Python file B.py:

  class B(object):

    def execute():
      # do stuff...

Notice that, not only is the code simpler, but the extra layers of object-creation, subclassing and all of that noise are gone from the call stack. You pass a function to B and it invokes it when needed, with the appropriate parameters.

Even in Perl, functions can be passed as subroutine references and invoked by the caller. In C and C++, function pointers aren't the same thing at all, but for simple callbacks, they work well enough.

Java is fundamentally flawed in this way, and I'm hoping that they crank out a version 8 or whatever, wherein they finally give up and allow real functional programming.

Building an Android App Development Workstation

I've recently started doing Android App development. It's actually not trivial to get started, and I hit a number of difficult roadblocks along the way. I started off by thinking long and hard about OS. I'm a Linux guy at heart, and Android is Linux-based, so a Linux system seemed the right choice, but I didn't want to have to have two desktops, and in my spare time I also play Windows-based games, so in the end I decided on Windows with a Linux virtual machine.

Next up was my choice of hardware. I already have a good keyboard, mouse and display, so I only needed the box. Along the left, I've linked the Titanium Gamer AMTI7013 which is roughly the same hardware that I bought (though I got mine at MicroCenter). This has the Sandy Bridge-based i5 2500K. The system I picked up had more RAM, so you might want to explore picking up some extra if you buy that box.

My primary performance concern was the Android emulator. It's slow and greedy, so I wanted plenty of RAM and at least 2 if not 4 cores so that it would be guaranteed plenty of dedicated instruction pipeline.

Having picked the hardware, I needed software. It's amazing how much software you have to install to get going. I started with the Android Development Tools (ADT) installation guide. But in the end, I had to install all of this (I'll describe this for the Windows setup, since I'm not running the emulator under Linux):

• Chrome browser (you could go with Firefox if you wanted)
• Java SE
• Eclipse Helios for Java
• Android SDK starter package
• Then I followed the ADT installation guide
• I had to make sure to install the USB plugin and all of the API levels I will be developing for

Optional components that I found useful included:

• Google Plugin for Eclipse - Useful for many purposes, but especially key if you have an app that uses Google App Engine (GAE) on the back-end; see the recent Google I/O talk/announcement and get:
• The Google Web Toolkit (GWT) 2.4 beta or later.
• Google App Engine SDK (make sure to get the Java version if you want to develop the back-end in Eclipse)
• Gimp for image/icon/logo creation and manipulation.
• VirtualBox for installing Ubuntu Linux
• Vim if you like having a quick-and-dirty editor around for opening random files (much lighter weight than opening files with Eclipse, but much more capable than notepad)

A few tips on getting things installed: make sure that you create a new directory that's separate from everything else (I made it C:\Dev) to put all of the downloaded items in. It's important that you not place Eclipse into existing system areas, since it doesn't always play nicely with the Windows file protection model.

Some notes about your app. You're going to need to sign it eventually, so you might as well get that out of the way when you're still in the "hello world" stage of development. To save a signed APK, just right-click on the top-level project name under the "Package Explorer" in Eclipse and select "Android Tools -> Export signed application package..." Once you've done this, you can use that key that you created for lots of useful purposes, not the least of which is to interact with Google APIs for services like Calendar and Maps. For example, there's a document that Goolge provides on extracting the md5 signature of the key and using it for access to the Maps API. To do that, you'll need the keytool program that came with Java SE, and can be found in that package's "bin" directory.

I'll continue to post more as I learn and get my first application up on the app market. Enjoy!

Python subprocess vs os.popen overhead

Let's say you're writing a Python program and you want to run an external command and read its output. The right thing to do used to be to:

 import os

 p = os.popen("command")

 output = p.read()

But there were a lot of ways to run programs depending on what kind of output you wanted to read (if any) what kind control you wanted (if any) and so on. Thus was born the subprocess module. In the current, 2.7 documentation for the os module, there's a note on popen:

Deprecated since version 2.6: This function is obsolete. Use the subprocess module. Check especially the Replacing Older Functions with the subprocess Module section.

Well, that's pretty definitive, right? Unfortunately, not so much.

Typically, process creation overhead doesn't matter a great deal. If a program needs to run another program, then the startup time involved in the creation of the process is probably an order of magnitude (often several) less than the time that the new program takes to do its work. So, you only typically care about process creation overhead when you're creating a very large number of parallel children.

Unfortunately, I work in the world of system monitoring, and in that world, creating a few hundred or thousand programs a second during peak times (subordinate monitoring tools) is not a rarity, and even when the load is much lighter, large amounts of process creation overhead isn't always ignorable. For example, if your system is doing a lot of IO, then large memory operations during process creation might reduce the amount of caching the system can do.

All of these factors lead me to test the subprocess module against os for a simple case: I want to run a process under a shell with standard output being captured. With the os module, my test looked like this:

 import os
 for i in range(10000):
    f = os.popen("exit 0")
    f.close()

With the subprocess module, my test looked like this:

 import subprocess
 for i in range(10000):
    f = subprocess.Popen("exit 0", shell=True)
    f.wait()

I timed the two scripts and came to a surprising conclusion: subprocess has about a 40% process creation overhead over os.popen! That's an awful lot of increase, so what could be going on? My next step was to use strace to determine what could be taking up that extra time. Here's a partial strace fo the subprocess example under Linux:

pipe([3, 4])                            = 0
fcntl(4, F_GETFD)                       = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
clone(...) = 3306
close(4)                                = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f358acc6000
read(3, "", 1048576)                    = 0
mremap(0x7f358acc6000, 1052672, 4096, MREMAP_MAYMOVE) = 0x7f358acc6000
close(3)                                = 0
munmap(0x7f358acc6000, 4096)            = 0
wait4(3306, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3306

The os.popen example had one less fcntl and an extra fstat, both of which are fairly light weight. The real culprit here are the mmap, mremap and munmap calls that subprocess is doing. Why are those there, I wondered. In looking at the subprocess code, it seems that these operations may be the result of thread creation which subprocess uses to manage reads and writes on subprocess inputs and outputs, but I'm not sure. What is clear is that the subprocess module is about 1,300 lines long while popen was a builtin supplied by the interpreter.

Conclusion

subprocess is a valiant attempt to make a complex snarl of library calls into a uniform tool. The problem is that process creation is one of the most fundamental operations that a language performs, and when a simple task like running an child process and reading its output becomes too heavy, a language suffers for it. Perhaps subprocess should be simplified and its convenience routines re-written as low-level operations that are optimized per-platform. Or perhaps os.popen should be undeprecated. After all, I'm willing to bet that managing fork, pipe and exec operations from Python will never be as low-impact as calling the C library popen(3) function.

Something new every day: Bourne Shell variables

I've worked with the Unix Operating System and its variants since the late 1980s. I've worked with the Bourne Again Shell (bash) since the early 1990s. And yet today I learned something new about variable expansion. In the startup scripts for a source code indexing system called OpenGrok, I found this gem:

somecommand ${PROG:+-c} ${PROG}

Now, I know that ${FOO-bar} will be replaced with the value of $FOO if it is currently set or "bar" if it's not. That much I learned many years ago, but this usage of "+" was new to me. After some testing, I found that "+" substitutes the following text if and only if the variable is set, otherwise it substitutes nothing. Thus if $PROG were set to "foo", the above text would execute:

somecommand -c foo

But if $PROG were not set, then somecommand would be run with no arguments at all. Very slick!

How I managed to go over 20 years without learning that, I'm unsure (then again, perhaps I've learned and forgotten it...)

A Google App Engine failure

Long ago, I wrote a Perl script that generates random names for my roleplaying games. It's a simple thing, but it can take input lists from any language and spit out similar-sounding made-up names. It's a powerful, but simple tool, and it seemed a natural fit for my first exploration of Google App Engine. Sadly, it didn't work out that way, and I thought it might serve as a useful caution to others who might plan the same sort of work.

The fundamental problem is that my app is IO-hungry. It reads in the entire source list every time someone asks for a made-up word, crunches it down into first-parts, mid-parts and end-parts (2-3 letter segments which are rooted at the beginning or end of the word or neither). We then sort the lists of parts according to frequency of occurrence and perform a weighted, random pick of a first part, then each subsequent part is chosen in the same way, but from a subset of all of the parts, which overlaps the previous segment. The combination of weighted choice and overlapping leads to words which tend to be pronounceable in the source language of the input list.

This process of reading and processing all of the words every time wasn't something I was going to be able to do in Google App Engine, however, since costs are associated with resources consumption. So, I set out to store the pre-digested versions of the input lists as sorted word-segments in the Google App Engine datastore. This is where my problems began. While it's entirely possible to store the data this way, what I found was that my need to access so many records from the database as I performed my random walk down the lists of word-parts left GAA gasping for breath. In practical terms, I'd created the world's slowest tool for producing babble. Of this, I'm sure my mother feels proud.

Frankly, I'm not sure what I can do about this. GAA just doesn't seem to have been designed for this sort of thing. A shame, really. Of course, I could pre-compute a queue of results for each source namelist and keep re-populating them with a periodic job, but that really seems like a cheesy way to solve a problem that takes a few seconds for my original Perl script.

Writing a Perl 6 URI module

I wanted to write a parser of some sort using Perl 6's spiffy parser language otherwise known as "rules". This is the super-extended regular expression syntax that Perl 6's own parser is written in, and it's not just powerful, it's easy to use. In fact, it's so easy to use that almost all of my time writing a URI parser module was spent on other aspects of the code than the parser itself.

First off, some background. Perl 6 has a URI module already. However, it relies on a number of Perl built-in character classes to match things like digits and alphanumerics. In reality, the RFCs that define URIs are very precise, and there are different specifications depending on what you need. So, I decided to re-write the module with a pluggable parser so that you could give a regular, modern URI and have it parse correctly, but you could also ask for special "IRI" parsing on an internationalized URI and the right thing would happen there. I even went so far as to bring in an older version of the specification as a legacy mode.

The current state of the Perl 6 parser and runtime called Rakudo is actually fairly solid for a pre-release implementation of such a complex language spec. There are some gaping holes, but they were all relatively easy to work around. Some of these included overly aggressive list-flattening, some operators that were broken at the time I wrote this code and the big one: named rules only work as a stand-alone grammer with a specific entry-point called TOP.

I worked around all of these issues and have, so far, been able to parse basic URIs according to RFC 3986. Here's a sample of what a Perl grammar for URIs looks like:

    token URI {
        ':' [ '?' ]? [ '#' ]?
    }

Here you can see most of the basics: "token" introduces a single expression within the grammar. It calls out to other tokens by enclosing their names in angle-brackets. Literal sequences are enclosed in single-quotes and sub-expressions can be enclosed in square-brackets with regular expression-like repetition counts such as ? for 0 or 1 matches.

In order to have a pluggable interface, I needed a class capable of providing me with two things for each grammar: the grammar itself and a set of routines which would tell me how to find the resulting URI elements in the match data. For this I defined an interface using Perl 6's roles:

  role URI::Specification {

      method parser() { ... }

      method scheme_path() { ... }

      # ... other _path methods here...

  }

Those ellipses are literal. They cause the methods to be required for any class composed with this role, but do not define any functionality themselves.

Each parser is then defined as:

  class URI::rfc3896 does URI::Specification {

      grammar URI::rfc3896::spec {

          token TOP { }

          # RFC definition of URI goes here.

      }

      method parser() { return ::URI::rfc3896::spec }

      method scheme_path() {

          gather do { take }

      }

      # And so on ...

  }

That's it. The only really funky bit here is the gather/take code in the scheme_path. That's the way Perl 6 defines a coroutine-like interface. The paths define how we traverse the match object to find match results. So, for example, the "scheme" (the "http" in "http:/www.example.com/") can only be matched in the URI rule's scheme sub-rule. Some URI elements, however, such as authority (the host name and port - possibly username as well) can be matched multiple ways, so these routines might return multiple lists of subrule names to traverse. I would have simply returned a list of lists, but Perl 6's parameter passing is very complex and currently some of the specification is not yet implemented. Right now, this manifests as overly aggressive list flattening when returning them from a subroutine or method.

This is why I used coroutines to return each of the sub-lists, one call at a time.

I'll continue to post new updates as my URI module nears readiness. For now, it's just awaiting some love on the other parsers, and I think it'll be ready to go.

Python class attribute annoyance

Python class attributes are fairly handy things, if somewhat visually misleading. Coming from other languages, you might expect this:

  class Foo(object):

    a = 10

To define a class whose instances will have one attribute called a. Not quite. a is actually what most languages call a "static attribute" or "static member" of the class itself, not the instances. Python calls these "class attributes." Once you know this, class attributes are a tool you'll reach for in a number of circumstances, but they have subtle behaviors that can feel like bugs.

For example, today I was trying to do something like this:

  class Foo(object):

    a = 10

    b = a + 5

which works just fine and does what you might expect (a is 10 and b is 15). But, this will yield an error:

  class Foo(object):

    a = 10

    b = [ a+i for i in range(1,11) ]

You might expect b to contain [ 11, 12, ..., 20 ] but instead, you get an error telling you that a isn't defined. [Note: tested in Python 2.6 and 3.1] This subtle flaw exists because that a+i is actually being executed in a nested lexical scope, but because it was created inside of a class body, it fails to inherit what appears to be the parent scope and thus has no access to its lexically scoped variables. There are many ways to accomplish what you might have intended, here, but none of them are very clean. For example:

  class Foo(object):

    a = 10

    b = [ z+i for z in (a,) for i in range(1,11) ]

Now you are passing a as a parameter to that nested scope, so it works perfectly. It's certainly a stilted way to do this, but it works just fine.
Coming, as I do, from Perl, this feels very odd. Perl's OO model is, at best, a framework upon which to build your own. Even still, this kind of scoping problem just never happens. Any lexical scope introduced anywhere in Perl will have a parent scope which is visually quite obvious. Running into such subtle shifts in Python's behavior seems counter to its stated goal of simplicity and elegance.

Object Oriented Programming or "The Kids, These Days!"

It began in the mid-90s... the definition of OO programming started to wander. Before then, it was simple: OO was the trinity: inheritance, polymorphism and encapsulation. Most OO implementations supported some additional concepts such as data hiding or metaprogramming, but these were essentially their own fields of implementation and interest, only tangentially related to OO.

Over time, implementations began to become more entrenched as camps of OO theology. Java developers began to believe that data-hiding, interfaces and single-inheritance were core OO. Python and Ruby developers believed that metaprogramming and introspection were core OO. Smalltalk developers believed that the world was populated by idiots who could barely tie their shoes, much less be trusted with compilers. They're all wrong, of course (well, the Smalltalk folks might be on to something, but the Haskell people want to discuss the lack of rigor in their definition of "shoe").

Object oriented programming is still what it always was: a way of abstracting data using three basic tools. It's not the be-all, end-all of software design, and new ideas aren't to be judged as right or wrong, purely on whether or not they can be shoe-horned, retroactively into the definition. Equally, no language I've ever seen gets it completely right, and no language is unsuited to OO programming concepts, regardless of how much sugar they may lack. C (not to mean C++) is a fine language for OO development. Perl 5, interestingly, doesn't provide an OO system, only the basic tools required to build one. Python has a fairly robust object model, but one that many complain was "boiler plated on" to the core language (that's changing). Languages like Java integrate the object system so deeply into their core that you can't escape them, no matter how simple the task.

But none of this matters. Let's go back to first concepts and review what an object is and why it's a programming concept. Later, I'll get into why it's not a development concept, and review why that's a different thing entirely.

Inheritance is the first OO concept. It is fairly simple. A dog is a mammal. If we know what a mammal is, then we only have to describe what a dog has that's different from mammals in general. For example, a dog is a highly social/pack-oriented mammal that has a highly developed sense of smell, tends to be of medium size for the animal kingdom, and produces a largish litter of young. That's basic inheritance. There's no mystery, just a way of describing data in terms of its structure as unique from lesser-defined data types. The concepts of abstract types and of styles of single vs. multiple inheritance and traits/interfaces all fall out of this core idea, but they are not fundamental to the idea.

Polymorphism is up next. This is where you have a dog, but you want to bring it with you on an airplane. The airline has rules for how you bring a mammal onto the plane, so you tell the person at the checkin that you have a mammal. Now they know exactly how to deal with it. It's the same in OO programming, and exactly that simple. From this, we have derived many complex concepts, and every language implements polymorphism differently because of how deeply it ties into your parameter-passing mechanism, your data-hiding model and your type system. However, those are implementation details. OO is, again, a simple concept and straining it to include language-specific implementation details is only useful when engaging in language holy-wars. Notice that polymorphism relies directly on inheritance. There is no separating polymorphism off as its own concept in a world without inheritance. Polymorphism implies the existence of behaviors such as a dog's ability to run, nap or fetch. It is these behaviors which are polymorphic (all animals can sleep while only mammals can nurse their young).

Encapsulation is a bit what it sounds like. When you say that a dog has hair, you're not describing a unique concept. You're simply referencing a previously defined data type ("hair") and encapsulating it within the definition of a dog. Now, when you say, "here's a dog," I can ask, "what color is the dog's hair," and it makes perfect sense. You don't have to consult some external resource to answer the question, you just look at the dog and see that it has hair, and then look at the hair's color.

Now we can construct an object. Objects are data structures which embody polymorphic behaviors via inheritance while, simultaneously, providing encapsulation. A complete example using our dog would be the whole dog. A dog is an animal; it can be treated as one, and should behave abstractly like any other animal, though it might possess its own unique behaviors as well. It also contains all of the traits that one associates with dogs, many of which are full-fledged objects of their own (a heart that beats, a vocal apparatus with which it barks, etc.) All of these things describe a dog.

Now, that's the abstract. In practice, programming languages need to be able to describe all of this and use it, so there are two additional items: instantiation (the ability to bring an instance of an object into being); and members (encapsulated objects or primitive types—for those languages that distinguish). These aren't truly OO concepts, but implementation details common to nearly all OO languages.

Further complicating the issue are language features that have grown out of OO. Entire realms of these features exist, some rivaling the power of OO itself (metaprogramming comes to mind). Inheritance, for example, has been implemented as single, multiple and Java-like interface-augmented single. None of these are inherently wrong, nor are they part of the nature of OO programming.

Then there are software design concepts. Static classes, static members, privilege models, accessors, design patterns and so forth are all examples of software design and development tools and concepts which are valuable when interacting with OO languages and architecture. However, it's important to understand that OO exists on its own without these concepts. They help us to produce code that functions using OO, but a language which had none of these features and software that was designed without their benefit would still be "OO."

The Python Platform

Update: I believe that the example for this essay on Python as an incompatible platform is incorrect. The bug report that I submitted was updated 3 months later to indicate that I'd read the documentation wrong. I think the core point here is still valid. There's a lot of "not invented here" applied to the UNIX and Linux conventions in Python, but I chose a bad example, and for that I should apologize to the Python community. I like Python. I like programming in Python. I don't want to make it sound like I'm dismissing the language, here, just a particular trope in the community.

When Java came out, I remember the promise that it would be the write-once-run-anywhere language. It was supposed to free programmers from the need to tie their code to a platform, and instead they could simply write it. This never really happened. Instead, what we got was the Java (or more accurately the JVM) platform. It wasn't really a great platform as these things go, and for a short time that confused me. I wasn't sure why the smart people at Sun would be unable to create a decent platform on which to write generic code.

Then it came to me... It was Windows. You see, Sun had a pretty decent little operating system called Solaris (né SunOS), but Java was supposed to work everywhere, so at a minimum, it would have to accommodate the world's most popular desktop platform at the time (and still, though it has less market share now): Microsoft Windows. Windows has its own ideas about how a system should manage users, permissions, networking and a host of other things that programs want to interact with, so Java couldn't allow the same code to run everywhere while exposing the powerful semantics of the Solaris operating system. More broadly, it couldn't expose those core Solaris semantics that came from its Unix heritage, embodied in the POSIX standards. These standards are what make C, C++, Perl and many other language's standard libraries so powerful, and because of that power they are also widely portable. So useful are these standards that they have molded operating system after operating system, all based to some extent on Unix. Today this includes Linux, MacOS and a plethora of lesser-known systems, all of which have important niches in various industries such as HP/UX and AIX.

So, in the late 90s a new language started to gain popularity: Python. It didn't fall for Java's trap entirely. It was mostly in league with the POSIX way of thinking. Process management, file IO and many other aspects of the language were all very reminiscent of POSIX. However, Python suffered from a new problem: it was the anti-Perl. Perl, you see, is a programming language that became very popular in the early 90s, and Guido van Rossum, Python's original author made his feelings about Perl fairly clear early on. He wasn't fond of it, and Python was going to avoid its mistakes.

While correcting the perceived mistakes of another language might seem a noble goal, it has several pitfalls which must be avoided. One of the most obvious of these is avoiding something only because the original language embraces it. Python has had a rocky relationship with POSIX for just this reason. You see, Perl is a deeply POSIX-based, and even more specifically, Unix-based language. Python, as I said, is mostly a POSIX-friendly language as well, but there's a silent mistrust within the community of the platform that Python's nemesis language so readily embraces, and this has lead to a number of almost-entirely-POSIX-friendly choices which, when seen as a whole, yield the Python Platform.

This platform is not entirely POSIX-compatible, which means that users of Python and Python programs on both POSIX and Windows systems must adapt to it, in the same way (but to a lesser extent) that they must adapt to Java's platform.

OK, so that's the generalities, but what about specifics? One must look no further for a simple example than the Python standard library's command-line processing module, optparse. This module has a simple documentation bug, but that bug illuminates the Python Platform in stark detail. Here's an excerpt:

"... the traditional Unix syntax is a hyphen (“-“) followed by a single letter [...] Some other option syntaxes that the world has seen include: a hyphen followed by a few letters, e.g. "-pf" [...] These option syntaxes are not supported by optparse, and they never will be. This is deliberate: the first three are non-standard on any environment[...]"

This sounds reasonable, after all, why support oddball features? Well, it turns out they're not so oddball. The POSIX standard says that the compliant program "accepts any of the following as equivalent: 'cmd -ao arg path path', 'cmd -a -o arg path path' ..." notice that traditional Unix and POSIX programs such as the "ls" command will always accept these concatenated arguments. So why would Python tell us that this is non-standard on any environment? That goes back to the mistrust that Python has for POSIX. There's no compliance testing for the Python library's POSIX support because Python isn't a POSIX language. It's a Python language.

I decided to take the cautious approach here. I didn't want set anyone off, so when I submitted the following alternate wording to the Python folks:

"optparse has chosen to implement a subset of the GNU coding standard's command line interface guidelines, allowing for both long and short options, but not the POSIX-style concatenation of short options."

... I kept it simple and factual. I didn't attempt to suggest a rationale, and I made it clear that I wasn't asking for Python's behavior to change, just the documentation. This bug report sat for three months and then was silently lowered in priority, even though it required nothing more than a cut-and-paste documentation change.

This is but one example of where Python chooses to go its own way, eschewing the wisdom of a platform that has suited the likes of Sun, Apple, HP, IBM and countless FOSS developers for decades now. I like Python. I think it's a great language for certain types of tasks. I'm also a fan of its nemesis, Perl, along with a host of other languages I've used over the years, but time and time again, I see the Python community embrace a culture of exclusion and "not invented here." For Python to truly reach the potential that I'm sure it has, it will have to be able to embrace the tools that have worked well and only discard features which have been carefully considered and understood.

There's hope, though, that this will be the case. Guido's distaste for Perl and some other Unix tools like it may have fueled this fire, but it turns out he's a very reasonable person for the most part. In a recent blog post, he says, "It's no wonder that users are switching to the web as the platform for everything that used to live on the desktop -- with all its flaws (which I will discuss another time), web development still feels like a breeze compared to Windows development." I take this as a sign that he understands the power of a platform which works with consistency at all levels, and that he will continue to improve the Python Platform so that it builds synergy with the POSIX platform, and doesn't fight against it as if it were an opponent to be conquered.

The Power of Perl's Data::Dumper

Update: sorry about the code formatting. I don't appear to be able to get Blogger to play ball with me right now. Use your imagination, especially with respect to the python examples which simply won't work with the indentation shown.

I've been doing a fair amount of work in Python recently, and there are some things I really like about the language. However, being an old Perl programmer, I find myself desperately wanting a few features of Perl in Python, and one of them is Data::Dumper. This module lets you print out the contents of a variable as Perl code. Now, Python has some similar features such as pprint and pickle, but neither of them quite gives you what Data::Dumper does.

For example, here's a class definition:

  package Someclass;
  sub new {
   my($class, $param1, $param2) = @_;
   return bless { param1=>$param1, param2=>$param2 }, $class;
  }

When we dump out an instance of this class, we get:

  $VAR1 = bless( {
          'param2' => 2,
          'param1' => 1
        }, 'Someclass' );

This is exactly the code you need in order to re-create the object (which you get from pickle), but it's a human-readable copy of all of the state contained within the object at the same time.

In Python, you might write:

  class Someclass(object):
     def __init__(self,param1,param2):
       self.param1=param1
       self.param2=param2

But the pprint output just calls repr and you get:

  <__main__.someclass>

This is because Python relies on each object to provide its own serialization method, called __repr__. If you don't define it (and sadly, many don't define anything useful, here), there's no way to know just what it is that's going on under the hood other than by writing your own introspection code. You could, for example, treat the object as a dict and peruse its attributes:

 >>> pprint.pprint(x.__dict__)
  {'param1': 1, 'param2': 2}

but there are limitations to such an approach, especially when it comes to encapsulation.

Anyway, the point is, this is one area in which Lisp, Perl and other languages that can represent arbitrary data as code have an ease-of-use advantage over languages that cannot. Hopefully this is being addressed in future versions of the language (I'm not using 3.x yet).

Git, BitKeeper and the Power of Open Source

Update April 2012: The comparison page that I reference now just mentions "other SCM", but a side-bar continues to compare their product only to non-distributed, circa 1980s and 1990s offerings.


Back in the mists of 2002, debate ran hot in the Linux development community. The debate was over a proprietary source code management (SCM) tool called BitKeeper that was used as the primary SCM for the Linux Kernel. When a dispute with the vendor resulted in a schism between the Linux developer community and BitKeeper in 2005, the tool was dropped in favor of a replacement written by Linus Torvalds over a one-month period. To understand the importance of this achievement, understand that BitKeeper was written by eight developers over the course of three years and McVoy, its primary architect and original developer estimated that it would cost $12 million to do it again in an ordinary, non-startup company.

Instead, Torvalds sat down behind his keyboard and set out to replace it. How successful was he? If you look at BitKeeper's comparison page with other SCM tools today, you'll notice that it compares itself to many other tools (and makes quite a few rather large errors along the way), but none of them is git. In fact, none of the list are any of the next-generation tools that have followed in git's wake such as Bazzar or Mercurial. Why? Well, git is simpler, easier and better. It also happens to be radically faster. There's no point in comparing yourself with such a tool in public, since it's only going to make you look bad to say that the free tool is radically better.

McVoy also made the claim that a replacement for BitKeeper wouldn't be possible because it was too hard and programmers capable of doing the work wouldn't do it for free. Why is this? Well, it comes down to graph theory and its application to text revisions. Recognizing text differences is hard enough, but to extend that to maintaining a directed acyclic graph of revision histories and branches in a distributed way... well that's downright hard. Sure, it's hard, but then so is writing a POSIX-compatible kernel. The fact that there are now three excellent options out there for distributed source code management that excel at doing just what McVoy said would be impossible to reproduce should go a long way to demonstrating that free and open source software development is one of the most powerful new paradigms of engineering to come along since the invention of the functional specification.