Toward an Open Source television studio

In the software world, Open Source software has won. The last holdouts are still fighting the good fight, but the idea that you develop software in the open, while building a business model around local extensions, support and professional services takes such a myriad of forms that it's now hard to imagine a world without Open Source. Apple, Google, IBM, Red Hat, Oracle, VMware... they all make heavy use of and contribute to Open Source software.

Many have tried to apply this model to other areas. Wikipedia is probably the best known example of a non-software Open Source project, but there are many others. In Japan there's even a virtual pop star whose music and videos are essentially a gigantic Open Source project. Today, I got to thinking about what Open Source television would be like, and I think it would be very interesting. Here's how I'd see it working:

A studio or consortium of studios develop a setting and some central players. Let's say you want to set a comedy in a business (like The Office). You develop a "bible," as the industry calls it, that describes the business in some detail; its important players, background and the elements of the world that it's in that differ from the real world, if any. Then, you write a second bible that describes the show in terms of main characters, the shape of any given episode and some of the stylistic signatures. You release the first part as Open Source (say, under CC-By-SA), but the second half is proprietary.

Each time you release and episode, the show itself is licensed as normal, but you revise the setting with any new details that you've developed in your show.

Legal concerns: Television is a complicated world, and there's certainly much more scrutiny of where ideas come from and who owns them than in other industries. Because of this, it might be necessary to build some protections into the licensing of the open source piece That's a hurdle, to be sure, but I don't think it's a very high one.

As time goes on, the creator studio can continue to build new open or proprietary content in this world, licensing it as they see fit. Others can publish open source shows, books or other media set in the same world, but they cannot re-use the specific characters or details from the proprietary shows.

This gives the broadest number of options to the original creators of the show, but if a vibrant ecosystem forms around this nucleus then it probably makes sense to start publishing new content as open source. After all, television makes its money on advertising, and there is still going to be a demand for seeing episodes when they first air, which drives the audience numbers, which yields ad revenue.

In terms of secondary revenue such as DVD sales, proprietary shows will still have control. Open source shows will not, but there are ways around that. Special boxed sets, signed by the cast, for example, could drive sales of the original (this brings up a point: it may also be necessary to modify the licensing such that open source shows can produce proprietary packaging for media sales).

Anyway, the idea is simple: by building a community around the development of your core intellectual property, you give up some control while gaining massive productivity multipliers. This is how Google and Apple are able to innovate so fast. It's why Oracle was able to step into the business of distributing their own operating system, literally overnight. This isn't an idea, it's a proven business model, and frankly, I believe it's one that every industry is going to have to adopt or find a way to compete with.

Clamchowder (ne Perl 6) to offer shell migration package

This was an April Fools post.


The rich history of Unix shell scripting is often overlooked by the community of "real programmers" who work in languages like C, C++, C#, Cobol, COMTRAN, CPL, Common Lisp, CLOS, CLIPS, Clojure, and even some lesser known languages which start with other letters. But this has left decades of important infrastructure tools unavailable to the average programmer without using an external command like "system" which is often considered a security risk due to the existence of "." in everyone's path.

As a result, the Perl 6 community has announced a new suite of tools that aid in converting old shell scripts into this modern language with all of the features one might expect, and all of the other ones also. This conversion is quite difficult, which is why most languages do not provide such a tool suite. The reason for this is that shell scripts inherently rely on external programs, so the language isn't just a set of syntax, but a semantic pallet that includes the entire operating system. Perl 6 resolves this problem by providing a unique module for every existing operating system revision.

For example, if your shells script was written to run under BSD 4.2 Unix variant, Ultrix, then you would run the conversion tool like so:

  shell6 --system=unix.bsd.ultrix bofh.sh

Which would produce an output script called "bofh.pl". This script maintains a close compatibility with the legacy system by providing an execution environment that mirrors the expectations of the script. It will first boot a Perl 6 version of the 4.2 BSD kernel. This kernel is actually hand-written in Parrot Byte Code (PBC) giving it the blazing speed of an interpreted byte code, far out-performing the original Ultrix VAX-based systems, when run on modern hardware.

Next, Perl attempts to simulate the expected idiosyncrasies of these legacy systems. Because the 1980s pre-dated the development of the World Wide Web, there was a far greater dependency on communications protocols like UUCP, Usenet and FTP which are not nearly as widely used today. Thus, the shell6 translator also creates a "news spool" and establishes a "bang path" connection to other users of the service over which to communicate and share observations about the latest episode of Automan, Star Trek: The Next Generation or Babylon 5 with the simulated sense of urgency that only Perl 6's hyper operators can bring.


There are, however, some problems. There are still a number of simulated environments in which scripts do not behave as expected. In most of these cases, it is believed that the legacy scripts in question may depend on the ability to compile and run code in programming languages that start with the letter "C". For those cases, the next revision of shell6 intends to have a full parser (written in Perl 6 rules) for each of the common versions of these languages, but don't expect that to be released just yet, as there may be some concerns about embedded assembly that will require a more robust approach which simulates the entire hardware platform.

As the title of this article implied, there is also a team investigating the value of moving Perl 6 from the "P" namespace (let's face it, no real language has ever had a name that started with "P") to the more prestigious  "C" namespace as "Clamchowder", so you may wish to begin using that name in all of your Perl 6 communications with colleagues. Final debate has already begun, but stalled on the options of "Clamchowder," "ClamChowder," "Clam-Chowder," or "Clam_Chowder." There is a small contingent of "clmchdr" advocates who claim that Perl's name should be golfed for great justice, but nobody is taking them seriously.

Yes, today is April 1.

Perl 6 for finding non trivial powers less than 2 ^ 20

I recently noticed Terry Jones's blog post, "The eighty six non-trivial powers ≤ 2^20." I was immediately struck by how well this lent itself to being implemented in Perl 6, so I grabbed a handy Rakudo interpreter and translated it rather literally and then played with some slightly more idiomatic approaches. I still think I'm missing something with respect to the main loop, but I like what I came up with. The code is on github in my scripts repo.

In comparing the two (Terry's Python and my Perl), I really felt like this summed up the reason I want to code in Perl 6 more often. I work in Python for work, these days, and while it's a fine language with some very nice tools (django is especially nice), I frequently find myself wishing Rakudo were just a bit more production-ready.

So, without further ado, some thoughts about the two versions...

Code length is ignorable. Sure, the Perl is shorter in some areas and longer in others, but if that's interesting to you in terms of a mathematical problem, then there may be a problem.

However, in terms of clarity, I find two distinguishing features. One is strongly in Perl's favor, and one is mildly in Python's favor:

Pro Perl:

The largest thing I notice is that last block. It's much clearer to me that were'e concatenating the key with the value, after both are transformed in some way.

This reads a bit like, "print the formatted number in the key, followed by an equal followed by the values, joined internally with "^" and externally with commas."

Meanwhile the Python is something like, "print a formatted number, and equal and a formatted string where the number is in n and the formatted string is a comma joining a formatted pair of numbers joined with a "^" where the numbers are a and b where a and b are the tuples in powers indexed by n."

It's not that the code is *longer* that's the problem, but that it's far more disjoint. Even the indentation has to be complicated to suit the way they're mixing infix, functional and postfix grammatical elements.

Pro Python:

The declaration and use of powers is far cleaner in Python due to the handy defaultdict concept. This bypasses the defaulting that we do in Perl:

    (%powers{+$n} //= []).push([$a,$b]);

Beyond being punctuation-heavy, this is exactly the same underlying behavior as the Python. We default the bucket at $n to an empty list and then append the new item. The difference is that in Python the defaulting behavior is taken care of behind the scenes.

I say that this is minor because I understand this to be functionality that Rakudo Perl 6 has simply not implemented yet. You will eventually write:

    my Array %powers;
    ...
    %powers{$n}.push([$a, $b]);

very much like its Python counterpart.

Update: I'm told by the good folks on #perl6 that the right way to push onto a hash entry with a default empty array is:

    %powers.push: $n => [$a, $b];

Which is certainly cleaner than all of the  above, so woot!

Two free albums worth paying for

As readers of my essays know, I've been listening to free music recently. That's not to say that I participate in file sharing of otherwise for-pay music. Rather, I enjoy music which artists choose to share for free. I've even donated a bit or bought for-pay albums as a result.

This month, I've found two new items that really tickle my fancy. The first is Commence, the second free album from Paul Lisak & After the Ice, the first being It Happens All the Time.That album's clear and nearly hypnotic rifs and commanding vocals were my first taste of free music that I could actually love. Commence doesn't live up to this standard, but it's a solid rock/pop/alternative album which far surpasses the bulk of the free music I've been listening to of late. It's a bit bluesier than Happens which can definitely be nice if that's what you're in the mood for.

The second is the instrumental/ambient album, Shedneryan by Roger Subirana Mata whose work was covered in 2008 on Free Albums Galore (I've also linked to one of his CDs on Amazon, to the left, if you're not ad-blocking), an excellent blog for finding free music. This new album was released last month and I'm not sure that I agree with one poster to Jamendo who said that it was on-par with Tubular Bells, but it's an excellent atmospheric piece that I'd compare favorably with Tangerine Dream (especially the first track), Vas (the second track) and many of my other favorites. There's even what I think is a little bit of dark soundtrack ala Danny Elfman and John Massari thrown in for good measure.

Nuclear power: why is it so terrifying?

The earthquake triggers a massive
gas explosion.
Credit: Reuters

I'm reading accounts of the Fukushima Daiichi nuclear plant and its ongoing woes after Japan's 8.9 (now the USGS is saying 9.0) earthquake on the 11th. Now, this is a tragic event, to be sure, but I'm baffled by the laser-focus that this one event is getting. Power-generation of many sorts suffer catastrophic failure under these circumstances. Gas mains ignite, fuel tankers spill, oil is dumped into oceans and rivers. And yet, for some reason we're focusing on the one event that has yet to injure anyone. Sure, it might result in a deadly release of radiation that prevents a wide area from being re-built for the foreseeable future, but I get the impression that people are somehow equating this to some fundamental instability in nuclear power.

Keep in mind that this earthquake has literally removed entire towns. They'r gone. The buildings don't exist anymore, except as rubble, swept miles away. Under these circumstances, the low death toll (5000-10000, estimated) is a tribute to Japan's excellent preparedness. There's no one sitting a block away from the power plant wondering, "oh dear, is it going to affect me?" No, everyone is either dead or evacuated. Thousands are dead. Thousands more are missing. The fact that a nuclear power plant is suffering a failure is quite simply not changing the scope of this disaster.

But still, we continue to raise alarms and demand stricter regulations as a result of our irrational fear of what is clearly our safest form of power generation. We continue to put out gas fires and clean up oil spills at an ever-increasing rate, but after a 9.0 earthquake a nuclear plant is damaged? It must be time for stricter regulation!

Don't get me wrong. I'm all for strong safety surrounding nuclear power. I think Chernobyl teaches us what happens when basic safety technology is ignored. However, you don't then ratchet up the level of regulation every time a new worst-case scenario occurs. You perform reasonable risk assessments and regulate as a result of their findings when necessary. It's actually not very hard.

Python subprocess vs os.popen overhead

Let's say you're writing a Python program and you want to run an external command and read its output. The right thing to do used to be to:

 import os

 p = os.popen("command")

 output = p.read()

But there were a lot of ways to run programs depending on what kind of output you wanted to read (if any) what kind control you wanted (if any) and so on. Thus was born the subprocess module. In the current, 2.7 documentation for the os module, there's a note on popen:

Deprecated since version 2.6: This function is obsolete. Use the subprocess module. Check especially the Replacing Older Functions with the subprocess Module section.

Well, that's pretty definitive, right? Unfortunately, not so much.

Typically, process creation overhead doesn't matter a great deal. If a program needs to run another program, then the startup time involved in the creation of the process is probably an order of magnitude (often several) less than the time that the new program takes to do its work. So, you only typically care about process creation overhead when you're creating a very large number of parallel children.

Unfortunately, I work in the world of system monitoring, and in that world, creating a few hundred or thousand programs a second during peak times (subordinate monitoring tools) is not a rarity, and even when the load is much lighter, large amounts of process creation overhead isn't always ignorable. For example, if your system is doing a lot of IO, then large memory operations during process creation might reduce the amount of caching the system can do.

All of these factors lead me to test the subprocess module against os for a simple case: I want to run a process under a shell with standard output being captured. With the os module, my test looked like this:

 import os
 for i in range(10000):
    f = os.popen("exit 0")
    f.close()

With the subprocess module, my test looked like this:

 import subprocess
 for i in range(10000):
    f = subprocess.Popen("exit 0", shell=True)
    f.wait()

I timed the two scripts and came to a surprising conclusion: subprocess has about a 40% process creation overhead over os.popen! That's an awful lot of increase, so what could be going on? My next step was to use strace to determine what could be taking up that extra time. Here's a partial strace fo the subprocess example under Linux:

pipe([3, 4])                            = 0
fcntl(4, F_GETFD)                       = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
clone(...) = 3306
close(4)                                = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f358acc6000
read(3, "", 1048576)                    = 0
mremap(0x7f358acc6000, 1052672, 4096, MREMAP_MAYMOVE) = 0x7f358acc6000
close(3)                                = 0
munmap(0x7f358acc6000, 4096)            = 0
wait4(3306, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3306

The os.popen example had one less fcntl and an extra fstat, both of which are fairly light weight. The real culprit here are the mmap, mremap and munmap calls that subprocess is doing. Why are those there, I wondered. In looking at the subprocess code, it seems that these operations may be the result of thread creation which subprocess uses to manage reads and writes on subprocess inputs and outputs, but I'm not sure. What is clear is that the subprocess module is about 1,300 lines long while popen was a builtin supplied by the interpreter.

Conclusion

subprocess is a valiant attempt to make a complex snarl of library calls into a uniform tool. The problem is that process creation is one of the most fundamental operations that a language performs, and when a simple task like running an child process and reading its output becomes too heavy, a language suffers for it. Perhaps subprocess should be simplified and its convenience routines re-written as low-level operations that are optimized per-platform. Or perhaps os.popen should be undeprecated. After all, I'm willing to bet that managing fork, pipe and exec operations from Python will never be as low-impact as calling the C library popen(3) function.

Evaluating the "readability" of programming languages

Being a Perl programmer (among many other languages), I've often entered into the debate over what "readability" is. People who don't program in Perl and many who do, find it difficult to read. The problem is that those who know the language typically don't mean the same thing at all when they say, "Perl is unreadable," as those who don't know the language. Perl isn't unique in having readability concerns directed at it. C, C++, Lisp, FORTRAN, Haskell, Ruby, PHP, Bourne Shell and friends, AWK, and dozens of others have had the same complaints leveled against them, and in many cases, the same dichotomy exists between those who know the language and those who don't.

So, if you don't program in a language, what makes it "readable?" Presumably, it "reads" as if it were a language you know. For example, if many of your operators are English words, and you seek to minimize the amount of punctuation in a language, then people who don't know the language might feel more comfortable with it than if you re-purpose ever bit of punctuation that one particular extended keyboard symbol set provides (yes, I'm looking at you, APL). Now, once you learn that symbol set, of course, you have an entirely different problem on your hands: can you understand the code that you can now map to a sequence of operations in your mind?

It turns out that this latter question is where many users of Perl over the years have staked out a more sophisticated set of concerns. For example, the language provides no fundamental object model, just the tools with which to build one. So, when you see code that uses objects, you have to wonder, "what is this doing?" When you have to ask that, then I think it's fair to criticize code as "unreadable."

On the other hand, I've had a debate with someone recently about this snippit of Perl 6:

 1, 1, *+* ... *

This is the infinite Fibonacci sequence in Perl 6, and while it's an obscure looking thing to those who don't work with Perl 6 on a regular basis, to someone who knows the language, there is no ambiguity at all. You never find yourself wondering what the hidden mechanics are, here, because there are none. It's simply a series of numbers, generated by adding the previous two values together to get the next value.

Perl 6 has its problems with readability, to be sure, but until we have a very large base of programmers using it on a regular basis, I don't know that we'll have a clear handle on what those are. I think the adverbial tagging of expressions will either end up being a boon or a substantial hindrance to readability, for example. The ability to build mini-languages ala Lisp is extremely powerful, but to say that it's open to levels of abuse that could make the International Obfuscated C Programming Contest look like a poetry slam is an understatement of epic proportions. Will this be a problem in reality? Maybe.

Moving past Perl, however, I'd love to have a generic set of metrics to apply to any language so that we can understand what it is that we're talking about. I see there being 5:

First, a definition: a "symbol" is any sequence of 1 or more characters that have a defined meaning in the language. In Perl, for example, "my", "+" and "die" are all symbols. The name of a function is not a symbol because its meaning is not part of the definition of the language.

Granularity - This is a measure of code abstraction. High granularity means that it takes many symbols to represent a given concept while low granularity means that a smaller number of symbols is required. One measure of granularity is the number of symbols in an average piece of code vs. cyclomatic complexity, but I don't believe that this is sufficient in modern programming languages.

Density - A simple average of the number of characters (in whatever character set you like) per symbol the language uses.

Vocabulary - The number of symbols in the language.

Textiness - The ratio of symbols which are comprised of "letters" (which may vary in meaning by character set) to those comprised of non-letters or a mix.

Morphability - The ease with which a programmer may change the behavior of the language or invoke behavior which is ambiguous at compile-time. This includes everything from textual macros to operator overloading to Lisp-style macros to simple polymorphism. Every way in which a piece of code may have multiple valid meanings or the meaning might change depending on the behavior of previously evaluated code. It's important to understand that "if a then b else c" has two behaviors that depend on the value of a, but if we could expect this code to compute pi because of something I've done ahead of time, then it would demonstrate a high morphability.

These five metrics need more specific definitions. We need to understand what their domains are and how different languages map into those domains. I'll tackle that in a latter article...