Perl 6, Python, hyperoperators and list comprehensions

I use Python every day at work, and I do like the language. There are things about it that annoy me, but I don't think that's ever not been true of any language I've used. One of Python's best features is its list comprehensions. These short snippets of code can embody so much work that it often feels like Python is writing your code for you.

At night, I go home and work on Perl 6, the upcoming update to the decades-old programming language which adds features from nearly every programming language you've ever heard of (and some you haven't). The direct equivalent of the list comprehension in Perl 6 is the same as it was in Perl 5: map. Here's how you use map in Perl 6:

 map {$_ + 10}, (1,2,3,4)

This yields the same list as the Python:

 [ x + 10 for x in 1,2,3,4 ]

Ah, but the astute among you are noticing that the Perl uses a variable name that's always the same, thus making nested map statements painful due to the need to create temporary variable names manually. In Perl 5, this was true, but we can name those temporaries quite easily now:

 map { $^x + 10 }, (1,2,3,4)

Perl sees these temporaries from left to right and considers them positional parameters in the order that they appear to the current block (which is also a closure).

But Perl 6 gives us something more than map. In fact, map will be used much less frequently in Perl 6 because of hyperoperators. A hyperoperator is an operator that takes another operator as a parameter and augments its behavior. In Perl 6, hyperoperators can do this:

 (1,2,3,4) <<+>> 10


or the Unicode equivalent:


 (1,2,3,4) «+» 10

This takes the + operator and makes it work on the list given on the right side, adding the value on the left to each item and returning the newly created list of results. So, we never need to create a closure in order to ask Perl to do some particular binary operation on all elements of a list. Instead, we just pass the list, the operator and the right hand side to a hyperoperator and it does all the heavy lifting. We don't even need to see the temporary variable that's being used.

List comprehensions like Python's [ ... for ... ] and Perl's map are extremely valuable things for doing complex operations, but when what you really want is just to perform a simple operation on the elements of a list, hyperoperators give you what you need without the trappings you don't care about.

Note: There are actually multiple forms of hyperoperator depending on how "DWIMy" (Do What I Mean) you want it to be and on which of its arguments. See Synopsis 3's section on Hyperoperators for more.

5 things you can do with Lists in Perl 6, Python and Ruby

I think practical examples of doing the same sorts of tasks in different programming languages can be wonderful tools. Recently, an IT student in Poland named Konrad posted a followup on his blog to the 2007 Ruby blog, "5 things you can do with a Ruby array in one line (PLUS A FREE BONUS!!)" by drewolson. He updated this for Python. Of course, having worked with Perl 6 quite a lot recently (see my Google Buzz posts titled "Your daily dose of Perl 6"), I was compelled to do the same for that language. See below for the results. Notice that Konrad chose temporary variable names that were much shorter than drewolson's, so the visual comparison between Ruby and Pyhthon in terms of code size is somewhat unfair, but I'll go with the original names where I need temporaries, just to be fair to Ruby.

Summing elements


Here, the original Ruby example printed the result, but I've trimmed that out for consistency with the rest of the examples.


Ruby:

  my_array.inject(0){|sum,item| sum + item}

Python:

  sum(my_list)

Perl 6:

  [+] @my_array

Double every item

Ruby:

  my_array.map{|item| item*2 }

Python:

  [2 * x for x in my_list]

Perl 6:

  @my_array <<*>> 2

Finding all items that meet your criteria (such as being divisible by 3)

Ruby:

  my_array.find_all{|item| item % 3 == 0 }

Python:

  [x for x in my_list if x % 3 == 0]

Perl 6:

  grep {$^item %% 3}, @my_array

  # or...

  grep * %% 3, @my_array

Combine techniques

Ruby:

  my_array.find_all{|item| item % 3 == 0 }.inject(0){|sum,item| sum + item }

Python:

  sum(x for x in my_list if x % 3 == 0)

Perl 6:

  [+] grep * %% 3, @my_array

Sorting

For more information on how Perl 6 sorting works and what the *-autoclosure syntax that I've used above does, see carl's excellent Perl 6 Advent Calendar post.


Ruby:

  my_array.sort
  my_array.sort_by{|item| item*-1}

Python:

  sorted(my_list)
  sorted(my_list, reverse=True)

Perl 6:

  @my_array.sort;
  @my_array.sort: -*;

And there you have it. Enjoy the many choices you have in programming languages!

Lost: What the !@#& was going on?!

Lost's pilot episode pulled me in. It was one of the best television shows I'd ever seen. The writing was brilliant, the actors were not only excellent, but all clearly gave everything they had (with one or two exceptions, but I'm able to ignore than in an ensemble cast). By the second season, the show had ground to a halt, and I became bored. I'd also been burned by Alias, Abrams' previous creation, and was not eager to be strung along for season after season again. I stopped watching late in the second season, and didn't come back until the fifth. The fourth and fifth seasons definitely picked up the pace of the show, but I was left wondering how much of what was going on would be revealed. The show was clearly set on a very cosmic trajectory, and to play that out would risk alienating a large percentage of the viewship (no matter how you play out a cosmic ending, it always alienates someone who feels that your story conflicts with their beliefs).

So, it was with trepidation that I approached the sixth season, and in fact in the final two and a half hour movie, they never did play out the larger back-story. This post is aimed at exploring what was actually going on and whether, speculation aside, we had enough information to understand what it was that was going on. This will involve spoilers for the entire series. If you haven't watched the final seasons, I suggest you do, but go in expecting a non-reveal. I liked the last episode, but I felt the way I felt at the end of Donnie Darko: clearly someone knew where they wanted to go with this, but decided they didn't have the time, creative freedom or desire to follow through. In the case of DD, the story played out in supplemental materials on the Web. In the case of Lost, I think the creators simply don't want to explain what they feel they've sufficiently hinted at, for fear of losing the sense of mystery.

OK, so let's see if we can extract reason from this show. There is really only one unanswered question of merit: what is the island? The other questions ("what are the numbers," "why do pregnant women die, etc." are secondary to this central theme and may not have answers outside of fan speculation).

Writing a Perl 6 URI module

I wanted to write a parser of some sort using Perl 6's spiffy parser language otherwise known as "rules". This is the super-extended regular expression syntax that Perl 6's own parser is written in, and it's not just powerful, it's easy to use. In fact, it's so easy to use that almost all of my time writing a URI parser module was spent on other aspects of the code than the parser itself.

First off, some background. Perl 6 has a URI module already. However, it relies on a number of Perl built-in character classes to match things like digits and alphanumerics. In reality, the RFCs that define URIs are very precise, and there are different specifications depending on what you need. So, I decided to re-write the module with a pluggable parser so that you could give a regular, modern URI and have it parse correctly, but you could also ask for special "IRI" parsing on an internationalized URI and the right thing would happen there. I even went so far as to bring in an older version of the specification as a legacy mode.

The current state of the Perl 6 parser and runtime called Rakudo is actually fairly solid for a pre-release implementation of such a complex language spec. There are some gaping holes, but they were all relatively easy to work around. Some of these included overly aggressive list-flattening, some operators that were broken at the time I wrote this code and the big one: named rules only work as a stand-alone grammer with a specific entry-point called TOP.

I worked around all of these issues and have, so far, been able to parse basic URIs according to RFC 3986. Here's a sample of what a Perl grammar for URIs looks like:

    token URI {
        ':' [ '?' ]? [ '#' ]?
    }

Here you can see most of the basics: "token" introduces a single expression within the grammar. It calls out to other tokens by enclosing their names in angle-brackets. Literal sequences are enclosed in single-quotes and sub-expressions can be enclosed in square-brackets with regular expression-like repetition counts such as ? for 0 or 1 matches.

In order to have a pluggable interface, I needed a class capable of providing me with two things for each grammar: the grammar itself and a set of routines which would tell me how to find the resulting URI elements in the match data. For this I defined an interface using Perl 6's roles:

  role URI::Specification {

      method parser() { ... }

      method scheme_path() { ... }

      # ... other _path methods here...

  }

Those ellipses are literal. They cause the methods to be required for any class composed with this role, but do not define any functionality themselves.

Each parser is then defined as:

  class URI::rfc3896 does URI::Specification {

      grammar URI::rfc3896::spec {

          token TOP { }

          # RFC definition of URI goes here.

      }

      method parser() { return ::URI::rfc3896::spec }

      method scheme_path() {

          gather do { take }

      }

      # And so on ...

  }

That's it. The only really funky bit here is the gather/take code in the scheme_path. That's the way Perl 6 defines a coroutine-like interface. The paths define how we traverse the match object to find match results. So, for example, the "scheme" (the "http" in "http:/www.example.com/") can only be matched in the URI rule's scheme sub-rule. Some URI elements, however, such as authority (the host name and port - possibly username as well) can be matched multiple ways, so these routines might return multiple lists of subrule names to traverse. I would have simply returned a list of lists, but Perl 6's parameter passing is very complex and currently some of the specification is not yet implemented. Right now, this manifests as overly aggressive list flattening when returning them from a subroutine or method.

This is why I used coroutines to return each of the sub-lists, one call at a time.

I'll continue to post new updates as my URI module nears readiness. For now, it's just awaiting some love on the other parsers, and I think it'll be ready to go.

Icons from Wikimedia Commons

I'm often lacking for exactly the right Icon for Web work, and then I remember Wikimedia Commons. For a number of open-licensed UIs I've used the Wikimedia images to help create a look that implies I spent an awful lot of time and energy. These icons vary in quality because they were uploaded from different sources, but you can get some of the best icons available from this site (along with, of course, the best freely licensed photographs and other media). Commons is a gem, and if you don't use it, you're probably missing out.

The Crystal Icons from KDE were uploaded as PNG, but the original SVG icons can be acquired from their original site. These are very clean and plastic-feeling icons for everything from the power button you see on the left to various arrows to hands to gears. Because they were uploaded in fairly high resolution, using the PNG versions isn't all bad, but you'll still get a better final result if you work from the SVG.

There are also many icons for specific applications. You can use these when writing reviews or otherwise mentioning the applications in quesiton. Many of the application icons come from the Crystal set as above, but some are directly lifted from open source projects, such as the Gaim (now pidgin) icon you see to the left.

Other icons include arrows, globes, hands and many other cateogries. You can browse all of the categories from Wikimedia's "icons by subject" page.

To use these icons, I recommend always selecting SVG format images and then using Inkscape . Load an SVG icon up in Inkscape and select File, Export Bitmap... to save it as a PNG file, setting the height and width to exactly what you want. For now, PNG is the way Web browsers deal with icons the best, though in the future, SVG will be taking over that role as older browsers fade away.

Fire alarms: Among the worst UIs in history

So a fire alarm just went off in my office building. Here's what you hear:

"*beep* Your attention please. There has been a report of an emergency in this building. If your floor evacuation tone sounds after this message, please leave the building. *beep* Your Attention please. ..."

OK, am I the only person who sees the problem with that announcement? Is the second *beep* just a repeat of the message or am I supposed to read that as "my floor's evacuation tone?" What does my floor's signal tone sound like? Is it different from other floors? Does our floor's signal tone rule relative to the suckage of other floors' signal tones?

So here's an idea: Instead of a useless recorded message, have the damned thing announce the floor that the problem exists on (or floors). I'd really love to hear, "Floors 8 through 10 should consider jumping out the windows because a fire is quickly sucking the oxygen out of the stair wells!" Now that would be a warning I could do something with.

Test Wave Post

I'm testing out Google Wave integration. Please, let me know what you think in the Wave, below.