Wikileaks, Assange and Stross

Charles Stross has a great bit on his blog about Julian Assange. He is trending toward irrational exuberance, here, but I have to agree on the core point: anyone willing to publish these documents (Google for "Wikileaks" if you're not aware of the situation) is a kind of hero. It used to be that you went to the papers, but the papers are gasping for air and can't jeopardize their relationship with the government. Is there anything all that awful in the leaks? Not really, but it does make it harder, as Stross points out, for any group larger than 5-10 people to do anything they're going to be shocked or embarrassed by when it hits the Interwebs. This, to my thinking, is a good thing.

The counter-claim I've heard is that this makes it harder for foreign governments to trust us with their secrets (one example being given relates to Yemen cooperating over Al Qaeda raids, a reasonable concern). However, this is a reason to keep such secrets ... well, secret. Keeping them on file in a massive government bureaucracy isn't that, and the fact that someone then leaks that information is on the head of the person who does so, IMHO. Interestingly, we're not even talking about that person. We're just talking about the guy who did the actual publishing, which seems odd. He's just one guy with a Web site and no access to internal U.S. government information. Why not just prevent him from getting the information in the first place?

On a less wholesome note, I'll point out that Stross is a bit off on his analysis of the rape charge. While this definitely looks like a case of politically-motivated and kind of whacked-out revenge rather than a real rape claim, the charge being leveled against Assange isn't that he slept with another woman after the claimant, but that he had unprotected sex. This, apparently, under Swedish law can constitute rape even if the sex is consensual. I'm a bit shocked by this, but none the less, this is the claim I read on Wikipedia which is citing a Sydney Morning Herald piece about the rape charges. The part of the case that seems weird, however, is that the Sweedish authorities responded to Assange's willingness to meet with them at the Sweedish embassy or Scotland Yard with a request to Interpol and the EU for extradition. That's going way over the top, it would seem, given that their request was for interrogation, not trial.

Anyway, the wonderful thing about the Internet is: someone's going to pop up and offer the same service, even if Assange is buried for his role in this. It's awful to see him go through what even the women in question admit are rape charges over consensual sex, but in the end, I think his fears that he'll be handed over to the U.S. and harmed are unfounded... at least, I hope that will be the case. I want to think we haven't sunk that far...

The I Can't Take It Anymore Diet

Several years ago I was diagnosed with severe sleep apnea with a minimum blood oxygen saturation of around 70% which is to say, "you're dying because your brain is starving." I was placed on a CPAP machine and was immediately relieved of almost all symptoms. Problem solved, right? Well, no.

Life on a CPAP machine isn't fun. You have a bulky thing that you have to carry around anywhere you intend to sleep and anyone who intends to sleep with you has to deal with your having medical equipment strapped to your face and making airflow noises all night long. Mind you, it's much quieter than most apnea sufferers themselves, and far less distressing to hear than a pause in breathing followed by a loud snoring, choking intake of breath, but it's not exactly sexy.

Those were just cosmetic/lifestyle issues though, and I could have lived with that. The real problem was that I found the thing painful to wear and while it gave me a better night's sleep, it also caused me to get less of it, due to the difficulty of falling asleep and plenty of events where I would wake up with it pulled partially off.

So, I decided to give it up. That's not a small decision. I essentially decided to risk injury to my brain and/or death, so I needed a plan. Most apnea sufferers, myself included, have their breathing problems as a result of their weight. Over a certain, individual-specific weight threshold, your airway just doesn't work the way it was designed to. In my case, I was 265 lbs. and the threshold was about 230 lbs.

This made a diet a fairly obvious solution. (click the title for the rest of the story)

Something new every day: Bourne Shell variables

I've worked with the Unix Operating System and its variants since the late 1980s. I've worked with the Bourne Again Shell (bash) since the early 1990s. And yet today I learned something new about variable expansion. In the startup scripts for a source code indexing system called OpenGrok, I found this gem:

somecommand ${PROG:+-c} ${PROG}

Now, I know that ${FOO-bar} will be replaced with the value of $FOO if it is currently set or "bar" if it's not. That much I learned many years ago, but this usage of "+" was new to me. After some testing, I found that "+" substitutes the following text if and only if the variable is set, otherwise it substitutes nothing. Thus if $PROG were set to "foo", the above text would execute:

somecommand -c foo

But if $PROG were not set, then somecommand would be run with no arguments at all. Very slick!

How I managed to go over 20 years without learning that, I'm unsure (then again, perhaps I've learned and forgotten it...)

A Google App Engine failure

Long ago, I wrote a Perl script that generates random names for my roleplaying games. It's a simple thing, but it can take input lists from any language and spit out similar-sounding made-up names. It's a powerful, but simple tool, and it seemed a natural fit for my first exploration of Google App Engine. Sadly, it didn't work out that way, and I thought it might serve as a useful caution to others who might plan the same sort of work.

The fundamental problem is that my app is IO-hungry. It reads in the entire source list every time someone asks for a made-up word, crunches it down into first-parts, mid-parts and end-parts (2-3 letter segments which are rooted at the beginning or end of the word or neither). We then sort the lists of parts according to frequency of occurrence and perform a weighted, random pick of a first part, then each subsequent part is chosen in the same way, but from a subset of all of the parts, which overlaps the previous segment. The combination of weighted choice and overlapping leads to words which tend to be pronounceable in the source language of the input list.

This process of reading and processing all of the words every time wasn't something I was going to be able to do in Google App Engine, however, since costs are associated with resources consumption. So, I set out to store the pre-digested versions of the input lists as sorted word-segments in the Google App Engine datastore. This is where my problems began. While it's entirely possible to store the data this way, what I found was that my need to access so many records from the database as I performed my random walk down the lists of word-parts left GAA gasping for breath. In practical terms, I'd created the world's slowest tool for producing babble. Of this, I'm sure my mother feels proud.

Frankly, I'm not sure what I can do about this. GAA just doesn't seem to have been designed for this sort of thing. A shame, really. Of course, I could pre-compute a queue of results for each source namelist and keep re-populating them with a periodic job, but that really seems like a cheesy way to solve a problem that takes a few seconds for my original Perl script.

How not to open source a roleplaying game

After the phenomenal success of open source software development in the 1990s, someone at Wizards of the Coast decided to try to follow this model for their roleplaying game, Dungeons & Dragons, purchased in 1997 when they acquired TSR. Specifically, they decided to publish a cut-down set of rules called the SRD, which represented the core of what it takes to publish a Dungeons & Dragons-compatible game. The idea at the time, which worked well, was to encourage what marketing people call an "ecosystem" of publishers who made everything from full roleplaying game systems to source books for D&D.

All of this made sense, but Wizards had opened the genie's bottle, and there was no putting it back. As long as they continued to encourage their new ecosystem, they were assured of their place as king of the mountain. However, in 2008, they announced that they would end support for D&D version 3.5 and begin publishing version 4.0. This new version would not be available to third party publishers for creating their own games, at least not at first (they have since published a new, more restrictive set of rules called the GSL).

At the same time, a small publisher named Paizo had been licensed the rights to publish Dragon and Dungeon magazines. These two magazines were in decline at the time, but Paizo manged gain the enthusiasm of the roleplaying community by dipping into the well of nostalgia that many players had for the game. The printed updates to classic adventures in Dungeon and breathed new life into old stapes of Dragon such as the "Ecology of the..." series. They also fanned the flame of the original Dungeons & Dragons setting: Greyhawk. Their most impressive accomplishment was the enthusiasm generated by their "Adventure Paths." These 12-issue serial adventures were published in Dungeon magazine and the first was then collected as a hardcover book. Overall Paizo did quite a bit of cheerleading for Wizards and in 2007 they received their reward: notice that their licenses to all Dungeons & Dragons products were being revoked so that Wizards could develop a stand-alone Web site to replace the print magazines in coordination with the launch of the 4.0 edition.

Of course, the consensus at the time was that Paizo was doomed, but Wizards' Open Gaming License for the 3.5 edition was their way out. They immediately converted all existing Dungeon and Dragon subscribers over to new products based on the Open Gaming License. In less time than most publishers take to decide to take on a project, Paizo had their own campaign setting for D&D 3.5, published under the OGL along with a new adventure path, Rise of the Runelords. However, the OGL also allowed for full 3.5-compatible systems, and that was Paizo's next step. Over the course of the next two adventure paths that they published, Paizo continued to work on their variant system: the Pathfinder RPG.

Today, with several adventure paths published under their new system and a constant stream of supplement books published for their world of Golarion, one has to wonder if Wizards of the Coast is feeling burned. The popularity of the Pathfinder RPG hasn't reached the level of D&D, but it continues to build a dedicated base of adherents and is frequently referred to as "D&D 3.75."

So should Wizards not have created their "ecosystem?" Of course they should, it was a shot in the arm to what many declared a dead product. What they should not have done is assume that because they wanted to move on with a new edition that the industry would either follow or contentedly watch their businesses crumble. Wizards should have built support for 4.0 among their publishers and then eased into it without yanking the rug out from under Paizo. This would have resulted, at the very least, in reducing the number of long-term players that walked away from their version of the game.

Google's WEBP image format and World of Warcraft

I got my hands on Google's new image file format today and started testing it on some images I had lying around. I specifically wanted to see how it did with bad JPEGs of rendered sceenes. Hard edges and re-processed artifacts are something that JPEG traditionally handles very poorly, so there's some real, practical benefit to having a new format that can do these things well.

I took 9.5MB of input screenshots from my library and got 3.9MB of images out at 75% quality. Now, 75% in JPEG is pretty poor, but these don't look all that bad at all. At least not worse overall than the input (WoW compresses its screenshots pretty heavily).

More interestingly, however, here are some visual comparisons. All of these images are PNG format, converted either from the original JPEG input or from the converted WEBP at the given resolution.

Original screenshot (cropped region saved as PNG from original JPEG). Full file size: 541KB.
This is the 75% quality version of the same image as WEBP. Full file size: 257KB.
And this is the same WEBP conversion again, this time at 85%. Full file size: 347KB.
A second original image, again cropped and saved as PNG from a JPEG original screenshot. Full file size: 189KB. (re-compressed at 75% quality as JPEG, it was 132KB).
A 75% quality WEBP conversion. Full file size: 72KB.
An 85% quality WEBP conversion. Full file size: 109KB.

The lessons I learned from this are:

• WEBP does have some visual loss, even at 85% when it comes to highly saturated color, especially red.
• The file size drop is quite dramatic, even over a re-compressed JPEG at a lower quality.
• Overall, the look of the WEBP files is impressively smooth.
• I noticed (not in the samples, above) that splotchy regions of similar color, but varying value were sometimes flattened to the point that information was clearly being thrown away. At 75% quality this was a substantial change, but at 85% quality, it was hardly noticeable.

I really look forward to this format being available in the majority of browsers, so I can safely start using it!

Quatermass poster

While I was sick, recently, I found myself watching the 1979 British TV series, "Quatermass," which is a 4-episode mini-series that follows the exploits of an old 1950s and 1960s character that was a sort of prototype for Doctor Who. This series follows him through a new adventure where young people are either becoming ultra-violent (resulting in a breakdown of the inner-cities all over the world) or they're leaving the cities to wander under the name, "Planet People." These Planet People are basically hippies who are convinced that they'll be taken up to another, unspoiled planet by benevolent aliens. That's the ground-work that the first two episodes lay out. Oddly, I loved the story, but hated the series. I found it slow and exceedingly dated. The cold war tension is a bit overwrought (but then in 1979 that wasn't surprising) and the females in the show all have one of two reactions to the hippies: they fall in line, often without on-screen comment, or they go into hysterics. There's only one female character who responds sanely the the events and even she is clearly not very stable. Sad, really.

The most damning point, though, is that the narrative never really takes over. We're never quite sure why the Americans are deferring to our hero, nor why he's the only one who can think of the ultimate solution, which isn't, in fact, very clever. It's not clear why the hippies scare people so badly, nor why it takes quite so long to come to some of the conclusions that people reach. In effect, the series is a long string of events, some of which are painfully boring, and many of which do not appear to affect the overall plot (especially when so many characters that are established with painful deliberation are killed off-screen).

Anyway, as a 2-hour movie, I think I'd enjoy a modern re-make. As 4, hour-long episodes, it was torturous. But the easily recognized professor and the oft-repeated tag-line of the hippies, "soon," gave me the idea for the poster below (click the title to go on if you're seeing this on the front-page), which I'm threatening to turn into a button and wear to conventions. Enjoy!

The Great Humor Friendly Spam Is Translate!

I'm having far too much fun reading my spam box via Google Translate. If you haven't tried this game (which I'm called "The Great Humor Friendly Spam Is Translate!"), then you really need to give it a shot. In Gmail, click on your spam folder and then click on any message whose subject is in another language. At the top of the message, you should see a "Translate" link above the body. Click the translate link and read until you stop being able to breathe.

For example, here are some gems from my inbox:

From the Chinese:

why sales staff who are always working attitude, do not want more input?

From a Russian message describing the generic plot of all disaster movies:

It all begins with small problems, then all these problems
noticeable on the Congress is going to scientists and government
and concludes that the planet Earth (town, country, continent)
remain there for long, and starts fighting with the elements
successful or not.

From a Japanese site that appears to be offering dating and/or romantic advice:

I send this email to members of the audience to applicants capped.
Cross your fingers unsubscribe, thank you from the URL at the end

A Spanish ad for spam-based marketing suggests this nugget of wisdom:

GHOSTS ARE NOT ALWAYS GIVE YOU SUPPORT

This came as a shock to me, but I'm slowly coming to terms with the idea that I need to approach the living for my tech support needs.

Stowe Vermont: What to do, where to go

The falls near Stowe's town center.

I just spent 3 days in Stowe, Vermont and I have to say, it was the best vacation since San Diego for me. Comparing it to the last time I was in Stowe, I have to say that it was a bit more built-up, which isn't all good, but the primary features that attracted me are still very much in force: beautiful mountains, great places to walk and a decent selection of food and relaxation.

Bike Trail

There is a long bike trail, accessible from many points throughout the town, and I recommend going for walks down it as often as you can. It's ideal day or night, and parallels a beautiful stream. In the fall, this is probably the most "Vermonty" you're going to get.

On the Mountain

The view from Mt. Mansfield.

We went to Spruce Peak Lodge and took the Gondola up Mt. Mansfield. This turns out to have been a high-point of the trip. The bar at the top is relatively small, and I didn't get food, so I have no idea if it would have been worth it, but just getting drinks and looking out over 70 miles of mountains (you can just see the New Hampshire Presidential Range in the distance on the horizon in the center of the picture to the right) was mind-blowing. You feel as if you're being let in on a secret, here. It's just too amazing to be the product of a $25 lift ride. We decided to walk down the mountain, which was interesting. If you do this, I have to caution: take breaks and don't rush. Also look for and avoid longer, wider grass. These are wet areas that will be very dangerous to walk on. I'd also ask the operators at the top what trails are best to walk down. You can also continue up from this point, which will take you on a winding path to the actual summit of Mt. Mansfield, which I'm told is even more impressive. We were ill-prepared, but I'll definitely do that hike next time!

Gracie's

We went to Gracie's for dinner, which happened to be right next door to our inn (Arbor Inn, a very small, and really nice little place). This is a really fabulous little place to get dinner, and lacks much of the pretense of many of the newer, touristy places in Stowe. I had a wonderful stuffed chicken breast, but the highlight of the night was the beef tenderloin tips stroganoff, which was truly fabulous. I believe I read that they welcome well-behaved dogs, but I didn't see any while I was there. Definitely recommend the place to dog-lovers, though, as the dog kitsch and puns never stop!

Waterfall

The falls off of Mountain Rd. (108) in Stowe.

The morning before we went to the mountain, we went for a much less demanding hike: a simple, 10 minute trek from the road gets you to one of the most beautiful little waterfalls I've ever seen. There are 3 main vantage points of increasing difficulty to reach. The first was the most awe-inspiring, and it's the easiest. This was a view from the top of the falls, down into the mini-canyon that has been formed by the water flowing through a large chunk of granite. This canyon is maybe 2-3 feet wide and 15 feet deep. I recommend standing well back from the edge, as there's nothing that would stop you from becoming stuck if you fell into the swiftly-flowing water. Next up, you can walk down a steep path to reach the edge of the falls at the bottom. From here you can see the actual falls and the pool they drain into as well as glimpse some of the river as it flows away down a rocky bed. Finally, you can descend the rough trail (very rough) to the river bed, below. You have to climb down some rocks, but it's short and requires no equipment (4-6 feet depending on where you do it). This brings you to a point where you can stand out in the middle of the stream bed (depending on water flow, I'd imagine) and look up at the falls. On the side, you are bracketed by a high cliff that seems to indicate that there's some substantial erosion still continuing, here. It looks like people continue on from here (lots of cairns down-stream), but we didn't venture any further down-stream.

Other Places to Eat

When we arrived, we went to The Shed, which isn't bad, but didn't excite either of us, and they over-cooked my burger (which was otherwise fine). We also visited Sunset Grille (apparently unrelated to the Boston establishment of the same name) and found it to be surprisingly local feeling. We even got a few hairy eyeballs from the locals at the bar, which made us feel just unwelcome enough ;-)

Of course, you can't get to Stowe, at least from the south, without a quick stop at the Cold Hollow Cider Mill. It's a tourist trap, but one with the best cider donuts I've ever tasted. Not too sweet, not to large. Just a good donut with a hint of cider in the mix. We also grabbed a couple gallons of cider to take home on the way out and shared them with friends when we got back.

Conclusion

If you go to Stowe, I'd suggest staying at the Arbor Inn if you don't mind a pure bed & breakfast, but if you want more developed accommodations, you might be able to get a good deal with one of the larger resorts in the area. I hear that Top Notch is having problems, so unless you can get a crazy deal there, I might avoid it until those get settled out, but I did hear good things about Stowehof.

Other than that, don't make the mistake of over-scheduling. Do whatever strikes your fancy, but don't plan more than 2-3 events if you're up there for an extended weekend. It's good to just play it by ear.

Hardest Trivia Questions Ever, Part Two

Since my list of the hardest trivia questions ever has become one of the most popular blog posts I've ever done, I thought it was time to do a sequel. Once again these are culled from Wikipedia's "Did You Know" front-page feature, this time during the month of September, 2010. There are many more of these per day, now, so do take a look at the archives.

(Also see the follow-on: Part 3)

15 September 2010

Q: The German battleship ____ (pictured) engaged the Russian battleship Slava during the Battle of the Gulf of Riga during World War I
A: SMS Braunschweig

Ubuntu 10.04: Pulseaudio, nfs, automounter and my dismay

I upgraded to Ubuntu 10.04 last night. Ouch.

My first problem was that, suddenly, the automounter was going totally berserk. It would randomly mount/unmount all of my NFS filesystems. I couldn't do anything.

This morning, I found the solution to that problem, but that only served to further confuse me: pulseaudio was freaking out over not being able to create a file under ~/.pulse. I deleted that directory and suddenly my automounter worked again... OK, that's spooky. I'm assuming that pulseaudio was loading audio drivers on startup and the fact that it kept re-starting meant that the module interface was constantly being banged around, resulting in poor service for any calls to loaded drivers. Surprising my system was stable at all, if that's the case.

Anyway now I'm finding lots of horrible little problems:

• Rythmbox used to hide itself when you clicked on its notification icon... no longer
• Pidgin seems to have been put into tabbed window mode by default which is painful
• Evolution seems to squirrel your password away and use it for the password to your keychain... this means that when your password changes, you have to remember the old one in order to access your saved passwords. It seems like this should at least come with a big red button labeled ("just start over") so I can stop having the password dialog box pop up.
• System services are no longer managed by the standard init.d script interface. Joy, a non-standard init. Just what I needed.
• Process names are getting more and more absurd. The fact that something is running on my system called rtkit is deeply wrong. I nearly had a heart attack before I googled for it (of course, it's not alone: liboobs was one I saw scroll by as I was installing)

Overall, 10.04 does not impress me. It seems to have been put together as a way to prepare for future updates rather than as a stable starting point for long term support.

PS: Another issue just cropped up: outbound mail from evolution seems to have been silently failing all day. I finally realized that people weren't getting my mail and went to check the settings. Sigh.

The public domain and why it's important

I've been thinking a lot about the public domain ever since I wrote my proposal for a reform of copyright law in the United States. In that proposal I discussed the value of allowing works to expire (I set a time frame of 30 years, but that number is arbitrary; the important element is the expiration), but I continue to hear corporations who own copyrights "explain" how their business will be in ruin, should their works expire, fueling the continued extension of copyright terms each time they are about to expire. But, this fails to explain why the public domain was considered valuable enough to enrich with expired works in the first place. What is it that we, the public and the creators of new works derive from copyright expiration and the public domain?

When we discuss the public domain today, it can be difficult to understand its true value because so few works expire today. However, the works which have already expired have had a deep impact on our modern culture. One need look no further than Walt Disney Corporation's success in adapting public domain works such as Snow White and the Seven Dwarfs, Cinderella, Pinocchio, The Hunchback of Notre Dame, Alice in Wonderland, and The Jungle Book. How is it, then, that we continue to argue that copyright terms must be extended in order to protect our cultural heritage? Is it possible that such examples are just outliers and the public domain doesn't actually benefit the public and our culture? Hardly. In order to illustrate that point, let me provide a few examples:

It's likely impossible to fully account for the impact of William Shakespeare in  modern culture. Hamlet, alone, has spawned dozens of adaptations for film and television, not to mention its continued performances and adaptations on stage. Film alone accounts for over fifty adaptations of the play! Overall, there are over 400 adaptations of Shakespeare's plays, just in film.

Since the copyright expired in 1956 there have been over 40 adaptations of The Wonderful Wizard of Oz and related books and characters in film, television and stage.

The script for Braveheart was based mainly on Blind Harry's 15th century epic poem, The Actes and Deidis of the Illustre and Vallyeant Campioun Schir William Wallace.

The 1985 film, Ran, by Akira Kurosawa is based on legends of the daimyo Mōri Motonari, as well as on the Shakespearean tragedy King Lear.

The 1959 film, Ben-Hur, was the third film version of Lew Wallace's 1880 novel Ben-Hur: A Tale of the Christ, though I have not been able to determine if, in fact, the novel's copyright had expired by 1959, it does seem likely that it had.

The Wizard of Oz, Braveheart, Ran and Ben-Hur are all listed in the Internet Movie Database top 250 films of all time. How could it be that our popular culture could be so influenced from the public domain and yet we continue to argue that enriching the public domain by allowing works to expire is harmful?


The simple fact is that corporations fear losing any source of income, regardless of how much they might ultimately benefit from a copyright system that enriches the pool of works upon which they might draw. This is understandable, but should not be the basis on which we form our laws.

TV credenza

I've been looking at TV stands and credenzas for some time now, but I didn't see anything that met my needs and was well designed. In the end, I was going to give up and build something... then I found the Prepac TV Console. At first I thought it was kind of nice looking. Height was a big thing for me. I bought a conservatively small TV, and most stands are fine for 50" behemoths, but they're too short for what I have. At 26.75" this stand was a perfect height, and the shelves looked well placed.

I had already decided to buy it when I clicked on the image to see a close-up. That's when my mind was blown. The swing-out DVD/CD racks are beautiful and the holes for cables in the back aren't holes... they're notches cut into opposing, sliding panels that let you get equipment in and out easily from the back (also affording you trivial cabling access).

I've seen a ton of TV stands over the past few months and this has to be the best one I've laid eyes on, hands-down. What's more, for the price, it's a steal!

Scott Pilgrim: What comic book movies should be

I saw Scott Pilgrim vs. The World with friends last night. I had not realized, when I saw it, that it was based on a graphic novel, but it makes perfect sense. The film is as visually creative as Tim Burton's recent Alice in Wonderland but with a Wayne's World sensibility. The first image you see is an 8-bit video game-style rendering of the Universal Pictures logo. This is accompanied by an appropriately retro version of the usual signature theme. From this point on, the movie firmly establishes itself as a movie / video game hybrid, and I expect that audiences will fall into two camps: those who are aware of video game (specifically console and hand-held video game) culture who will enjoy the humor and visuals and those who are not and won't.

I can't say enough about the work they've put in to layering a video game world over the movie. I'll likely have to watch the movie again just to pick up on some of the touches I missed, but keep an eye on out-of-focus backgrounds. These are often fully rendered and the subtle touches are just as much a part of the story as the costuming and makeup.

The love story is essentially ignorable. Boy meets girl, girl has sketchy past, boy sees through all that, love springs eternal. It's nothing you haven't seen before. On the other hand, the secondary relationships are absolutely priceless. Scott's chaste relationship with an underage girl, his gay roommate with whom he shares a bed, and his cartoon-thin band mates are the real fulcrums of this story of a young man who hasn't yet given up being a boy.

Is it all good? No. What's entertaining about the film is its constant string of humorous twists on the culture of the generation that was born in the late 1980s to early 1990s. Some of the references are spot-on. Some of them miss the mark or just don't play well to the audience (I'm not in the target demographic, however, so there might be a very different mix of humor that works for people who are). The film walks a fine line between goofy spoof and cutting satire and sometimes comes up short on both sides. Still, the vast majority of movie works well and it's a rare movie that attempts as much as Scott Pilgrim.

I refer the frequent reader of my blog back to The best days of cinema were... 2009? where I described why I believe that creativity and quality story telling are alive and well in modern movie making. I'll happily add Scott Pilgrim vs. The World to the list of films that make my point.

Parrot and PIR: The best tool you don't use

For over 10 years now, many eyes have been on Perl 6, watching and waiting for it to be stable enough to use. Its promise is vast. Powerful "rules" will transform complicates parsing tasks into trivial libraries; A multi-method OO system like Common Lisp, but without the syntactic hurdles of learning to live with Lisp, a typed dynamic language that doesn't tie you down to type management. All of these features seem like the white whale of Melville's story, and yet they all exist today in a usable and stable form. Why doesn't anyone know about this? Well, I'm here to change that.

Parrot, the virtual machine first intended for Perl 6 and developed in parallel with that project, has the underlying mechanisms for everything Perl 6 wants to accomplish. It even has a pseudo-Perl 6 language called NQP (Not Quite Perl) for writing rules and other Perl 6-like functionality. But one thing slowing people down, to date, has been the lack of a coherent tutorial on using Parrot as a general purpose programming tool. Enter Parrot Babysteps. This Web tutorial (which is also a Github project) aims to teach people to program with Parrot. Its examples range from the typical and trivial, "Hello world" to a star catalog manager.

Parrot code is anything but beautiful. The language is modeled on assembly and is designed to be easily transformed into machine code via a JIT compiler, so don't expect it to be full of syntactic sugar like Perl. On the other hand, the combination of JIT compilation and high level constructs gives you the kind of power and performance that even Java is hesitant to offer.

Zen Alarm Clock

Zen Alarm Clock... it sounds like some kind of scam, doesn't it? Some New Age, crystal-based frob that's supposed to make you wake up.

But I bought my first Zen Alarm Clock back in ... oh it had to be '96 or so. I was having a hard time waking up, and no alarm clock worked. I can sleep through a jack hammer, and have, literally. Someone recommended this clock and I tried it out, even though it cost $100, which seemed like a ton for an alarm clock. The difference was staggering. Instead of doing something loud and obnoxious enough to force you to wake up immediately, it uses a long series of increasingly frequent chimes, none of which on their own would wake you.

By the time you become aware that it's going off, you've been gradually waking up for a good 10 minutes or so, which means that you don't feel like you're still asleep, being propelled only by the adrenaline release triggered by a loud noise. Instead you're just "up". If you put the clock on the other side of the room, that final act of getting out of bed and walking over to turn it off is usually sufficient, even if you're sleep deprived, to wake you up thoroughly.

My old clock is now breaking down after about 15 years, and it's time to upgrade to the digital version. I just purchased it today, and hope to get it in the mail in a few days. Can't wait.

Contest: Primes with an unusual number of a digit

I'd like to propose a quick contest. Here are the rules:

1. You must submit your entry by September 2, 2010, 23:59 (sorry, I can't wait until 2011, the next prime year)
2. Your submission must be a sequence of decimal digits.
3. The number that these represent must be prime (which make up (in order) a prime number. You may use a test like Rabin-Miller, which will be the method used to validate the submission. Note that it is therefore possible for the submission to fail such a test, even if it passed for the sumitter.
4. The number must be composed of at least either 155 decimal digits or 512 bits (either will do).
5. The winner will be the number which has the highest ratio of any one digit. For example, "23" has a 50% ratio of the digit "2" while 1011 has a 75% ratio of "1"
6. You must send your entries to essay-contest@ajs.com

As an example, here is a number:

1074934746  0579280428  5352135601  3625508347  1908730962  4560090445  3404800574  5211453514  0988380000  3920392507  2147060801  0391490408  0059833120  1609426475  4887127734  07857

which has 31 zeros for a ratio of 20%. Obviously, your submission should seek to beat at least this relatively low number.

You can submit as many times as you like, but in order to avoid making me unhappy with you, I suggest waiting until the last minute (or whenever you decide to stop searching) and submit the best result you have by then.

The winner will get a lifetime subscription to this blog and their name prominently featured in the article in which the number is published.

For fun, I'll be writing my own solver in Rakudo Star Perl 6. It won't be very efficient, so I expect it to get seriously trounced, but I'll do it for the fun. Even if I find the best number, I'll publish the best submission from someone else.

The best days of cinema were ... 2008?

We've all heard it. The claim goes that, back in the day, they made movies like  North by Northwest, Some Like It Hot and Ben-Hur, and that was just one year! Now we get sequels to movies that sucked, which themselves suck and an endless stream of romcoms that don't even rise to the level of sucking.

But this isn't quite true. Sure, you have stand-out years like 1959, but if you look at IMDB's top 250 movies, you'll notice something interesting. The top-voted movies of all time are fairly evenly spread across the decades with a big bump toward the end. Why? Well, in part a move you've seen recently tends to be more impressive in your memory, so movies that pre-date IMDB aren't always very highly rated.

But that doesn't entirely explain the phenomenon. 2008 for example, has:

The Dark Knight, WALL·E, Gran Torino, Slumdog Millionaire, The Wrestler, In Bruges, Let the Right One In, Changeling and The Curious Case of Benjamin Button

Now, I'll be the first to say that some of those represent short-term fascination or novelty. Still, I think Gran Torino, Slumdog Millionair, The Wrestler and Let the Right One In certainly do compare well to their historical analogs. Going back a year, you have films like No Country for Old Men and There Will Be Blood. Meanwhile, in 1945, 1956, 1970 and 1971 there is only one movie that made the list each of those years. Why? Because movie-making has been inconsistent throughout history and that's both good and bad.

Certainly, if movie making followed any one formula, no matter how well crafted that formula, it would have precluded some of these films. Instead, it's a hectic and chaotic process that yields one or two great movies every year and a handful of very good films.

2008 had 9 movies last year in the top 250, beating out the next-best year by two movies (1957, 1995, 1999, 2003, 2004 and 2007 all tied at 7 films). While this makes me wish I'd been alive in 1957 to sample the amazing creative output of that generation, it also makes me glad that I'm around now. So far 2010 has 4 movies on the list, and I think we'll see at least one of those (Inception, which immediately jumped to the #3 spot on opening weekend, not an easy feat) stay on the list for many, many years to come.

By year, here's the number of top-250 movies:

1921: 1
1925: 1
1926: 1
1927: 2
1930: 1
1931: 2
1933: 2
1934: 1
1936: 1
1938: 1
1939: 3
1940: 4
1941: 2
1942: 1
1943: 1
1944: 1
1945: 1
1946: 4
1948: 3
1949: 2
1950: 4
1951: 3
1952: 3
1953: 3
1954: 5
1955: 2
1956: 1
1957: 7
1958: 2
1959: 5
1960: 3
1961: 3
1962: 3
1963: 2
1964: 1
1965: 1
1966: 3
1967: 3
1968: 4
1969: 2
1970: 1
1971: 1
1972: 2
1973: 2
1974: 3
1975: 5
1976: 3
1977: 2
1978: 1
1979: 4
1980: 4
1981: 2
1982: 3
1983: 2
1984: 3
1985: 2
1986: 3
1987: 2
1988: 5
1989: 1
1990: 1
1991: 2
1992: 2
1993: 3
1994: 6
1995: 7
1996: 2
1997: 4
1998: 5
1999: 7
2000: 6
2001: 6
2002: 3
2003: 7
2004: 7
2005: 3
2006: 7
2007: 6
2008: 9
2009: 6
2010: 4

The Laundry series by Charles Stross

The Laundry series is an ongoing series of books by Charles Stross. It's a genre mashup which includes Lovecraftian horror, science fiction, comedy, spy thriller and computer industry elements. Put simply, it's the story of a computer geek turned Bondesque British spy, set against a backdrop of horrible, multi-dimensional creatures that would like nothing more than to eat his brains (or at least cohabit with them).

The setup is rather brilliant on its own, and it's no surprise that it was quickly turned into its own roleplaying game. Our hero was a budding computer scientist who discovered just a bit too much about the nature of his field. In fact, it turns out that under the surface of the well-ordered mathematics of computer science lies magic. "Spells" are just complex mathematical problems like those introduced by Alan Turing during World War II, and those who accidentally solve them are quickly scooped up by The Laundry, the magical equivalent of MI6, as a possible risk to national security.

The Laundry is therefore populated with very smart people whose only option is to work for a giant government bureaucracy for the rest of their lives. The only way to move on to anything remotely rewarding appears to be the path to field agent status and that's where we pick up in the first book, The Atrocity Archives, with our hero Bob Howard as he embarks on his very first field mission.

The combination of computer science, general geekdom, horror and spy elements gives the books a hilariously perverse tone. There are times that I've had to stop reading, just to catch my breath, and that doesn't happen very often for me.

The second book in the series has been out for about a year and is titled The Jennifer Morgue. It traces Bob's exploits in a subsequent adventure that is more directly styled on the works of Ian Fleming, most especially the Bond novels.

The next book in the series is about to be released, and is called The Fuller Memorandum. It should be out by July 6, according to Amazon.com. I highly recommend picking up the first book and working your way up to The Fuller Memorandum, but much like the Bond books or movies, I don't think you absolutely have to read them in order. Certainly The Jennifer Morgue is a good enough book to stand on its own, should you prefer to start there, and I have high hopes for this next book for similar reasons.

Collatz graphs and Perl 6

I wanted to play around a bit with numbers in Perl 6, so the Collatz Conjecture seemed like a good place to start.

The Collatz Conjecture is a simple statement which has some massive implications:

f(n) = if n is even n/2, otherwise (n x 3)+1

Given that n is a positive integer, repeated applications will eventually yield 1.

So, for example, n=10. On the first step, we get 5 because 10 is even, so we divide by two. On the second step, we get 16 because 5 is odd, so we multiply by 3 and add 1. Because 16 is a power of 2, we know that continuing to divide it by 2 will eventually give us 1, and all results until we reach 1 will be even so we're done. In fact, so far every number tested works this way but no one can, as yet, mathematically prove that there isn't an exception.

My code is fairly simple. It works backwards from 1. You can only get to 1 from 2 because there's no odd, positive integer (0 doesn't count, here) that when multiplied by 3 and incremented by 1, is 1. So then we work on 2, which can only be arrived at from 4. 4 can be arrived at from 1, but 1 is the end of the process, so we ignore that. It can also be arrived at from 8. 8 Can be arrived at from 16, but as we know from above 16 can be arrived at from 5 as well as 32, so that's our first branch in the tree. This process can be repeated for as long as you have enough computer memory to store all of the current branch-points.

That's exactly what I do, and I configured it so that you simply input a depth of tree you want to output and it outputs every value in a tree of that depth.

If you would like to see the code, check it out on github. The output svg file is also in the same directory for the tree-like output and the radial output from the twopi filter, along with the Graphviz source. I generated the following graph with the help of my good friends Graphviz and rsvg. The whole depth-20 graph looks like this:

But, of course, it's hard to make anything out there, so here's a close-up:

There are some interesting things that jump out at me in this. Because you can only ever get to a multiple of 3 from its double (since n-1 is never divisible by 3 if n is), once you hit any number that's divisible by 3, you embark on a straight line, like the one you see above at 213. The other thing that becomes obvious is that even fairly low numbers (like 19) don't appear until very late in the process. That there's no duplication in this graph is really fascinating.

The most interesting part of the Collatz Conjecture, I think, is that it can be expressed so easily that anyone with a rudimentary amount of math skills can understand it. This makes it accessible to a wider audience than any other unsolved problem in math that I can think of, and yet it's never been solved. Interesting stuff.

Anyway, enjoy your own explorations of the Collatz Conjecture and don't get too engrossed, or you might find that you're a bit too much like this XKCD comic: http://xkcd.com/710/

Are your passwords safe in MD5 or SHA-1 formats?

I've read, over and over again, various questions and seemingly authoritative statements about the security of various hashing algorithms. I've gotten kind of tired of reading misinformation, so here's some detail that you can trust.

• US-CERT of the U. S. Department of Homeland Security said MD5 "should be considered cryptographically broken and unsuitable for further use."
• The document that this was stated in is titled, "Vulnerability Note VU#836068: MD5 vulnerable to collision attacks"
• A collision attack is where an attacker, given access to a hashed password (or other plain text), crafts a password that yields the same result when hashed. Thus to a password authentication system, the crafted "collision" seems to be the correct password.
• If your password hashing scheme does not use a salt, none of this is interesting to you, as you have little or no security to speak of given an attacker (internal or external) who has access to your hashed passwords.

OK, so what does any of this mean to you? Is MD5 secure? Well, not really. It is possible, with moderate hardware investment and access to the hashed password to generate a "skeleton key." No one can "crack" the original password in a reasonable amount of time that I know of or that I've read about, but access to an equivalent password solves many problems for an attacker, even if they can't then take that password and use it against other services (since those services would not be using the same "salt" which prevents the same password hashing the same way on two different sites or services).

The question you have to ask yourself is this: why are you hashing passwords? Is it to protect them, should someone gain access to your systems from the outside? Is it to protect them from those who have access to the data store? In these cases, md5 is at best a weak protection, but it is significantly better than some of the alternatives (DES, etc.) which are breakable in practically no time.

But MD5 is used in many places besides password hashing. Should we stop using it there? Probably not.

For example many backup and data validation tools use MD5 to make sure that data has not been modified (either to initiate a backup/copy or to safeguard against accidental local change). These purposes are still served just as well now as they were when MD5 was introduced, and the fact that MD5 has been proven to have possible collision attacks does not really impact the data integrity aspect of the algorithm. Of course, there are cases where MD5 will identify a block as unchanged when it has, in fact, changed. This is true for all hashing algorithms, but the reason that MD5 was initially considered acceptable for this purpose was that the chance of that happening without malicious intent is astronomically small (that malicious intent was not believed to be as much of a factor then is not interesting to us, now). It would be a bit like dropping a penny down into one of those boxes with water where the goal is to land it on a small platform, and just as you dropped it, an earthquake struck, causing the penny to bounce off the platform, jump back up through the slot and blind you. Just as I don't recommend avoiding such games because of the risk of blindness, I don't think you need to stay away from hashing algorithms (including MD5) in order to avoid missing a data update. If you think someone might be waiting for you to drop the penny so they can set off some dynamite, then you have a different kind of problem, and MD5 might not be the best choice (e.g. if you're performing MD5 checksums in order to verify that a system's software has not been compromised).

Now, that changes as your risk profile changes. There are times, I believe, where it makes sense to take extra precautions. For example, if you're making constant backups of large amounts of rapidly changing data whose integrity in original and backup form has a high risk associated (e.g. medical data), then I might use two hashing algorithms to perform the verification. MD5 might be a fine choice for one of them, but I'd use SHA-1 or something similar on top of it. It's still astoundingly unlikely to be an issue, but there's a time an place for being stupidly extra-certain and if you can afford the extra CPU cycles, why not compute two hashes while you're looking at the data?

What about SHA-1? Hasn't that been broken too? No, SHA-1 has known weaknesses which will likely yield security-impacting attacks in the future, but as of now, these weaknesses have yet to be translated into actual attack vectors. It's certainly worth staying on top of, and keeping a flexible hashing scheme (ala the OpenBSD/LDAP schema) in your application in order to upgrade to SHA-3 when it becomes available and has been thoroughly tested, but for now SHA-1 is an excellent choice for anything short of military/state-secret sorts of crypto-hashing needs.

Bruce Schneier, who is recognized around the world as an authority on cryptographic security, had this to say about the news regarding SHA-1:

They can find collisions in SHA-1 in 2⁶⁹ calculations, about 2,000 times faster than brute force. Right now, that is just on the far edge of feasibility with current technology.
Jon Callas, PGP's CTO, put it best: "It's time to walk, but not run, to the fire exits. You don't see smoke, but the fire alarms have gone off." That's basically what I said last August.

It's time for us all to migrate away from SHA-1.
Most of the hash functions we have, and all the ones in widespread use, are based on the general principles of MD4. Clearly we've learned a lot about hash functions in the past decade, and I think we can start applying that knowledge to create something even more secure.

Hash functions are the least-well-understood cryptographic primitive, and hashing techniques are much less developed than encryption techniques. Regularly there are surprising cryptographic results in hashing ... we still have a lot to learn about hashing.

Perl 6, Python, hyperoperators and list comprehensions

I use Python every day at work, and I do like the language. There are things about it that annoy me, but I don't think that's ever not been true of any language I've used. One of Python's best features is its list comprehensions. These short snippets of code can embody so much work that it often feels like Python is writing your code for you.

At night, I go home and work on Perl 6, the upcoming update to the decades-old programming language which adds features from nearly every programming language you've ever heard of (and some you haven't). The direct equivalent of the list comprehension in Perl 6 is the same as it was in Perl 5: map. Here's how you use map in Perl 6:

 map {$_ + 10}, (1,2,3,4)

This yields the same list as the Python:

 [ x + 10 for x in 1,2,3,4 ]

Ah, but the astute among you are noticing that the Perl uses a variable name that's always the same, thus making nested map statements painful due to the need to create temporary variable names manually. In Perl 5, this was true, but we can name those temporaries quite easily now:

 map { $^x + 10 }, (1,2,3,4)

Perl sees these temporaries from left to right and considers them positional parameters in the order that they appear to the current block (which is also a closure).

But Perl 6 gives us something more than map. In fact, map will be used much less frequently in Perl 6 because of hyperoperators. A hyperoperator is an operator that takes another operator as a parameter and augments its behavior. In Perl 6, hyperoperators can do this:

 (1,2,3,4) <<+>> 10


or the Unicode equivalent:


 (1,2,3,4) «+» 10

This takes the + operator and makes it work on the list given on the right side, adding the value on the left to each item and returning the newly created list of results. So, we never need to create a closure in order to ask Perl to do some particular binary operation on all elements of a list. Instead, we just pass the list, the operator and the right hand side to a hyperoperator and it does all the heavy lifting. We don't even need to see the temporary variable that's being used.

List comprehensions like Python's [ ... for ... ] and Perl's map are extremely valuable things for doing complex operations, but when what you really want is just to perform a simple operation on the elements of a list, hyperoperators give you what you need without the trappings you don't care about.

Note: There are actually multiple forms of hyperoperator depending on how "DWIMy" (Do What I Mean) you want it to be and on which of its arguments. See Synopsis 3's section on Hyperoperators for more.

5 things you can do with Lists in Perl 6, Python and Ruby

I think practical examples of doing the same sorts of tasks in different programming languages can be wonderful tools. Recently, an IT student in Poland named Konrad posted a followup on his blog to the 2007 Ruby blog, "5 things you can do with a Ruby array in one line (PLUS A FREE BONUS!!)" by drewolson. He updated this for Python. Of course, having worked with Perl 6 quite a lot recently (see my Google Buzz posts titled "Your daily dose of Perl 6"), I was compelled to do the same for that language. See below for the results. Notice that Konrad chose temporary variable names that were much shorter than drewolson's, so the visual comparison between Ruby and Pyhthon in terms of code size is somewhat unfair, but I'll go with the original names where I need temporaries, just to be fair to Ruby.

Summing elements


Here, the original Ruby example printed the result, but I've trimmed that out for consistency with the rest of the examples.


Ruby:

  my_array.inject(0){|sum,item| sum + item}

Python:

  sum(my_list)

Perl 6:

  [+] @my_array

Double every item

Ruby:

  my_array.map{|item| item*2 }

Python:

  [2 * x for x in my_list]

Perl 6:

  @my_array <<*>> 2

Finding all items that meet your criteria (such as being divisible by 3)

Ruby:

  my_array.find_all{|item| item % 3 == 0 }

Python:

  [x for x in my_list if x % 3 == 0]

Perl 6:

  grep {$^item %% 3}, @my_array

  # or...

  grep * %% 3, @my_array

Combine techniques

Ruby:

  my_array.find_all{|item| item % 3 == 0 }.inject(0){|sum,item| sum + item }

Python:

  sum(x for x in my_list if x % 3 == 0)

Perl 6:

  [+] grep * %% 3, @my_array

Sorting

For more information on how Perl 6 sorting works and what the *-autoclosure syntax that I've used above does, see carl's excellent Perl 6 Advent Calendar post.


Ruby:

  my_array.sort
  my_array.sort_by{|item| item*-1}

Python:

  sorted(my_list)
  sorted(my_list, reverse=True)

Perl 6:

  @my_array.sort;
  @my_array.sort: -*;

And there you have it. Enjoy the many choices you have in programming languages!

Lost: What the !@#& was going on?!

Lost's pilot episode pulled me in. It was one of the best television shows I'd ever seen. The writing was brilliant, the actors were not only excellent, but all clearly gave everything they had (with one or two exceptions, but I'm able to ignore than in an ensemble cast). By the second season, the show had ground to a halt, and I became bored. I'd also been burned by Alias, Abrams' previous creation, and was not eager to be strung along for season after season again. I stopped watching late in the second season, and didn't come back until the fifth. The fourth and fifth seasons definitely picked up the pace of the show, but I was left wondering how much of what was going on would be revealed. The show was clearly set on a very cosmic trajectory, and to play that out would risk alienating a large percentage of the viewship (no matter how you play out a cosmic ending, it always alienates someone who feels that your story conflicts with their beliefs).

So, it was with trepidation that I approached the sixth season, and in fact in the final two and a half hour movie, they never did play out the larger back-story. This post is aimed at exploring what was actually going on and whether, speculation aside, we had enough information to understand what it was that was going on. This will involve spoilers for the entire series. If you haven't watched the final seasons, I suggest you do, but go in expecting a non-reveal. I liked the last episode, but I felt the way I felt at the end of Donnie Darko: clearly someone knew where they wanted to go with this, but decided they didn't have the time, creative freedom or desire to follow through. In the case of DD, the story played out in supplemental materials on the Web. In the case of Lost, I think the creators simply don't want to explain what they feel they've sufficiently hinted at, for fear of losing the sense of mystery.

OK, so let's see if we can extract reason from this show. There is really only one unanswered question of merit: what is the island? The other questions ("what are the numbers," "why do pregnant women die, etc." are secondary to this central theme and may not have answers outside of fan speculation).

Writing a Perl 6 URI module

I wanted to write a parser of some sort using Perl 6's spiffy parser language otherwise known as "rules". This is the super-extended regular expression syntax that Perl 6's own parser is written in, and it's not just powerful, it's easy to use. In fact, it's so easy to use that almost all of my time writing a URI parser module was spent on other aspects of the code than the parser itself.

First off, some background. Perl 6 has a URI module already. However, it relies on a number of Perl built-in character classes to match things like digits and alphanumerics. In reality, the RFCs that define URIs are very precise, and there are different specifications depending on what you need. So, I decided to re-write the module with a pluggable parser so that you could give a regular, modern URI and have it parse correctly, but you could also ask for special "IRI" parsing on an internationalized URI and the right thing would happen there. I even went so far as to bring in an older version of the specification as a legacy mode.

The current state of the Perl 6 parser and runtime called Rakudo is actually fairly solid for a pre-release implementation of such a complex language spec. There are some gaping holes, but they were all relatively easy to work around. Some of these included overly aggressive list-flattening, some operators that were broken at the time I wrote this code and the big one: named rules only work as a stand-alone grammer with a specific entry-point called TOP.

I worked around all of these issues and have, so far, been able to parse basic URIs according to RFC 3986. Here's a sample of what a Perl grammar for URIs looks like:

    token URI {
        ':' [ '?' ]? [ '#' ]?
    }

Here you can see most of the basics: "token" introduces a single expression within the grammar. It calls out to other tokens by enclosing their names in angle-brackets. Literal sequences are enclosed in single-quotes and sub-expressions can be enclosed in square-brackets with regular expression-like repetition counts such as ? for 0 or 1 matches.

In order to have a pluggable interface, I needed a class capable of providing me with two things for each grammar: the grammar itself and a set of routines which would tell me how to find the resulting URI elements in the match data. For this I defined an interface using Perl 6's roles:

  role URI::Specification {

      method parser() { ... }

      method scheme_path() { ... }

      # ... other _path methods here...

  }

Those ellipses are literal. They cause the methods to be required for any class composed with this role, but do not define any functionality themselves.

Each parser is then defined as:

  class URI::rfc3896 does URI::Specification {

      grammar URI::rfc3896::spec {

          token TOP { }

          # RFC definition of URI goes here.

      }

      method parser() { return ::URI::rfc3896::spec }

      method scheme_path() {

          gather do { take }

      }

      # And so on ...

  }

That's it. The only really funky bit here is the gather/take code in the scheme_path. That's the way Perl 6 defines a coroutine-like interface. The paths define how we traverse the match object to find match results. So, for example, the "scheme" (the "http" in "http:/www.example.com/") can only be matched in the URI rule's scheme sub-rule. Some URI elements, however, such as authority (the host name and port - possibly username as well) can be matched multiple ways, so these routines might return multiple lists of subrule names to traverse. I would have simply returned a list of lists, but Perl 6's parameter passing is very complex and currently some of the specification is not yet implemented. Right now, this manifests as overly aggressive list flattening when returning them from a subroutine or method.

This is why I used coroutines to return each of the sub-lists, one call at a time.

I'll continue to post new updates as my URI module nears readiness. For now, it's just awaiting some love on the other parsers, and I think it'll be ready to go.

Icons from Wikimedia Commons

I'm often lacking for exactly the right Icon for Web work, and then I remember Wikimedia Commons. For a number of open-licensed UIs I've used the Wikimedia images to help create a look that implies I spent an awful lot of time and energy. These icons vary in quality because they were uploaded from different sources, but you can get some of the best icons available from this site (along with, of course, the best freely licensed photographs and other media). Commons is a gem, and if you don't use it, you're probably missing out.

The Crystal Icons from KDE were uploaded as PNG, but the original SVG icons can be acquired from their original site. These are very clean and plastic-feeling icons for everything from the power button you see on the left to various arrows to hands to gears. Because they were uploaded in fairly high resolution, using the PNG versions isn't all bad, but you'll still get a better final result if you work from the SVG.

There are also many icons for specific applications. You can use these when writing reviews or otherwise mentioning the applications in quesiton. Many of the application icons come from the Crystal set as above, but some are directly lifted from open source projects, such as the Gaim (now pidgin) icon you see to the left.

Other icons include arrows, globes, hands and many other cateogries. You can browse all of the categories from Wikimedia's "icons by subject" page.

To use these icons, I recommend always selecting SVG format images and then using Inkscape . Load an SVG icon up in Inkscape and select File, Export Bitmap... to save it as a PNG file, setting the height and width to exactly what you want. For now, PNG is the way Web browsers deal with icons the best, though in the future, SVG will be taking over that role as older browsers fade away.

Fire alarms: Among the worst UIs in history

So a fire alarm just went off in my office building. Here's what you hear:

"*beep* Your attention please. There has been a report of an emergency in this building. If your floor evacuation tone sounds after this message, please leave the building. *beep* Your Attention please. ..."

OK, am I the only person who sees the problem with that announcement? Is the second *beep* just a repeat of the message or am I supposed to read that as "my floor's evacuation tone?" What does my floor's signal tone sound like? Is it different from other floors? Does our floor's signal tone rule relative to the suckage of other floors' signal tones?

So here's an idea: Instead of a useless recorded message, have the damned thing announce the floor that the problem exists on (or floors). I'd really love to hear, "Floors 8 through 10 should consider jumping out the windows because a fire is quickly sucking the oxygen out of the stair wells!" Now that would be a warning I could do something with.

Test Wave Post

I'm testing out Google Wave integration. Please, let me know what you think in the Wave, below.

Enforced bad passwords

Long ago, I got sick of sites that restrict me to bad passwords, but today I came across another and it has pushed me to yet again explain why it is that you should never restrict passwords without deeply compelling cause.

Today's offender was Boston Coach, the luxury livery service in that was founded by Boston's Fidelity Investments (the legend states that one day, Fidelity's owner, Ned Johnson wanted a cab and couldn't get one; the next day he had his own fleet of black sedans with smartly dressed drivers, trained to treat their passengers like royalty). Anyway, so I wanted to hire a Coach to take my mother and I to a concert for her birthday (I won't say what birthday; you're welcome, Mom). I had to sign up for an account on their Web site. They committed a few sins in the process:

1. Unless you really need a pseudonym, don't ask the user to create one. Instead, use the email address for logins.
2. Never restrict passwords unless you are technologically constrained to do so, and if you are, file a security bug with whatever braindead software it was that forced you to (or consider just dumping it).
3. Test your UI with and without JavaScript support. This sounds silly, but there are plenty of environments where people aren't allowed to enable unsafe browser features.
4. Never give the user an error without explaining what it is that they did to get it. Two examples came up, here: the password security policy and the number of occupants per car.

To get back to the heart of my concern: the password. I use PasswordSafe, a program originally written by renowned security expert, Bruce Schneier. It can happily generate a very random, long password. For example:

 K1/}"jUCF/byp6( : $1$0ZV5xOu3$iTgccli1bBSykSJxcOrfi.

Notice that this password would be nearly impossible to memorize, but because PasswordSafe stores it in an encrypted file, I just have to remember one, easier to remember password to access all of them. I tried to enter this very password for my account, but don't bother trying to use it... it was rejected. The confusing bit was that the UI informed me that I had not met "minimum password requirements." Wow, if that password doesn't measure up to Boston Coach's minimum requirements, they must spend all day, every day servicing lost password requests!

In reality, what they'd done is refused to accept any password with punctuation (resulting in 31 possible characters being removed from all possible passwords on a typical US keyboard). This is a tragic thing to do to your password security, and a company founded by Fidelity Investments should know better.

Anyway, they should fix their broken software and anyone else that uses such terrible requirements for passwords should get on it ASAP.

Twitter / Buzz: the new news?

Today's XKCD discusses the math behind a tweet out-distancing an earthquake (oddly, I read the comic before I heard about the quake). Later in the day, I found myself using Google's Buzz to post pictures of a fire in Boston. It's now getting to the point that I look to the Buzz map on my Droid before I consult Boston.com for local news. It's not that it's more rational or more considered. It's just a matter wanting to know what's going on now rather than a half hour ago. Sure, I can visit a regular news site and find more detail later on, but there's just nothing like having a few thousand potential "reporters" on the scene.

I suppose the future written by some science fiction authors is coming: we'll all be the on-the-scene "reporters" with actual journalists being the people who surf Twitter, Buzz, YouTube and so forth, the way they used to listen to police-band radio for a story. Once a journalist can tap into your head-mounted cam for a live feed hire you on the spot as a freelance photographer, there will be no story too fast to be fed into the hungry maw of the Internet.

Python class attribute annoyance

Python class attributes are fairly handy things, if somewhat visually misleading. Coming from other languages, you might expect this:

  class Foo(object):

    a = 10

To define a class whose instances will have one attribute called a. Not quite. a is actually what most languages call a "static attribute" or "static member" of the class itself, not the instances. Python calls these "class attributes." Once you know this, class attributes are a tool you'll reach for in a number of circumstances, but they have subtle behaviors that can feel like bugs.

For example, today I was trying to do something like this:

  class Foo(object):

    a = 10

    b = a + 5

which works just fine and does what you might expect (a is 10 and b is 15). But, this will yield an error:

  class Foo(object):

    a = 10

    b = [ a+i for i in range(1,11) ]

You might expect b to contain [ 11, 12, ..., 20 ] but instead, you get an error telling you that a isn't defined. [Note: tested in Python 2.6 and 3.1] This subtle flaw exists because that a+i is actually being executed in a nested lexical scope, but because it was created inside of a class body, it fails to inherit what appears to be the parent scope and thus has no access to its lexically scoped variables. There are many ways to accomplish what you might have intended, here, but none of them are very clean. For example:

  class Foo(object):

    a = 10

    b = [ z+i for z in (a,) for i in range(1,11) ]

Now you are passing a as a parameter to that nested scope, so it works perfectly. It's certainly a stilted way to do this, but it works just fine.
Coming, as I do, from Perl, this feels very odd. Perl's OO model is, at best, a framework upon which to build your own. Even still, this kind of scoping problem just never happens. Any lexical scope introduced anywhere in Perl will have a parent scope which is visually quite obvious. Running into such subtle shifts in Python's behavior seems counter to its stated goal of simplicity and elegance.

Safe browsing and virus removal

Sometimes you have to use Windows. There might be a game you like that only runs there or you might need a Windows-only program for work. Whatever it is that draws you to Windows, you know going in that you run the risk of your system becoming compromised (becoming a "zombie," getting a key-logger or any number of other harmful scenarios). OS/X is starting to feel the heat of increased market share as well, in case Mac fans thought they were somehow immune. In 2009 and now in 2010, Mac/OS + Safari did quite poorly in a challenge to compromise browsers. My brother just recently got some sort of malware that caused him to spam the family with bogus links, and I put together this overview of what to do in response. In case it's useful to others, here you go:

Preventative:

1) Always use Firefox to browse the web (Safari and Chrome are getting there, but currently don't have the suite of helpful and stable plugins that Firefox does) http://www.mozilla.com/en-US/firefox/personal.html
2) Always use the noscript plugin for Firefox http://noscript.net/ and add exceptions with care
3) If you're going to visit a site that might be questionable, use the "Tools -> Start Private Browsing" feature

Doing anything less is roughly equivalent to going on a sex tour of the third world without condoms. That's not a pretty metaphor, but neither is having your system infected with every bot this side of Robbie from Forbidden Planet.

As for cleaning your existing system... it's hard. The best and safest way is to back up your data and then use the re-install/recovery disks that came with the computer. If you want a less drastic approach (that isn't as guaranteed to work), then I suggest one of these resources:

AntiVir removal tool -- Avira, makers of my favorite free antivirus tool
McAfee Virus Removal Tools -- McAfee (about $90)
Symantec Removal Tools -- Symantec removal tools (free?)

I suggest figuring out what you have first. AntiVir, McAfee or Symantec can be used to do a full scan, and should turn this up. If not, try a malware removal tool like Spybot Search and Destroy SpyBot Search and Destroy (but be careful if you do a Web search for it... don't click on ads, and make sure you spelled it correctly).

To keep yourself safe in the future, make sure you have an up-to-date virus scanning tool (AntiVir has a free version that pops up a single ad for their product only, per day, asking you to buy the full version and there are paid programs from Symantec and McAfee). Also, make sure that you run the latest version of your browser (Firefox will auto-update with security fixes, but you should upgrade to the latest major version at least once every 6 months). Don't use IE. but if you really must, make sure it's updated to the very latest version. Microsoft's track record for keeping old browsers secure isn't very good.

Beyond that, consider doing everything that isn't Windows-specific in a virtual machine. You can get an easy-to-use virtual machine manager at http://www.virtualbox.org/wiki/Downloads and then download the install image for Ubuntu Linux and load it up in the virtual machine. This allows you to do things that would otherwise be unsafe in Windows within a safer environment. It's cumbersome, but the security return on your investment is well worth it.