A few years ago I set out on a personal journey to study and watch a performance of each of Shakespeare’s 37 plays. I’ve reached my goal and, though it’s not a usual topic around here, I wanted to get my thoughts down while fresh. I absolutely loved some of these plays and performances, and so I’d like to highlight them, especially because my favorites are, with one exception, not “popular” plays. Per tradition, I begin with my least enjoyed plays and work my way up. All performances were either a recording of a live stage or an adaptation, so they’re also available to you if you’re interested, though in most cases not for free. I’ll mention notable performances when applicable. The availability of a great performance certainly influenced my play rankings.

Like many of you, I had assigned reading for several Shakespeare plays in high school. I loathed these assignments. I wasn’t interested at the time, nor was I mature enough to appreciate the writing. Even revisiting as an adult, the conventional selection — Romeo and Juliet, Julius Caesar, etc. — are not highly ranked on my list. For the next couple of decades I thought that Shakespeare just wasn’t for me.

Then I watched the 1993 adaption of Much Ado About Nothing and it instantly became one of my favorite films. Why didn’t we read this in high school?! Reading the play with footnotes helped to follow the humor and allusions. Even with the film’s abridging, some of it still went over my head. I soon discovered Asimov’s Guide to Shakespeare — yes, that Asimov — which was exactly what I needed, and a perfect companion while reading and watching the plays. If stumbling upon this turned out so well, then I’d better keep going.

Wanting a solid set of the plays with good footnotes and editing — there is no canonical version of the plays — I picked up a copy of The Norton Shakespeare. Unfortunately it’s part of the college textbook racket, and it shows. The collection is designed to be sold to students who will lug them in bookbags, will typically open them face-up on a desk, and are uninterested in their contents beyond class. It includes a short-term, digital-only, DRMed component to prevent resale. After all, their target audience will not read it again anyway. Though at least it’s complete and compact, better for reference than reading.

In contrast, the Folger Shakespeare Library mass market paperbacks are better for enthusiasts, both in form and format. They’re clearly built for casual, comfortable reading. However, they’re not sold as a complete set, and gathering used copies takes some work.

Also essential was BBC Television Shakespeare, produced between 1978 and 1985. Finding productions of the more obscure plays is tricky, but it always provided a fallback. In some cases these were the best performances anyway! When I mention “the BBC production” I mean this series. Like many collections, they omit The Two Noble Kinsmen due to unclear authorship, and for this reason I’m omitting it from my list as well. As with any faithful production, I suggest subtitles on the first viewing, as it aids with understanding. Shakespeare’s sentence structure is sometimes difficult to parse by moderns, and on-screen text helps. (By the way, a couple of handy SHA-1 sums for those who know how to use them:)

0ae909e5444c17183570407bd09a622d2827751e
55c77ed7afb8d377c9626527cc762bda7f3e1d83

As my list will show, my favorites are comedic comedies and histories, particularly the two Henriads, each a group of four plays. The first — Richard II, 1 Henry IV, 2 Henry IV, and Henry V — concerns events around Henry V, in the late 14th and early 15th century. Those number prefixes are parts, as in Henry IV has two parts. In my list I combine parts as though a single play. The second — 1 Henry VI, 2 Henry VI, 3 Henry VI, Richard III — is about the Wars of the Roses, spanning the 15th century. Asimov’s book was essential for filling in the substantial historical background for these plays, and my journey was also in part a history study.

I especially enjoy villain monologues, and plays with them rank higher as a result. It’s said that everyone is the hero of their own story, but Shakespeare’s villains may know that they’re villains and revel it in it, bragging directly to the audience about all the trouble they’re going to cause. In some cases they mock the audience’s sacred values, which in a way, is like the stand up comedy of Shakespeare’s time. Notable examples are Edmund (King Lear), Aaron (Titus Andronicus), Richard III, Iago (Othello), and Shylock (The Merchant of Venice).

As with literature even today, authors are not experts in moral reasoning and protagonists are often, on reflection, incredibly evil. Shakespeare is no different, especially for historical events and people, praising those who create mass misery (e.g. tyrants waging wars) and vilifying those who improve everyone’s lives (e.g. anyone who deals with money). Up to and including Shakespeare’s time, a pre-industrial army on the march was a rolling humanitarian crisis, even in “friendly” territory, slaughtering and stealing its way through the country in order to keep going. So, much like suspension of belief, there’s a suspension of morality where I engage with the material on its own moral terms, however illogical it may be.

Now finally my list. The beginning will be short and negative because, to be frank, I disliked some of the plays. Even Shakespeare had to work under constraints. In his time none were regarded as great works. They weren’t even viewed as literature, but similarly to how we consider television scripts today. Also, around 20% of plays credited to Shakespeare were collaborations of some degree, though the collaboration details have been long lost. For simplicity, I will just refer to the author as Shakespeare.

(37) Timon of Athens

I have nothing positive to say about this play. It’s about a man who borrows and spends recklessly, then learns all the wrong lessons from the predictable results.

(36) The Two Gentlemen of Verona

Involves a couple of love triangles, a woman disguised as a man — a common Shakespeare trope — and perhaps the worst ending to a play ever written. The two “gentlemen” are terrible people and undeserving of their happy ending. Though I enjoyed the scenes with Proteus and Crab, the play’s fool and his dog.

(35) Troilus and Cressida

Interesting that it’s set during the Iliad and features legendary characters such as Achilles, Ajax, and Hector. I have no other positives to note. Cressida’s abrupt change of character in the Greek camp later in the play is baffling, as though part of the play has been lost, and ruins an already dull play for me.

(34) The Winter’s Tale

A baby princess is lost, presumed dead, and raised by shepherds. She is later rediscovered by her father as a young adult. It has a promising start, but in the final act the main plot is hastily resolved off-stage and seemingly replaced with a hastily rewritten ending that nonsensically resolves a secondary story line.

(33) Cymbeline

The title refers to a legendary early King of Britain and is set in the first century, but it is primarily about his daughter. The plot is complicated so I won’t summarize it here. It’s long and I just didn’t enjoy it. This is the second play in the list to feature a woman disguised as a man.

(32) The Tempest

A political exile stranded on an island in the Mediterranean gains magical powers through study, with the help of a spirit creates a tempest that strands his enemies on his island, then gently torments them until he’s satisfied that he’s had his revenge. It’s an okay play.

More interesting is the historical context behind the play. It’s based loosely on events around the founding of Jamestown, Virginia. Until this play, Shakespeare and Jamestown were, in my mind, unrelated historical events. In fact, Pocahontas very nearly met Shakespeare, missing him by just a couple of years, but she did meet his rival, Ben Jonson. I spent far more time catching up on real history, including reading the fascinating True Reportory, than I did on the play.

(31) The Taming of the Shrew

About a man courting and “taming” an ill-tempered woman, the shrew. The seeming moral of the play was outdated even in Shakespeare’s time, and it’s unclear what was intended. Technically it’s a play within a play, and an outer frame presents the play as part of an elaborate prank. However, the outer frame is dropped and never revisited, indicating that perhaps this part of the play was lost. The BBC production skips this framing entirely and plays it straight.

(30) All’s Well That Ends Well

Helena, a low-born enterprising young woman, saves a king’s life. She’s in love with a nobleman, Bertram, and the king orders him to marry her as repayment. He spurns her solely due to her low upbringing and flees the country. She gives chase, and eventually wins him over. Helena is a great character, and Bertram is utterly undeserving of her, which ruins the play for me in an unearned ending.

(29) Antony and Cleopatra

A tragedy about people who we know for sure existed, the first such on the list so far. The sequel to Julius Caesar, completing the story of the Second Triumvirate. Historically interesting, but the title characters were terrible, selfish people, including in the play, and they aren’t interesting enough to make up for it.

I enjoyed the portrayal of Octavian as a shrewd politician.

(28) Julius Caesar

A classic school reading assignment. Caesar’s death in front of the Statue of Pompey is obviously poetic, and so every performance loves playing it up. Antony’s speech is my favorite part of the play. I didn’t dislike this play, but nor did I find it interesting revisiting it as an adult.

(27) Coriolanus

About the career of a legendary Roman general and war hero who attempts to enter politics. He despises the plebeians, which gets him into trouble, but all he really wants is to please is mother. Stratford Festival has a worthy adaption in a contemporary setting.

(26) Henry VIII

He reigned from 1509 to 1547, but the play only covers Henry VIII’s first divorce. It paved the way for the English Reformation, though the play has surprisingly little to say it, or his murder spree. It’s set a few decades after the events of Richard III — too distant to truly connect with the second Henriad.

While I appreciate its historical context — with liberal dramatic license — it’s my least favorite of the English histories. It’s not part of an epic tetralogy, and the subject matter is mundane. My favorite scene is Katherine (Catherine in the history books) firmly rejecting the court’s jurisdiction and walking out. My favorite line: “No man’s pie is freed from his ambitious finger.”

(25) Romeo and Juliet

Another classic reading assignment that requires no description. A beautiful play, but I just don’t connect with its romantic core.

(24) The Merchant of Venice

An infamously antisemitic play where a Jewish moneylender, Shylock, loans to the titular merchant of Venice where the collateral is the original “pound of flesh,” providing the source for that cliche. Though even in his prejudice, Shakespeare can’t help but write multifaceted characters, particularly with Shylock’s famous “If you prick us, do we not bleed?” speech.

(23) Twelfth Night

Twins, a young man and a woman, are separated by a shipwreck. The woman disguises herself as a man and takes employment with a local duke and falls in love with him, but her employment requires her to carry love letters to the duke’s love interest. In the meantime the brother arrives, unaware his sister is in town in disguise, and everyone gets the twins mixed up leading to comedy. It’s a fun play. The title has nothing to do with the play, but refers to the holiday when the play was first performed.

The play is the source of the famous quote, “Some are born great, some achieve greatness, and some have greatness thrust upon them.” It’s used as part of a joke, and when I heard it, I had thought the play was mocking some original source.

(22) Pericles

A Greek play about a royal family — father, mother, daughter — separated by unfortunate — if contrived — circumstances, each thinking the others dead, but all tearfully reunited in a happy ending. My favorite part is the daughter, Marina, talking her way out of trouble: “She’s able to freeze the god Priapus and undo a whole generation.”

The BBC production stirred me, particularly the scene where Pericles and Marina are reunited.

(21) Richard II

Richard II, grandson of the famed Edward III, was a young King of England from 1367 to 1400. At least in the play, he carelessly makes dangerous enemies of his friends, and so is deposed by Henry Bolingbroke, who goes on to become Henry IV. The play is primarily about this abrupt transition of power, and it is the first play of the first Henriad. The conflict in this play creates tensions that will not be resolved until 1485, the end of the Wars of the Roses. Shakespeare spends seven additional plays on this a huge, interesting subject.

For me, Richard II is the most dull of the Henriad plays. It’s a slow start, but establishes the groundwork for the greater plays that follow. The BBC production of the first Henriad has “linked” casting where the same actors play the same roles through the four plays, which makes this an even more important watch.

(20) Othello

Another of the famous tragedy. Othello, an important Venetian general, and “the Moore of Venice” is dispatched to Venice-controlled Cyprus to defend against an attack by the Ottoman Turks. Iago, who has been overlooked for promotion by Othello, treacherously seeks revenge, secretly sabotaging all involved while they call him “honest Iago.” Though his schemes quickly go well beyond revenge, and continues sowing chaos just for his own fun.

I watched a few adaptions, and I most enjoyed the 2015 Royal Shakespeare Company Othello, which places it in a modern setting and requires few changes to do so.

(19) The Comedy of Errors

A fun, short play about a highly contrived situation: Two pairs of twins, where each pair of brothers has been given the same name, is separated at birth. As adults they all end up in the same town, and everyone mixes them up leading to comedy. It’s the lightest of Shakespeare’s plays, but also lacks depth.

(18) Hamlet

Another common, more senior, high school reading assignment. Shakespeare’s longest play, and probably the most subtle. In everything spoken between Hamlet and his murderous uncle, Claudius, one must read between the lines. Their real meanings are obscured by courtly language — familiar to Shakespeare’s audience, but not moderns. Asimov is great for understanding the political maneuvering, which is a lot like a game of chess. It made me appreciate the play more than I would have otherwise.

You’d be hard-pressed to find something that beats the faithful, star-studded 1996 major film adaption.

(17) Richard III

The final play of the second Henriad. Much of the play is Richard III winking at the audience, monologuing about his villainous plans, then executing those plans without remorse. Makes cheering for the bad guy fun. If you want to see an evil schemer get away with it, at least right up until the end when he gets his comeuppance, this is the play for you. This play is the source of the famous “My kingdom for a horse.”

I liked two different performances for different reasons. The 1995 major film puts the play in the World Word II era. It’s solid and does well standing alone. The BBC production has linked casting with the three parts of Henry VI, which allows one to enjoy it in full in its broader context. It’s also well-performed, but obviously has less spectacle and a lower budget.

(16) The Merry Wives of Windsor

The comedy spin-off of Henry IV. Allegedly, Elizabeth I liked the character of John Falstaff from Henry IV so much — I can’t blame her! — that she demanded another play with the character, and so Shakespeare wrote this play. The play brings over several characters from Henry IV. Unfortunately it’s in name only and they hardly behave like the same characters. Despite this, it’s still fun and does not require knowledge of Henry IV.

Falstaff ineptly attempts to seduce two married women, the titular wives, who play along in order to get revenge on him. However, their husbands are not in on the prank. One suspects infidelity and hatches his own plans. The confusion leads to the comedy.

The 2018 Royal Shakespeare Company production aptly puts it in a modern suburban setting.

(15) Titus Andronicus

A play about a legendary Roman general committed to duty above all else, even the lives of his own sons. He and his family become brutal victims of political rivals, and in return gets his own brutal revenge. It’s by far Shakespeare’s most violent and disturbing play. It’s a bit too violent even for me, but it ranks this highly because Aaron the Moore is such a fantastic character, another villain that loves winking at the audience. His lines throughout the play make me smile: “If one good deed in all my life I did, I do repent it from my very soul.”

I enjoyed the 1999 major film, which puts it in a contemporary setting.

(14) King Lear

The titular, mythological king of pre-Roman Britain wants to retire, and so he divides his kingdom between his three daughters. However, after petty selfishness on Lear’s part, he disowns the most deserving daughter, while the other two scheme against one another.

Some of the scenes in this play are my favorite among Shakespeare, such as Edmund’s monologue on bastards where he criticizes the status quo and mocks the audience’s beliefs. It also has one of the best fools, who while playing dumb, is both observant and wise. That’s most of Shakespeare’s fools, but it’s especially true in King Lear (“This is not altogether fool, my lord.”). This fool uses this “tenure” to openly mock the king to his face, the only character that can do so without repercussions.

My favorite performance was the 2015 Stratford Festival stage production, especially for its Edmund, Lear, and Fool.

(13) Macbeth

The shortest tragedy, a common reading assignment, and a perfect example of literature I could not appreciate without more maturity. Even the plays I dislike have beautiful poetry, but I especially love it in Macbeth.

The history behind Macbeth is itself fascinating. The play was written custom for the newly-crowned King James I — of King James Version fame — and even calls him out in the audience. James I was obsessed with witch hunts, so the play includes witchcraft. The character Banquo was by tradition considered to be his ancestor.

My favorite production by far — I watched a number of them! — was the 2021 film. It should be an approachable introduction for Shakespeare newcomers more interested in drama than comedy. Notably for me, it departs from typical productions in that Macbeth and Lady Macbeth do not scream at each other — perhaps normally a side effect of speaking loudly for stage performance. Particularly in Act 1, Scene 7 (“screw your courage to the sticking place”). In the film they argue calmly, like a couple in a genuine, healthy relationship, making the tragedy that much more tragic.

That being said, it drops the ball with the porter scene — a bit of comic relief just after Macbeth murders Duncan. There’s knocking at the gate, and the porter, charged with attending it, is hungover and takes his time. In a monologue he imagines himself porter to Hell, and on each impatient knock considers the different souls he would be greeting. Of all the porter scenes I watched, the best porter as the 2017 Stratford Festival production, where he is both charismatic and hilarious. I wish I could share a clip.

(12) King John

King John, brother of “Coeur de Lion” Richard I, ruled in early 13th century. His reign led to the Magna Carta, and he’s also the Prince John of the Robin Hood legend, though because it’s a history, and paints John in a positive light, that legend isn’t included. It depicts fascinating, real historical events and people, including Eleanor of Aquitaine. It also has one of my favorite Shakespeare characters, Phillip the Bastard, who gets all the coolest lines. I especially love his introductory scene where his lineage is disputed by his half-bother and Eleanor, impressed, essentially adopts him on the spot.

The 2015 Stratford Festival stage performance is wonderful, and I’ve re-watched it a few times. The performances are all great.

(11–9) Henry VI

As previously noted, this is actually three plays. At 3–4 hours apiece, it’s about the length of a modern television season. I thought it might take awhile to consume, but I was completely sucked in, watching and studying the whole trilogy in a single weekend.

Henry V died young in 1422, and his infant son became Henry VI, leaving England ruled by his uncles. As an adult he was a weak king, which allowed the conflicts of the previously-mentioned Richard II to bubble up into the Wars of the Roses, a bloody power conflict between the Lancasters and Yorks. The play features historical people including Joan la Pucelle (“Joan of Arc”), English war hero John Talbot, and Jack Cade. Richard III wraps up the conflicts of Henry VI, forming the second Henriad. When watching/reading the play, keep in mind that the play is anti-French, anti-York, and (implicitly) pro-Tudor.

Most of the first part was probably not written by Shakespeare, but rather adapted from an existing play to fill out the backstory. I think I can see the “seams” between the original and the edits that introduce the roses.

I loved the BBC production of the second Henriad. Producing such an epic story must be daunting, and it’s amazing what they could convey with such limited budget and means. It has hilarious and clever cinematography for the scene where the Countess of Auvergne attempts to trap Talbot (Part 1, Act 2, Scene 3). Again, I wish I could share a clip!

(8) Henry V

Due to his amazing victories, most notably at Agincourt where, for once, Shakespeare isn’t exaggerating the odds, Henry V is one of the great kings of English history. This play is a followup to Richard II and Henry IV, completing the first Henriad, and depicts Henry V’s war with France. Outside of the classroom, this is one of Shakespeare’s most popular plays.

The obvious choice for viewing is the 1989 major film, which, by borrowing a few scenes from Henry IV, attempts a standalone experience, though with limited success. I watched it before Henry IV, and I could not understand why the film was so sentimental about a character that hadn’t even appeared yet. It probably has the best Saint Crispin’s Day Speech ever performed, in part because it’s placed in a broader context than originally intended. The introduction is bold as is Exeter’s ultimatum delivery. It cleverly, and without changing his lines, also depicts Montjoy, the French messenger, as sympathetic to the English, also not originally intended. I didn’t realize this until I watched other productions.

The BBC production is also worthy, in large part because of its linked casting with Richard II and Henry IV. It’s also unabridged, including the whole glove thing, for better or worse.

(7–6) Henry IV

People will think I’m crazy, but yes, I’m placing Henry IV above Henry V. My reason is just two words: John Falstaff. This character is one of Shakespeare’s greatest creations, and really makes these plays for me. As previously noted, this is two plays mainly because John Falstaff was such a huge hit. The sequel mostly retreads the same ground, but that’s fine! I’ve read and re-read all the Falstaff scenes because they’re so fun. I now have a habit of quoting Falstaff, and it drives my wife nuts.

The Falstaff role makes or breaks a Henry IV production, and my love for this play is in large part thanks to the phenomenal BBC production. It has a warm, charismatic Falstaff that perfectly nails the role. It’s great even beyond Falstaff, of course. At the end of part 2, I tear up seeing Henry V test the chief justice. I adore this production. What a masterpiece.

(5) A Midsummer Night’s Dream

A popular, fun, frivolous play that I enjoyed even more than I expected, where faeries interfere with Athenians who wander into their forest. The “rude mechanicals” are charming, especially the naive earnestness of Nick Bottom, making them my favorite part of the play.

My enjoyment is largely thanks to a 2014 stage production with great performances all around, great cinematography, and incredible effects. Highly recommended. Honorable mention goes to the great Nick Bottom performances of the BBC production and the 1999 major film.

(4) As You Like It

A pastoral comedy about idyllic rural life, and the source of the famous quote “All the world’s a stage.” A duke has deposed his duke brother, exiling him and his followers to the forest where the rest of the play takes place. The main character, Rosalind, is one of the exiles, and, disguised as a man named Ganymede, flees into the forest with her cousin. There she runs into her also-exiled love interest, Orlando. While still disguised as Ganymede, she roleplays as Rosalind — that is, herself — to help him practice wooing herself. Crazy and fun.

A couple of my favorite lines are “There’s no clock in the forest” and “falser than vows made in wine.” It’s an unusually musical play, and has a big, happy ending. The fool, Touchstone, is one of my favorite fools, named such because he tests the character of everyone with whom he comes in contact.

It ranks so highly because of an endearing 2019 production by Kentucky Shakespeare, which sets the story in a 19th century Kentucky. This is the most amateur production I’ve shared so far — literally Shakespeare in the park — but it’s just so enjoyable. Their Rosalind is fantastic and really makes the play work. I’ve listened to just the audio of the play, like a podcast, many times now.

(3) Measure for Measure

A comedy about justice and mercy. The duke of Vienna announces he will be away on a trip to Poland, but secretly poses as a monk in order to get his thumb on the pulse of his city. Unfortunately the man running the city in his stead is corrupt, and the softhearted duke can’t help but pull strings behind the scenes to undo the damage, and more. He sets up a scheme such that, after his dramatic return as duke, the plot is unraveled while simultaneously testing the character of all involved.

I love so many of the characters and elements of this play. I smile when the duke jumps into action, my heart wrenches at Isabella’s impassioned speech for mercy (“it is excellent to have a giant’s strength, but it is tyrannous to use it like a giant”), I admire the provost’s selfless loyalty to the duke, I laugh when Lucio the “fantastic” keeps putting his foot in his mouth, and I cry when Mariana begs Isabella to forgive. All around a wonderful play.

Like so many already, a big part of my love for the play is the BBC production, which is full of great performances, particularly the duke, Isabella, and Lucio.

(2) Much Ado About Nothing

As the play that finally got me interested in Shakespeare, of course it’s near the top of the list. Forget Romeo and Juliet: Benedick and Beatrice are Shakespeare’s greatest romantic pairing!

Don Pedro, Prince of Aragon, stops in Messina with his soldiers while returning from a military action. While in town there’s a matchmaking plot and lots of eavesdropping, and then chaos created by the wicked Don John, brother to Don Pedro. It’s a fun, light, hilarious play. It also features another of Shakespeare’s great comic characters, Dogberry, famous for his malapropisms.

This is a very popular play with tons of productions, though I only watched a few of them. The previously-mentioned 1993 adaption remains my favorite. It does some abridging, but honestly, it makes the play better and improves the comedic beats.

(1) Love’s Labour’s Lost

Finally, my favorite play of all, and an unusual one to be at the top of the list. Much of the play is subtle parody and so makes for a poor first play for newcomers, who would not be familiar enough with Shakespeare’s language to distinguish parody from genuine.

The King of Navarre and three lords swear an oath to seclude themselves in study, swearing off the company of women. Then the French princess and her court arrives, the four men secretly write love letters in violation of their oaths, and comedy ensues. There are also various eccentric side characters mixed into the plot to spice it up. It’s all a ton of fun and ends with an inept play within a play about the “nine worthies.”

The major reason I love this play so much is a literally perfect 2017 production by Stratford Festival. I love every aspect of this production such that I can’t even pick a favorite element. I was hooked within the first minute.

I don’t say this because I hate PHP. There’s no reason for that: I don’t write programs in PHP, never had to use it, and don’t expect to ever need it. Despite this, I just can’t look away from PHP in the same way I can’t look away from a car accident.

I recently came across a link to the PHP manual, and morbid curiosity that caused me to look through it. It’s fun to pick an arbitrary section of the manual and see how many crazy design choices I can spot, or at least see what sort of strange terminology the manual has invented to describe a common concept. This time around, one such section was on anonymous functions, including closures. It was even worse than I expected.

In some circumstances, closures can be a litmus test. Closure semantics are not complex, but they’re subtle and a little tricky until you get hang of them. If you’re interviewing a candidate, toss in a question or two about closures. Either they’re familiar and get it right away, or they’re unfamiliar and get nothing right. The latter is when it’s most informative. PHP itself falls clearly into the latter. Not only that, the example of a “closure” in the manual demonstrates a “closure” closing over a global variable!

I’d been told for years that PHP has closures, and I took that claim at face value. In fact, PHP has had “closures” since 5.3.0, released in June 2009, so I’m over a decade late in investigating it. However, as far as I can tell, nobody’s ever pointed out that PHP “closures” are, in fact, not actually closures.

Anonymous functions and closures

Before getting into why they’re not closures, let’s go over how it works, starting with a plain old anonymous function. PHP does have anonymous functions — the easy part.

function foo() {
    return function() {
        return 1;
    };
}

The function foo returns a function that returns 1. In PHP 7 you can call the returned function immediately like so:

$r = foo()();  // $r = 1

In PHP 5 this is a syntax error because, well, it’s PHP and its parser is about as clunky as Matlab’s.

In a well-designed language, you’d expect that this could also be a closure. That is, it closes over local variables, and the function may continue to access those variables later. For example:

function bar($n) {
    return function() {
        return $n;
    };
}

bar(1)();  // error: Undefined variable: n

This fails because you must explicitly tell PHP what variables you intend to access inside the anonymous function with use:

function bar($n) {
    return function() use ($n) {
        return $n;
    };
}

bar(1)();  // 1

If this actually closed over $n, this would be a legitimate closure. Having to tell the language exactly which variables are being closed over would be pretty dumb, but it still meets the definition of a closure.

But here’s the catch: It’s not actually closing over any variables. The names listed in use are actually extra, hidden parameters bound to the current value of those variables. In other words, this is nothing more than partial function evaluation.

function bar($n) {
    $f = function() use ($n) {
        return $n;
    };
    $n++;  // never used!
    return $f;
}

$r = bar(1)();  // $r = 1

Here’s the equivalent in JavaScript using the bind() method:

function bar(n) {
    let f = function(m) {
        return m;
    };
    return f.bind(null, n);
}

This is actually more powerful than PHP’s “closures” since any arbitrary expression can be used for the bound argument. In PHP it’s limited to a couple of specific forms. If JavaScript didn’t have proper closures, and instead we all had to rely on bind(), nobody would claim that JavaScript had closures. It shouldn’t be different for PHP.

References

PHP does have references, and binding a reference to an anonymous function is kinda, sorta like a closure. But that’s still just partial function evaluation, but where that argument is a reference.

Here’s how to tell these reference captures aren’t actually closures: They work equally well for global variables as local variables. So it’s still not closing over a lexical environment, just binding a reference to a parameter.

$counter = 0;

function bar($n) {
    global $counter;
    $f = function() use (&$n, &$counter) {
        $counter++;
        return $n;
    };
    $n++;  // now has an effect
    return $f;
}

$r = bar(1)();  // $r = 2, $counter = 1

In the example above, there’s no difference between $n, a local variable, and $counter, a global variable. It wouldn’t make sense for a closure to close over a global variable.

Emacs Lisp partial function application

Emacs Lisp famously didn’t get lexical scope, and therefore closures, until fairly recently. It was — and still is by default — a dynamic scope oddball. However, it’s long had an apply-partially function for partial function application. It returns a closure-like object, and did so when the language didn’t have proper closures. So it can be used to create a “closure” just like PHP:

(defun bar (n)
  (apply-partially (lambda (m) m) n))

This works regardless of lexical or dynamic scope, which is because this construct isn’t really a closure, just like PHP’s isn’t a closure. In PHP, its partial function evaluation is built directly into the language with special use syntax.

Monkey see, monkey do

Why does the shell command language use sigils? Because it’s built atop interactive command line usage, where bare words are taken literally and variables are the exception. Why does Perl use sigils? Because it was originally designed as an alternative to shell scripts, so it mimicked that syntax. Why does PHP use sigils? Because Perl did.

The situation with closures follows that pattern, and it comes up all over PHP. Its designers see a feature in another language, but don’t really understand its purpose or semantics. So when they attempt to add that feature to PHP, they get it disastrously wrong.

That index where you can find these 343 articles is sorted newest-first, because it correlates with best-first. It’s a trend I hope to continue.

History

Before blogging, I had a simple Penn State student page showcasing a few small — and, in retrospect, trivial — side projects (1, 2, 3), none of which went anywhere. Around the beginning of my final semester of college, I was inspired by Mark Dominus’ The Universe of Discourse and Andy Owen’s Friendly Robot Overlord (gone since 2011) to start my own blosxom blog. It would be an outlet to actively discuss my projects. Some time later GitHub was founded, and I switched to a static blog hosted by GitHub Pages, which is where it lives to this day.

It’s been more challenging to manage all this content than I ever anticipated. It’s like maintaining a large piece of software, except it’s naturally more fragile. Any time I make a non-trivial change to the CSS, I have to inspect the archives to check if I broke older articles. If I did, sometimes it’s a matter of further adjusting the CSS. Other times I’ll mass edit a couple hundred articles in order to normalize some particular aspect, such as heading consistency or image usage. (Running a macro over an Emacs’ Dired buffer is great for this.)

I decided in those early days to Capitalize Every Word of the Title, and I’ve stuck with this convention purely out of consistency even though it’s looked weird to me for years. I don’t want to edit the old titles, and any hard changeover date would be even weirder (in the index listing).

With more foresight and experience, I could have laid down better conventions for myself from the beginning. Besides the universal impossibility of already having experience before it’s needed, there’s also the issue that the internet’s conventions have changed, too. This blog is currently built with HTML5, but this wasn’t always the case — especially considering that predates HTML5. When I switched to HTML5, I also adjusted some of my own conventions to match, since, at the time, I was still writing articles in raw HTML.

The mobile revolution also arrived since starting this blog. Today, about one third of visitors read the blog from a mobile device. I’ve also adjusted the CSS to work well on these devices. To that third of you: I hope you’re enjoying the experience!

Just in case you haven’t tried it, the blog also works really well with terminal-based browsers, such as Lynx and ELinks. Go ahead and give it a shot. The header that normally appears at the top of the page is actually at the bottom of the HTML document structure. It’s out of the way for browsers that ignore CSS.

If that’s not enough, last year I also spent effort making the printed style of my articles look nice. Take a look at the printed version of this article (i.e. print preview, print to PDF), and make sure to turn off the little headers added by the browser. A media selector provides a separate print stylesheet. Chrome / Chromium has consistently had the best results, followed by Firefox. Someday I’d like for browsers to be useful as typesetting engines for static documents — as an alternative to LaTeX and Groff — but they’ve still got a ways to go with paged media. (Trust me, I’ve tried.)

With more foresight, I could have done something better with my permanent URLs. Notice how they’re just dates and don’t include the title. URLs work better when they include human-meaningful context. Ideally I should be able to look at any one of my URLs and know what it’s about. Again, this decision goes all the way back to those early days when I first configured blosxom, not knowing any better.

URLs are forever, and I don’t want to break all my old links. Consistency is better than any sort of correction. I’m also practically limited to one article per day, though this has never been a real problem.

Motivation

For me, an important motivation for writing is to say something unique about a topic. For example, I’m not interested in writing a tutorial unless either no such tutorial already exists, or there’s some vital aspect all the existing tutorials miss. Each article should add new information to the internet, either raw information or through assembling existing information in a new way or with a unique perspective.

I also almost constantly feel like I’m behind the curve, like I don’t know enough or I don’t have enough skill. As many of you know, the internet is really effective at causing these feelings. Every day lots of talented people are sharing interesting and complicated things across all sorts of topics. For topics that overlap my own desired expertise, those projects have me thinking, “Man, I have no idea how to even start doing that. There’s so much I don’t know, and so much more I need to learn.” Writing articles as I learn is a great way to keep on top of new subjects.

This is tied to another problem: I have a tendency to assume the things I’ve known for awhile are common knowledge. This shows up in a few ways. First, if everyone knows what I know, plus they know a bunch of things that I don’t know, then I’ve got a lot of catching up to do. That’s another source of feeling behind the curve.

Second, when writing an article on a topic where I’ve got years of experience, I leave out way too many important details, assuming the reader already knows them. When an article I regard as valuable gets a weak response, it’s probably related to this issue.

Third, after three years of teaching, it seems like it’s becoming more difficult to put myself in the student’s shoes. I’m increasingly further away from my own days learning those early topics, and it’s harder to remember the limitations of my knowledge at that time. Having this blog really helps, and I’ve re-read some of my older articles to recall my mindset at the time.

Another way the blog helps is that it’s like having my own textbook. When teaching a topic to someone — and not necessarily a formal mentee — or even when just having a discussion, I will reference my articles when appropriate. Since they’re designed to say something unique, my article may be the only place to find certain information in a conveniently packaged form.

Finally, the last important motivating factor is that I want to spread my preferred memes. Obviously the way I do things is the Right Way, and the people who do things differently (the Wrong Way) are stupid and wrong. By writing about the sorts of topics and technologies I enjoy — C, low-level design, Emacs, Vim, programming on unix-like systems, my personal programming style — I’m encouraging others to follow my lead. Surely I’m responsible for at least one Emacs convert out there!

“Formal” professional writing

Despite having three or four novels worth of (mostly) technical writing here, my formal workplace writing leaves much to be desired. I’m no standout in this area. In the same period of time I’ve written just a handful of formal memos, each about the same length as a long blog post.

Why so few? These memos are painful to write. In order to be officially recognized, the formal memo process must be imposed upon me. What this means is that, compared to a blog post, these memos take at least an order of magnitude longer to write.

The process involves more people, dragged out over a long period of time. The writing is a lot less personal, which, in my opinion, makes it drier and less enjoyable. After the initial draft is complete, I have to switch to vastly inferior tools: emailing the same Microsoft Office documents back and forth between various people, without any proper source control. The official, mandated memo template was created by someone who didn’t know how to effectively operate a word processor, who had a poor sense of taste (ALL CAPS HEADINGS), and who obviously had no training in typesetting or style.

At the end of this long process, it’s filed into a system with no practical search capability, and where it will be quietly purged after five years, never to be seen again. Outside of the reviewers who were directly involved in the memo process, somewhere between zero and two people will have actually read it. Literally.

Arguably the memo might more polished than a blog post. I’m skeptical of this, but suppose that’s true. I’d still much rather have written ten less-polished blog posts than one more-polished memo. That’s also ten shots to produce, by chance, a more valuable article than the single, precious memo.

Let’s broaden the scope to academic papers. Thanks to some great co-workers — all three of whom are smarter and handsomer than me — a year ago I finally got a published academic paper under my belt (and more to come): ROP Gadget Prevalence and Survival under Compiler-based Binary Diversification Schemes (and I said memos were dry!). A ton of work went into this paper, and it’s far more substantial than any memo or single blog post. The process was a lot more pleasant (LaTeX instead of Word), and the results are definitely much more polished than a typical blog post. It reads well and has interesting information to present.

This all sounds great until you consider the impact. According to ACM’s statistics, the paper has been accessed 130 times as of this writing. (Yes, providing an unofficial link to the paper like I just did above doesn’t help, but I ran out of those crappy “free” links. Sue me.) Sure, the PDF might have been passed around in untrackable ways, but I also bet a lot of those accesses were just downloads that were never read. So let’s split the difference and estimate around 130 people read it.

What kind of impact does a blog post have? Talking about these numbers feels a bit taboo, like discussing salaries, but it’s important for the point I’m about to make.

• July 2017: Rolling Shutter Simulation in C
  • 10,400 unique visitors
  • #22 most popular
• August 2017: A Tutorial on Portable Makefiles
  • 14,300 unique visitors
  • #16 most popular
• June 2017: Switching to the Mutt Email Client
  • 23,100 unique visitors
  • #5 most popular
• May 2015: Raw Linux Threads via System Calls
  • 40,500 unique visitors
  • #1 most popular, not even counting the Japanese publication

Note that all but the last has been published for less time than our paper. The average time on these pages is between 5 and 6 minutes, so these are actual readers, not just visitors that take one glance and leave. Thanks to the information age, a technical blog article on established blog can reach an audience 100 times larger than a journal for a fraction of the effort and cost. There are other benefits, too:

1. I get immediate feedback in the form of comments and email (open peer review).

2. The content is available for free (open access). It’s trivial to link and share blog articles.

3. Even more, this entire blog is in the public domain. If you don’t believe me, check out the public domain dedication in the footer of this page. It’s been there for years, and you can verify that yourself. Every single change to this blog in the past 6 years has been publicly documented (transparency).

4. When I write about a topic, I make it a goal to provide the code and data to try it for yourself (open data and materials). This code and data is also either in the public domain, or as close to it as possible.

5. Link aggregators and social media are great at finding the best stuff and making it more visible (censorship resistance). When I have a big hit, it’s often Reddit or Hacker News driving people to my article. Sometimes it’s other blogs.

In 2017, a blog is by far the most effective way to publish and share written information, and have more scientific quality than journals. More people should be publishing through blog articles than traditional forms of publications, which are far less effective.

Since this has proven to be such an effective medium, I’m going to make a promise right here, right now. And as I explained above with transparency, there are no take-backs on this one. If you’re reading this, it’s on the public record forever. I promise to deliver another 10 years of free, interesting, useful content. This stuff is going to be even better, too. On this blog’s 20th anniversary, September 1st, 2027, we’ll see how I did.

• Download: twenty-sided-tale.epub
• Repository: https://github.com/skeeto/twenty-sided-tale

To build the book yourself, you will only need make and pandoc.

Why did I want this?

Ever since I got a tablet a couple years ago, I’ve completely switched over to e-books. Prior to the tablet, if there was an e-book I wanted to read, I’d have to read from a computer monitor while sitting at a desk. Anyone who’s tried it can tell you it’s not a comfortable way to read for long periods, so I only reserved the effort for e-book-only books that were really worth it. However, once comfortable with the tablet, I gave away nearly all my paper books from my bookshelves at home. The remaining use of paper books is because either an e-book version isn’t reasonably available or the book is very graphical, not suited to read/view on a screen (full image astronomy books, Calvin and Hobbes collections).

As far as formats go, I prefer PDF and ePub, depending on the contents of the book. Technical books fare better as PDFs due to elaborate typesetting used for diagrams and code samples. For prose-oriented content, particularly fiction, ePub is the better format due to its flexibility and looseness. Twenty-Sided Tale falls in this latter category. The reader gets to decide the font, size, color, contrast, and word wrapping. I kept the ePub’s CSS to a bare minimum as to not get in the reader’s way. Unfortunately I’ve found that most ePub readers are awful at rendering content, so while technically you could do the same fancy typesetting with ePub, it rarely works out well.

The Process

To start, I spent about 8 hours with Emacs manually converting each article into Markdown and concatenating them into a single document. The ePub is generated from the Markdown using the Pandoc “universal document converter.” The markup includes some HTML, because Markdown alone, even Pandoc’s flavor, isn’t expressive enough for the typesetting needs of this particular book. This means it can only reasonably be transformed into HTML-based formats.

Pandoc isn’t good enough for some kinds of publishing, but it was sufficient here. The one feature I really wished it had was support for tagging arbitrary document elements with CSS classes (images, paragraphs, blockquotes, etc.), effectively extending Markdown’s syntax. Currently only headings support extra attributes. Such a feature would have allowed me to bypass all use of HTML, and the classes could maybe have been re-used in other output formats, like LaTeX.

Once I got the book in a comfortable format, I spent another 1.5 weeks combing through the book fixing up punctuation, spelling, grammar, and, in some cases, wording. It was my first time editing a book — fiction in particular — and in many cases I wasn’t sure of the correct way to punctuate and capitalize some particular expression. Is “Foreman” capitalized when talking about a particular foreman? What about “Queen?” How are quoted questions punctuated when the sentence continues beyond the quotes? As an official source on the matter, I consulted the Chicago Manual of Style. The first edition is free online. It’s from 1906, but style really hasn’t changed too much over the past century!

The original articles were written over a period of three years. Understandably, Shamus forgot how some of the story’s proper names were spelled over this time period. There wasn’t a wiki to check. Some proper names had two, three, or even four different spellings. Sometimes I picked the most common usage, sometimes the first usage, and sometimes I had to read the article’s comments written by the game’s players to see how they spelled their own proper names.

I also sunk time into a stylesheet for a straight HTML version of the book, with the images embedded within the HTML document itself. This will be one of the two outputs if you build the book in the repository.

A Process to Improve

Now I’ve got a tidy, standalone e-book version of one of my favorite online stories. When I want to re-read it again in the future, it will be as comfortable as reading any other novel.

This has been a wonderful research project into a new domain (for me): writing and editing, style, and today’s tooling for writing and editing. As a software developer, the latter overlaps my expertise and is particularly fascinating. A note to entrepreneurs: There’s massive room for improvement in this area. Compared software development, the processes in place today for professional writing and editing is, by my estimates, about 20 years behind. It’s a place where Microsoft Word is still the industry standard. Few authors and editors are using source control or leveraging the powerful tools available for creating and manipulating their writing.

Unfortunately it’s not so much a technical problem as it is a social/educational one. The tools mostly exist in one form or another, but they’re not being put to use. Even if an author or editor learns or builds a more powerful set of tools, they must still interoperate with people who do not. Looking at it optimistically, this is a potential door into the industry for myself: a computer whiz editor who doesn’t require Word-formatted manuscripts; who can make the computer reliably and quickly perform the tedious work. Or maybe that idea only works in fiction.

What prompted this? In my Emacs Chat, Sacha asked me if this blog was part of Planet Emacsen (currently, it’s not). If my tags are accurate, only about 25% of my articles are about Emacs, so most of my blog isn’t relevant there. Tag feeds will go a long way to help support these “planet” aggregators, should they want to include my articles. For example, Planet Emacsen would use my Emacs feed.

Static Site Generation

I couldn’t practically support these extra feeds until recently. Remember, this blog is statically generated. More feeds means more content to generate, because articles are duplicated in whole for each feed. In past years, Jekyll would probably take on the order of an hour to do all this for a single build. Fortunately, Jekyll has improved dramatically, especially in the past year or so, and these feeds have little impact on the total build time. It’s currently around 10 seconds or so. Not bad at all!

A consequence of being statically generated is that you can’t ask for a combination of tags as a single feed. It would be a combinatorial nightmare (billions of feeds). Plus, the request would have to normalize the tag order (e.g. alphabetical) or else the combinatorial explosion to be far worse (i.e. exceeding the number of atoms in the universe). So I hope you can forgive me when subscribing to each tag individually.

Duplicate Articles

What if an article matches multiple tags? It will appear in each feed where it’s tagged, possibly showing up multiple times in your web feed reader. Fortunately, this is where Atom saves the day! I’m leveraging Atom’s prudent design to make this work cleanly. Articles’ UUIDs are consistent across all of these feeds, so if your web feed reader is smart enough, it will recognize these as being the same article. For example, this article is f47e5404-cc4a-3cc0-01ce-a844c04721b8 regardless of which feed you see it in.

Unfortunately, Elfeed isn’t smart enough for this. Sorry! In order to better support all the broken RSS feeds out there, I had to compromise on entry keying. I couldn’t trust RSS feeds to provide me a reasonably unique key, so, transitively, Elfeed doesn’t fully trust Atom’s UUIDs either. These RSS feeds are broken largely because RSS itself is a broken mess. When making new feeds in the future, please use Atom!

Atom requires that every feed and article have a proper UUID. It doesn’t matter where you get the feed from. You could subscribe to the same exact feed at three different URLs (mirrors perhaps) and your reader could reliably use the UUIDs to avoid duplication. Or, if you’re subscribed to an aggregator like Planet Emacsen, and it includes content from a feed to which you’re also directly subscribed, your reader client should be able to merge these articles. In comparison, RSS not only doesn’t require UUIDs, it actively discourages them with its broken guid tag, so merging content from multiple sources is impossible with RSS.

Anyway, if most of my content doesn’t suit you, you can now subscribe to the subset that does. Aren’t Atom feeds cool?

Julia is a new programming language primarily intended for scientific computing. It’s attempting to take on roles that are currently occupied by Matlab, its clones, and R. “Matlab done right” could very well be its tag-line, but it’s more than that. It has a beautiful type system, it’s homoiconic, and its generic function support would make a Lisp developer jealous. It still has a long ways to go, but, except for some unfortunate issues, it’s off to a great start.

Speaking strictly in terms of the language, doing better than Matlab isn’t really a significant feat. Among major programming languages, Matlab’s awfulness and bad design is second only to PHP. Octave fixes a lot of the Matlab language, but it can only go so far.

For both Matlab and R, the real strength is the enormous library of toolboxes and functionality available to help solve seemingly any scientific computing task. Plus the mindshare and the community. Julia has none of this yet. The language is mostly complete, but it will take years to build up its own package library to similar standards.

If you’re curious about learning more, the Julia manual covers the entire language as it currently exists. Unfortunately anything outside the language proper and its standard library is under-documented at this time.

A Beautiful Type System

One of the first things you’ll be told is that Julia is dynamically typed. That is, statically typed (C++, Java, Haskell) versus dynamically typed (Lisp, Python, JavaScript). However, Julia has the rather unique property that it straddles between these, and it could be argued to belong to one or the other.

The defining characteristic of static typing is that bindings (i.e. variables) have types. In dynamic typing, only values and objects have types. In Julia, all bindings have a type, making it like a statically typed language. If no type is explicitly declared, that type is Any, an abstract supertype of all types. This comes into play with generic functions.

Both abstract and concrete types can be parameterized by other types, and certain values. The :: syntax it used to declare a type.

type Point {T}
  x::T
  y::T
end

This creates a Point constructor function. When calling the constructor, the parameter type can be implicit, derived from the type of its arguments, or explicit. Because both x and y have the same type, so must the constructor’s arguments.

# Implicit type:
Point(1, -1)
# => Point{Int64}(1,-1)

# Explicit type:
Point{Float64}(1.1, -1.0)
# => Point{Float64}(1.1,-1.0)

Point(1, 1.0)
# ERROR: no method Point{T}(Int64,Float64)

The type can be constrained using <:. If Point is declared like the following it is restricted to real numbers. This is just like Java’s Point<T extends Number>.

type Point {T <: Real}
  x::T
  y::T
end

Unlike most languages, arrays aren’t built directly into the language. They’re implemented almost entirely in Julia itself using this type system. The special part is that they get literal syntax.

[1, 2, 3]
# => Array{Int64,1}

[1.0 2.0; 3.0 4.0]
# => Array{Float64,2}

Each Array is parameterized by the type of value it holds and by an integer, indicating its rank.

The Billion Dollar Mistake

Julia has avoided what some call The Billion Dollar Mistake: null references. In languages such as Java, null is allowed in place of any object of any type. This allowance has lead to many run-time bugs that, if null didn’t exist, would have been caught at compile time.

Julia has no null and so there’s no way to make this mistake, though some kinds of APIs are harder to express without it.

Generic Functions

All of Julia’s functions are generic, including that Point constructor above. Different methods can be defined for the same function name, but for different types. In Common Lisp and Clojure, generic functions are an opt-in feature, so most functions are not generic.

Note that this is significantly different than function overloading, where the specific function to call is determined at compile time. In multimethods, the method chosen is determined by the run-time type of its arguments. One of Julia’s notable achievements is that its multimethods have very high performance. There’s usually more of a trade-off.

Julia’s operators are functions with special syntax. For example, the + function,

+(3, 4)
# => 7

A big advantage is that operators can be passed around as first-class values.

map(-, [1, 2, 3])
# [-1, -2, -3]

Because all functions are generic, operators can have methods defined for specific types, effectively becoming operator overloading (but better!).

function +(p1::Point, p2::Point)
  return Point(p1.x + p1.y, p2.x + p2.y)
end

Point(1,1) + Point(1, 2)
# => Point{Int64}(2,3)

(Note that to write this method correctly, either Point or the method should probably promote its arguments.)

Foreign Function Interface

Julia has a really slick foreign function interface (FFI). Libraries don’t need to be explicitly loaded and call interfaces don’t have to be declared ahead of time. That’s all taken care of automatically.

I’m not going to dive into the details, but basically all you have to do is indicate the library, the function, the return type, and then pass the arguments.

ccall((:clock, "libc"), Int32, ())
# => 2292761

Generally this would be wrapped up nicely in a regular function and the caller would have no idea an FFI is being used. Unfortunately structs aren’t yet supported.

Julia’s Problems

Not everything is elegant, though. There are some strange design decisions. The two big ones for me are strings and modules.

Confused Strings

Julia has a Char type that represents a Unicode code point. It’s a 32-bit value. So far so good. However, a String is not a sequence of these. A Julia string is a byte-array of UTF-8 encoded characters.

Indexing into a string operates on bytes rather than characters. Attempting to index into the middle of a character results in an error. Yuck!

"naïvety"[4]
# ERROR: invalid UTF-8 character index

I don’t understand why this behavior was chosen. This would make sense if Julia was an old language and this was designed before Unicode was established (e.g. C). But, no, this is a brand new language. There’s no excuse not to get this right the first time. I suspect it has to do with Julia’s FFI.

Clunky, Closed Modules

Julia’s module system looks like it was taken right out of Scheme’s R6RS. This isn’t a good thing.

The module definition that wraps the entire module up in a single syntactic unit. Here’s an example from the documentation. According to the style guide, the body of the module is not indented out.

module MyModule
using Lib
export MyType, foo

type MyType
  x
end

bar(x) = 2x
foo(a::MyType) = bar(a.x) + 1

import Base.show
show(io, a::MyType) = print(io, "MyType $(a.x)")
end

That final end seals the module for good. There’s no opening the module back up to define or redefine new functions or types. If you want to change something you have to reload the entire module, which will obsolete any type instances.

Compare this to Clojure, where the module isn’t wrapped up in a syntactical construct.

(ns my.module
  (require : [clojure.set :refer [rename-keys]]))

Common Lisp’s defpackage also works like this. At any time you can jump into a namespace and make new definitions.

(in-ns 'my.module)

This is absolutely essential to interactive development. The lack of this makes Julia far less dynamic than it should be. Combined with the lack of a printer, Julia is not currently suitable as an interactive interpreter subprocess (Slime, Cider, Skewer, etc.).

This is a real shame, because I’d like to start playing around with Julia, but right now it feels like a chore. It’s needlessly restricted to a C++/Java style workflow.

I’ll probably revisit Julia once it’s had a few more years to mature. Then we’ll see if things have improved enough for real use.

At least it is for me. This past week at work I've been furiously rushing work on a project written in Java. It's completely my fault that I'm using Java since I'm the one who picked the language. I wanted to use Java the platform, but I don't know any other JVM languages well enough to use them in place of Java the language for a project at work.

It's all sorts of little things that make writing Java so exhausting for me. At the end of the day it I just feel irritated. I hate the absolute lack of functional programming and that I have to specify everything at such a low level. The whole reason we program in high-level languages is so we can express algorithms more concisely, but Java fails at this.

Here's an example of what I'm talking about, something I basically did a few times today. Let's say you have an array of floats, nums and you want to sum them and return the result (or maybe use it in another expression). In Lisp it's very straightforward.

(reduce '+ nums)

"Reduce the sequence nums by addition." Notice that it's more about saying what I want to do rather than how to do it. I don't have to introduce any temporary storage or iterators. To do the same thing in Java it will look something like this.

int sum = 0;
for (double num : nums) {
    sum += num;
}
return sum;

If you're using an older Java without the enhanced looping construct it gets even uglier. I had to introduce a variable for accumulation and a second variable for iteration. This sort of thing has to be done all over the place in Java, and it greatly increases the cognitive overload when reading Java code.

This instruction is more about telling the computer the how rather than my overall intention. One problem with telling it the how is that I've unnecessarily locked in a specific algorithm and ordering. The literal instruction says that the numbers must be added in order, sequentially. My Lisp instruction doesn't do that.

It gets even worse when you complicate it slightly by adding a second array and, say, multiplying it pairwise with the first.

for (int i = 0; i < nums1.length; i++) {
    sum += nums1[i] * nums2[i];
}
return sum;

Now the loop gets more complex. I have to tell it how to increment the iterator. I have to tell it to check the bounds of the array. The iterator is a misdirection because the actual number stored in it isn't what's important. Again, the Lisp method is much more concise.

(reduce '+ (map 'list '* nums1 nums2))

"Map the two sequences by multiplication into a list, then reduce it by addition." Unfortunately we start to leak a little bit into the how here. I am telling it that the intermediate structure should be a list, because map forces me to pick a representation. Besides that, I am only describing my overall intention and not the obvious details.

So with Java my days become filled with the tedious low-level algorithm descriptions that I have to hammer out over and over and over. Death by a thousand paper cuts.

Lisp isn't the only language that has a (generally) much better approach; it's just my favorite. :-) Most languages with at least some decent functional facilities will also do the above concisely.

As I get more involved with tabletop RPGs, specifically Dungeons and Dragons, I find there are some related attributes that I wish these game systems had. While I'm sure there are systems do have some of these, I wish whatever I happen to be using had all of them.

Print friendly. The source material tends to be very colorful and graphical. While this can be a good thing, especially when illustrating monsters (Show, not tell!), it's bad if you want to print out your own materials. I want the crucial information available in a crisp, clean monochrome form of some sort. Not only could I reproduce material for use in notes and handouts, but I could create my own condensed sets of information by composing these crisp forms.

For example, in the D&D monster manual each monster has a nice concise block containing all the information — defenses, health, abilities, etc. — needed to use that monster. This is great, but it's on a brownish background, in a red-ish box. So close to being what I want. But even then, do I have legal permission to reproduce this information? And so ...

Licensing. The closest thing tabletop gaming has to a Free Software license would be the Open Game License (OGL), which is still pretty restrictive. I would love for the source materials to be licensed at least loosely enough that I could print out my own copies for cheap (assuming they are print friendly, per above). Have some new players sitting down at the table? To get them started, give them that stapled-together player handbook you printed out. There's RPG evangelism for you.

The Fudge role-playing game system has both these attributes down pretty well. The Fudge manual is very print friendly PDF with explicit permission to share it with your friends. However, just as yacc is a compiler compiler, Fudge is really a game system system, a system for creating game systems, so it's only part of what is needed to play a game.

Useful software tools. One specific example is character creation software. Creating a new character can be burdensome, especially for a new player. Software that allows a player to select some basic options from a menu and produce a printable, error-free character sheet can save a lot of time.

Fourth edition Dungeons and Dragons has a character builder, but it is a humongous piece of junk. It's proprietary, Windows-only, bulky, and slow. For a program that merely generates printouts based on a few user selections from some simple menus, it has some extremely excessive system requirements (much higher than the ones they claim). And it requires a reboot to install too. A human can produce the same results by hand inside of a half hour, so for a computer there is virtually no computation involved. So what is it doing? Worse of all, the fourth edition license expressly forbids competing character creation software, so no one can legally produce a reasonable one. All this thing should be is a database of available character abilities, some character sheet logic, and a postscript printer.

Fortunately there are some decent, generic world generation tools for GMs out there, such as random inn generators, random dungeon generators, and so on. And another one. I've mentioned this before.

If you know any systems that fit the above descriptions well, go ahead and link them in the comments!

While regular expressions have limited usefulness, especially in larger programs, they're still very handy to have from time to time. It's usually difficult to write a lexer or tokenizer without one. Because of this several languages build them right into the language itself, rather than tacked on as a library. It allows the regular expressions to be stored literally in the code, treated as its own type, rather than inside a string. The problem with storing a regular expression inside a string is that it can easily make an already complex regular expression much more complex. This is because there are two levels of parsing going on.

Consider this regular expression where we match an alphanumeric word inside of quotes. I'm going to use slashes to delimit the regular expression itself.

/"\w+"/

Notice there is no escaping going on. The backslash is there is indicate a special sequence \w, which is equal to [a-zA-Z0-9_]. This will get parsed and compiled into some form in memory before it is run by a program. If the language doesn't directly support regular expressions then we usually can't put it in the code as is, since the language parser won't know how to deal with it. The solution is to store it inside of a string.

However, our regular expression contains quotes and these will need to be escaped when in a quote delimited string. But I no longer need slashes to delimit my regular expression.

"\"\w+\""

Did you notice the error yet? If not, stop and think about it for a minute. Our special sequence \w will not make it intact to the regular expression compiler. That backslash will escape the w during the string parsing step, leaving only the w. The string we typed will get parsed into a series of characters in memory, performing escapes along the way, and then that sequence will be handed to the regular expression compiler. So we have to fix it,

"\"\\w+\""

That's getting hard to understand, compared to the original. Now let's throw a curve-ball into this: let's match a backslash at the beginning of the word. The normal regular expression looks like this now,

/"\\\w+"/

We have to escape our backslash to make it a literal backslash, so it takes two of them. Now, when we want to do this in a string-stored regular expression we have to escape both of those backslashes again. It looks like this,

"\"\\\\\\w+\""

Now to match a single backslash we have to insert four backslashes! Quite unfortunately, Emacs Lisp doesn't directly support regular expressions even though the language has a lot of emphasis on text parsing, so a lot of Elisp code is riddled with this sort of thing. Elisp is especially difficult because sometimes, such as during prompts, you can enter a regular expression directly and can ignore the layer of string parsing. It's a very conscious effort to remember which situation you're in at different times.

Perl, Ruby, and JavaScript have regular expressions as part of the language and it makes a lot of sense for these languages; they tend to do a lot of text parsing. Python does it partially, with its r' syntax. Any string preceded with an r loses its escape rules, but it also means you can't match both single or double quotes without falling back to a normal string with escaping. Common Lisp may be able to do it with a reader macro, but I've never seen it done.

Remember those two levels of parsing when writing string stored regex. It helps avoid hair-pulling annoying mistakes.

Your BitTorrent client probably has DRM in it, even if it's Free software. Torrent files (.torrent) may contain a special, but unofficial, "private" flag. When it does, clients are supposed to disable decentralized tracking features, regardless of the user's desires. This is a direct analog to the copy-prevention flag that PDFs may have set, which tell PDF viewers to cripple themselves, except that your Free PDF reader is actually more likely to ignore it.

It's impossible to simply open the torrent file and turn off the flag. The client has to be modified, fixing the purposeful defect, to ignore it. Note, simpler clients that don't have these features in the first place don't have this problem, since they don't have any features to disable.

The private flag exists because modern BitTorrent trackers can function without a central tracker. If the central tracker is down, or if the user doesn't want to use it, the client can fetch a list of peers in the torrent from a worldwide distributed hash table. It's one big decentralized BitTorrent tracker (though any arbitrary data can be inserted into it). Clients also have the ability to tell each other about peers when they are doing their normal data exchange. Thanks to this, clients can transcend central trackers and join the larger global torrent of peers. It makes for healthier torrents.

Anyone who knows a few peers involved with a torrent can join in, regardless of their ability to talk to the central tracker. But private tracker sites don't want their torrents to be available outside to those outside of their control, so they proposed an addition to the BitTorrent spec for a "private" flag. Clients with decentralized capabilities are advised cripple that ability when the flag is on, so no peer lists will leak outside the private tracker. This flag was never accepted into the official spec, and I hope it never is.

Unfortunately the private trackers set an ultimatum: obey the private flag or find your client banned. The client developers fell in line and, and as far as I am aware, no publicly available clients will use decentralized tracking while the flag is on. At one point, the BitComet client ignored the flag and was banned for some time until it was "fixed".

The private flag wasn't placed in front with the rest of the metadata where it belonged. It's intentionally placed at the end of the torrent file inside of the info section. This means that the flag is part of the info_hash property of the torrent, which is the global identifier for the torrent. Unset or remove the private flag and the hash changes, creating a whole new torrent without any seeds.

This is DRM, an artificial restriction imposed on the user. It's insulting. Users should be the ones that control what happens with their computers. The reasonable approach to a private flag is that, when the private flag is enabled, decentralized tracking is turned off by default, but can be re-enabled by the user should they desire. That way the desired behavior is indicated but the user has the final say, not some unrelated website operator.

I rarely use private trackers, since they are nearly pointless, but I still find this private flag set on public torrents, probably from someone simply reposting the torrent file from a private site. It's annoying to run into. It makes the torrents weaker.

Debian, which is my distribution of choice, is generally good about removing DRM from the software it distributes. For example, the PDF readers in the repositories have their DRM disabled (i.e. xpdf). So why not do the same thing for all the intentionally defective BitTorrent clients?

I went on the Debian IRC channel and brought up the issue only to find out that everyone thought a little DRM was reasonable. So then I filed a bug report on it, which was simply closed citing that the DRM is a beneficial "feature" and that removing the intentional defect would make the clients "poorer". They also insisted that it's part of the spec when it's not. I'm really disappointed in Debian now.

Now, I could modify a client to ignore the flag, but it's not useful if I am the only one not running DRM. It takes two to tango. A client used by many people would have to be fixed before it becomes beneficial.

So when someone asks for an example of Free Software or Open Source software with DRM in it, you can point to BitTorrent clients.

Update November 2011: I've since spent a lot more time with widescreen monitors, and the web has changed a bit, so I somewhat changed my mind about this topics, as you can see by the page around you.

Web pages aren't a static medium, like books, brochures, or pamphlets. The web is not print. Accordingly, the layout of web pages should not be locked to some static width, but instead flow to fill the width of the browser like a liquid. Web pages should normally have a liquid layout.

One of the most obvious problems with the fixed layout occurs when the browser window is stretched wider than the designer had intended.

I, as a user, have little control my viewing of the website. I'm stuck reading through a keyhole. It gets much worse if the browser isn't as wide as the designer intended: a horizontal scrollbar appears and navigation becomes very difficult. My laptop runs at a resolution of 1024x768, and I frequently come across pages where this is an issue. And according to Jakob Nielsen, in 2006 77% of user's screens were 1024 pixels wide or less.

See the liquid for yourself right here: adjust the width of your browser and watch this text flow to fill the screen. You can also bring it in pretty far before you clip an image and the horizontal scrollbar appears. The exact width depends only on the widest image being displayed. This also comes into play if you adjust the font size.

Using a liquid layout allows the page to work well with a wide variety of screen widths, and most importantly, gives users lots of control over how they view the site. It's very unfortunate that (in my experience) most websites employ a poor, fixed layout. Even web design "expert" websites will ironically hand out web design tips from within these annoying confines. One of the biggest culprits driving this is Wordpress, which has this flawed layout by default.

The very worst offenders tend to be websites with little actual content, like corporate websites or "artist" portfolios. The less usable the page, the less I wanted to be there anyway.

So please drop the fancy, low-usability web designs for something with much better usability. Your users will probably appreciate it.

I use Adblock Plus to block advertisements and, more importantly, invisible privacy-breaking trackers (most people aren't even aware of these). I think ad-blocking is actually easier than ever, because ads are served from a relatively small number of domains, rather than from the websites themselves. Instead of patterns matching parts of a path, I can just block domains.

Adblock Plus emphasizes this by providing, by default, a pattern matching the server root. Example,

http://ads.example.com/*

But sometimes advertising websites are trickier, and their sub-domain is a fairly unique string,

http://ldp38fm.example.com/*

That pattern isn't very useful. I want something more like,

http://*.example.com/*

Unfortunately Adblock Plus doesn't provide this pattern automatically yet, so I have to do it manually. I think this pattern is less obvious because the URL format is actually broken. Notice have have two matching globs (*) rather than just one, even though I am simply blocking everything under a certain level.

Tim Berners-Lee regrets the format of the URL, and I agree with him. This is what URLs like http://ads.example.com/spy-tracker.js should look like,

http://com/example/ads/spy-tracker.js

It's a single coherent hierarchy with each level in order. This makes so much more sense! If I wanted to block example.com and all it's sub-domains, the pattern is much simpler and less error prone,

http://com/example/*

To anyone who ever reinvents the web: please get it right next time.

Update: There is significant further discussion in the comments.

Gmail has a nice feature: when delivering e-mail, everything including and after a + in a Gmail address is ignored. For example, mail arriving at all of these addresses would go to the same place if they were Gmail addresses,

account@example.com
account+nullprogram@example.com
account+slashdot@example.com

Thanks to this feature, when a user acquires a Gmail account, Google is actually providing about a googol (as in the number 10¹⁰⁰) different e-mail addresses to that user! Quite appropriate, really.

I have seen other mailers do similar things, like ignoring everything after dashes. A nice advantage to this is when registering at a new website I can customize my e-mail address for them by, say, throwing the website name in it. Because I have a google of e-mail addresses available, it is impossible to run out, so I can give every person I meet their own version of my address. The custom address can come in handy for sorting and filtering, and it will also tell me who is selling out my e-mail address. This, of course, assumes that someone isn't stripping out the extra text in my address to counter the Gmail feature.

However, in my personal experience, most websites will not permit +'s in addresses. This is completely ridiculous, because it means that virtually every website will incorrectly invalidate perfectly valid e-mail addresses. Even major websites, like coca-cola.com, screw this up. They see the + in the address and give up.

In fact, if I do a Google search for "email validation regex" right now, 9 of the first 10 results return websites with regular expressions that are complete garbage and will toss out many common, valid addresses. The only useful result was at the fifth spot (linked below).

For the love of Stallman's beard, stop writing your own e-mail address validators!

Why shouldn't you even bother writing your own? Because the proper Perl regular expression for RFC822 is over 6 kilobytes in length! Follow that link and look at that. This is the smallest regular expression you would need to get it right.

If you really insist on having a nice short one and don't want to use a validation library, which, again, is a stupid idea and you should be using a library, then use the dumbest, most liberal expression you can. (Just don't forget the security issues.) Like this,

.+@.+

Seriously, if you add anything else you most almost surely make it incorrectly reject valid addresses. Note that e-mail addresses can contain spaces, and even more than one @! These are valid addresses,

"John Doe"@example.com
"@TheBeach"@example.com

I have not yet found a website that will accept either of these, even though both are completely valid addresses. Even MS Outlook, which I use at work (allowing me to verify this), will refuse to send e-mail to these addresses (Gmail accepts it just fine). Hmmm... maybe having an address like these is a good anti-spam measure!

So if your e-mail address is "John Doe"@example.com no one using Outlook can send you e-mail, which sounds like a feature to me, really.

So, everyone, please stop writing e-mail validation regular expressions. The work has been done, and you will only get it wrong, guaranteed.

This is a similar rant I came across while writing mine: Stop Doing Email Validation the Wrong Way.