Like a Hurricane

This is going to be a phenomenally difficult piece to write because it involves digging up some unpleasant memories and facing some unpleasant truths which have haunted me for years. It is, in essence, a confession and a condemnation combined. It is also a chronicle of what is probably the single most cowardly action I’ve ever taken in my life.

I thought long and hard about whether to name names in this or not. If I name names the story has authenticity and the condemnatory aspect of it rings stronger. On the other hand it also distracts from a reality that most developers would rather not face, to wit: it is not only these people and these companies who do the kinds of things I’m about to relate. This is my most shameful moment of my career in software (and also, not coincidentally, the beginning of the end of it), but I am by no means alone in this.

So I’ve decided to leave in the names of the companies involved (where I can recall them) and leave out specific people. That should give enough verisimilitude to ensure that the story is believed without muddying the picture with accusations of defamation (although, truth be told, it would be kind of comical watching them try to sue me).

Waiting for the Hurricane

This happened when I worked for Pronexus, a small firm that at the time primarily made a Visual Basic toolkit for IVR applications. The company had just survived what was an existential crisis (which is to say that had three people not put in superhuman effort for a few months, the company would have failed to meet payroll). Money was finally coming in, easing some of the tensions, but the company was still in the psychological “must sell at any cost” stage of things.

I had been tasked with adding support for Rhetorex (a company which no longer exists but was purchased by Lucent and then later absorbed, erasing its very existence, as Alcatel bought it and turned it into Alcatel-Lucent) T1 interface cards for the PC. The cards were actually OEM cards manufactured by Mitel and therein lay the rub. Rhetorex wasn’t making a lot of money selling the T1 interface cards. They sold them solely to value-add to sales of Rhetorex’s massive voice processing resource cards (connected via MVIP to the T1 cards, among other trunk interfaces including Rhetorex’s own POTS cards). I suspect they were selling them at break-even or possibly loss-leader prices (to compete with then mega-rival Dialogic) and, thus, any expenses attached to supporting the boards ate into their profits in a field that was increasingly short on profit. (The market for IVR cards was approaching saturation at the time.)

Now the Mitel T1 cards were fine. They worked spiffy. Even the drivers Rhetorex had for them (it’s unclear to me who actually wrote them: Mitel or Rhetorex) were fine… provided you were doing what most people do with T1 cards: taking incoming calls. There was, however, a little… problem with outbound calls. Unpredictably—it could happen on your first call or it could happen on your hundredth, but it would eventually happen—the bus would lock so tight that your whole computer froze solid. (This was an ISA bus, so there was no protection from bad hardware/driver issues.) This rendered any application involving outbound dialing impossible until such a time as the bug was found and fixed. Rhetorex, however, was not interested in fixing this bug. I eventually had to advise my boss and the sales staff that the Rhetorex T1 cards could not be used for any kind of application that may initiate calls (which included certain classes of call centre, incidentally.

Fast forward a few weeks. Imagine my surprise when I got a phone call from a customer who complained that his system was frequently freezing up when, get this, making outbound T1 calls. Curious, I enquired further about the application and found out that it was a HURRICANE WARNING SYSTEM. It was entirely an outbound calling application. The customer had some custom hardware (sort of a really primitive precursor to a modern-day blade server) with an ISA backplane in which he wanted to stick 20 cards: 5 T1 cards and 15 voice resource cards. He had a database link to the phone company to link geographical areas to phone numbers. He had a real-time feed from the National Weather Service (I think it was) for hurricane watches. The application would monitor hurricane warnings and, on certain trigger conditions, would dial people in affected areas with warnings about and instructions for dealing with the potentially oncoming hurricane.

It was, quite literally, an application in which lives were at stake. And the sales staff sold these guys cards that they knew didn’t work. Worse, the boss knew as well and specifically instructed me not to tell these customers that the problem was with the driver. Everybody laboured under the false hope that Rhetorex would come through with driver fixes and we’d make the (rather impressive for us at the time) sale.

So I was specifically instructed by my boss to lie to the customer. I was told to blame the hardware first. When that failed (they switched to a regular, stock PC for development purposes) I was told the problem must be in their connection logic in the VBVoice application. When that failed I was told… well, you get the picture. I was instructed to lie to customers who were trying to save lives. For several months I stalled and misdirected these poor bastards even though I knew damned well what the bug was, whose fault it was and that it wasn’t going to get fixed anytime soon.

Rock You Like a Hurricane

I’d like to say that this story has a happy ending. I’d like to say that I finally stood up to my boss, told the customer the truth and took the consequences. I’d like to say the customer went with another company that had functioning hardware. I’d like to but I can’t.

I was at a very vulnerable (fiscally) part of my life at the time. I was deep in debt and the local economy was such that finding another job, especially after being dismissed for basically stabbing my employer in the back, was not viable. I caved. I went along with the boss’ instructions. I torpedoed a system designed to save lives in some of the worst natural disasters humans face just so I could keep my miserable, soul-sucking job. The only good thing to come out of the fiasco was that about six weeks after this started I finally snapped and called up Rhetorex, leaving on their voice mail system an epic rant that was considered so shocking it made its way to the ears of Rhetorex’s CEO.

To be fair, Rhetorex’s CEO, upon hearing my rant, decided that there was very obviously a major problem in his company’s product support. He called up my boss and talked to him about the voice mail (thinking that my boss had sent me to do the hit). He actually committed to fixing the bug (I’m not sure if they ever did because I left my job not long after this incident for a much better job with much higher salary) and apologized for the lackadaisical approach his company had taken before. Of course after that my boss hauled me onto the carpet and essentially put me on probation because I “could have irreparably damaged our relationship with Rhetorex” with my stunt. (It was another impetus to leave, you see.)

Edited to add…

One of my readers pointed out that what happened to the project wasn’t clear. I hinted at it above, but let me spell it out. When I said this…

I torpedoed a system designed to save lives in some of the worst natural disasters humans face just so I could keep my miserable, soul-sucking job.

…I meant that the project, in the end, was cancelled. A system for saving lives of people in natural disasters was called off because I was too much of a coward to tell the truth.

Hurricane

This incident was probably the nadir of my professional software life and the beginning of my eventual departure from the field. I was (and still am) ashamed of myself for how I dealt with that situation and I’ve found since then that every company I would ever be likely to work for would do the same or worse. There’s simply no sense of professional responsibility in software at large. (Hell, there’s no sense of professionalism of any kind in most software shops!)

Some people will condemn me for my choice. Go for it. I doubt you could say anything harsher than I’ve said to myself over the years. This incident weighs heavily on my conscience a decade and a half later. I’ve done lots of soul-searching in that time and mark this as probably the lowest thing I’ve ever done. So pour your outrage over me. It’s nothing compared to what I’ve done to myself. But I salute you for trying because it means you’ve got a soul and you’ve got morals.

Others, however, are worse. They will try to defend my choice. They will try to tell me that it’s perfectly OK to worry about petty pecuniary matters over people dying when I held the key to those people not dying. These people I hold in contempt that cannot be spelled out in words. And, sadly, these people are, in my experience, the majority of those who write software. They would, under similar circumstances, make the same choices (or even worse ones) without a second’s thought nor introspection. They are, in a word, narcissists (or maybe just plain old misanthropes).

Now it’s time for you to look into your own hearts. Ask yourself (and answer honestly): were you faced with the same choices I was, how would you have reacted? WOuld you have stood up to your boss, lost your job, torpedoed your career to save abstract lives thousands of kilometres away? Or would you have taken my craven route and crumpled under your boss’ assault?

Let’s hope you choose the former. Because the industry already has enough sociopaths.

User friendlessness

Anybody who’s known me for any length of time knows I’m an advocate of the end-user in software development. I think the contempt that programmers show for end-users is a disease. I’m hardly unique in this. A quick search of the usual suspects in tech blogs finds a lot of support for this position… with one odd oversight.

Where I seem to differ from my former colleagues is in thinking that programmers are end-users too. If you take a look at programmers’ tools, however, and compared them to equivalent tools outside of the programming craft, you’d be shocked (if you’re honest with yourself) at the disparity.

Programmers are treated as second-class citizens to even the “lusers” that they themselves deride so often.

The symptoms

Programming tools include everything (I mean everything) programmers use to make software. This includes programming languages, library suites, operating system services, text editors, debuggers and even the word processors programmers should be using to write documentation. (More on that below.) When you look at these, however, you’ll see that the tools are very badly constructed.

• They’re limited in scope and difficult to integrate with other tools (gdb, the autotools suite, Vim).
• They’re massive stovepipe systems that pretty much demand you commit to them and them alone if you want to use them at all (Emacs, Eclipse).
• They’re poorly documented to the point that the lore of experts is the only way to really get ahead (Ruby on Rails, 99.44% of F/OSS libraries, Erlang’s standard library).
• They’re inconsistently constructed so that knowledge of one portion can actively harm knowledge of another (Win32, Erlang’s collections, Common Lisp’s entire library).
• They’re filled with loads of mind-numbing ceremony necessary for even the most trivial work (Java, C#).
• They’re filled with redundant information that is a regular source of error as one part is updated without another part getting properly updated as well (Java EE or pretty much any other “enterprise” software package).
• They use a model of modularity that breaks so frequently a minor change typically involves a massive rebuild (C, C++ and any other language that uses textual inclusion instead of proper modularity).
• They have poorly-defined semantics around their abstractions with a lot of harsh, sharp corners to catch on (C, C++, Java, C#, JavaScript, PHP, Ruby, Python… let’s face it, pretty much every popular programming language and most of the unpopular ones).

It’s almost as if even the developers who do understand that they are writing for their users have forgotten that programmers are end-users too whose productivity is nuked when (not if!) their tools suck.

So let’s take a look at a few common kinds of failure.

Verbosity (you know, just like this rant!)

Go take a look at Hello, world implemented in a bunch of different languages. Look at the ceremony surrounding some of the popular programming languages in this simple task:

• C (7 lines);
• C++ (5 lines);
• Java (7 lines);
• COBOL (4 lines);
• PHP (3 lines).

Look at other languages as well, the ones that aren’t popular (or even known, for that matter!). There are quite a few doozies in there. Component Pascal, for example, or Modula-2 or Dylan.NET or elastiC or Fantom or Mercury… you get the idea.

Now the “Hello, world” program isn’t quite a fair test of a language because sometimes the ceremony is there for things that really only start to apply when you deal with non-trivial code. That being said, there is room to argue that you shouldn’t need to use the ceremony until you, well, need it. And seven lines to say “Hello, world?” That smells quite a bit. (It’s amazing to me when COBOL, of all languages, is less verbose than others!)

The real check is how much of that ceremony scales. If you only have to type the ceremony once and then have the rest of the code be mostly compact and concise, then this opening ceremony is tolerable. If, however, you have to wrap everything in equivalent ceremony with every piece you touch (I’m looking at you here, Java, and even more so at Java EE!) then the code rapidly balloons out of control and into the realm of the unusable.

Documentation

This is a pet peeve of mine, to be honest, so feel free to tune me out here and skip to the next. F/OSS libraries in particular are guilty of this, but most software is to some extent or another. Documentation, if it exists at all, is purely reference documentation. There’s no tutorial documentation (which is ironic given how much pointless ceremony is attached to most libraries!). There’s not even a birds-eye overview user guide.

Now my examples here are going to come from the Erlang/OTP distribution. This is not because I hate Erlang (quite the contrary!) but rather because it’s the documentation I’m most familiar with. Here are some documentation sub-failures to look at. (There are many more and I could write a whole blog series on documentation that sucks.)

Hard to find

Erlang’s I/O system is based on a notion of “group leaders”. It’s an important concept that you actually do have to understand if you want efficient I/O in Erlang. Pity that it’s basically undocumented. Do you think you’ll find it on erlang.org? Fat chance! Here’s where you find it (and that only when trapexit.org is actually up and running). Or you can infer it, of course, by Using the Source, Luke.

This is really bad documentation for such a vital feature. Why it languishes in an external document referenced from a third-party forum is beyond me.

Addenda and Corrigenda

An astute reader has pointed out that there exists hints of documentation for the group leaders. The erlang module does mention group leaders in passing in the form of two API calls. It doesn’t explain what any of it means but you get a hint of their existence here.

There is also documentation now on the I/O protocol which goes into more detail on group leaders. It doesn’t explain the rationale at all, but it at least shows you the nuts and bolts of its operation. This is an improvement, but still fails to be good documentation because knowing how to do something doesn’t help if you don’t know when or why.

Further, the erlang BIFs module uses the term “group leader” but there’s no link between the docs and the second document uses completely different terminology. If you don’t have access to a guru to point you to Robert Virding’s “Erlang Rationale” paper you would not be able to link the concepts without using the source. If even then.

Utterly incomprehensible

Those people who know me in the Erlang community know of my deep and abiding hatred of the sofs module. I’m sure it’s a good module in terms of it being well-written and useful. It’s just that its documentation is quite possibly the worst documentation I have ever seen anywhere. Having no documentation would, in my opinion, be superior to what’s there.

The documentation begins, for some bizarre reason I can’t fathom, with a long screed on set theory. It defines terms that no programmer shouldn’t already know: empty set, equality, intersection, subsets, disjoint sets, etc. etc. etc. It then leaps straight into a description of the data types and functions in alphabetical order: pure reference documentation, in other words. And even there the documentation remains bizarrely unhelpful. Consider this gem from the first exported function:

“Creates a function. a_function(F, T) is equivalent to from_term(F, T), if the result is a function. If no type is explicitly given, [{atom, atom}] is used as type of the function.”

What’s missing from the sofs documentation? Well, quite frankly, the actual documentation. There’s nothing showing the lifecycle of sofs usage. When using sofs, for example, what function do you use first? (Hint: set. Or from_term. Or from_external. Or from_sets. I’m guessing on all of this, though.) I suspect that sofs is badly underused and that is mostly because of its documentation.

Uncivilized APIs

A perfect example of uncivilized APIs again comes from the Erlang world. (Seriously, I’m not harshing on Erlang because I hate it! I love it and just use it for examples because I know it!)

Look at the collections in Erlang. The dict module and the orddict module, for example, are identical in their interface (with a potentially nasty semantic difference buried in the Description section). Now look at the gb_sets module. Despite many operations being similar (you insert items, delete items, etc.) the function names are not. If you choose to change your data representation from a dict, say, to a gb_set, you’re rewriting all your code. This is annoying and acts as a barrier to experimenting with different data representations.

A civilized API would, in the specific case of collections, ensure that similar operations (insert and delete, say) have the same name and the same order of operands, etc. This is not the case in Erlang’s collection set.

Of course uncivilized APIs spread far beyond Erlang. Let’s look at Microsoft’s Win32 API. CreateFile is a good example. Look at the massive ceremony surrounding just creating (or opening) a file! Seven—seven—parameters, two of which are “optional” (which means 99.44% of the time they’re going to be NULL and thus merely clutter the interface for no good reason). And that’s not all, of course. There’s also CreateFile2, CreateFileTransacted and LZOpenFile to brighten up your day. And, of course, you CreateFile whether you’re opening an existing file or, you know, actually creating a new one. (Why? Because CreateFile actually means CreateFileHandle. Which brings up the other piece of stupidity. You Create a File but you Close a Handle. Because after making one of the most comically verbose APIs available for a very basic operation, of course you’re going to make things “easier” by having a single function to close everything from files to windows to threads to… well, practically anything. Except sockets.)

Stovepipe systems

Emacs is a(n in)famous tool for programmers. Many top-notch developers swear by it. Many others swear at it. What nobody can deny, however, is that if you use Emacs, you go into it all the way. There’s no such thing as a casual Emacs user because if you’re using it only casually you’re not getting its purported benefits. Using Emacs is a lifestyle choice, not a programming tool.

Emacs also, quite infamously, doesn’t play well with others. While I’m not a huge fan, necessarily, of the “standard” GUI way of doing things, the fact that it is a standard is important. I can switch from Microsoft Word (pre- that stupid “ribbon” thing, I mean) to Abiword to LibreOffice Write to pretty much any end-user oriented software product without any difficulty; I’ll be reasonably productive in it in a day and will gradually ramp up to all the shortcuts and power user things I need as I get used to the changes. This even extends to the Ctrl+C/X/V thing: in any GUI program anywhere I can Ctrl+C to copy text and Ctrl+V to paste it.

Except Emacs.

Emacs has to do everything differently. This is, in fact, why I don’t use Emacs. I use enough other programs that Emacs sets up loads of destructive interference that damages not only my text editing productivity but also my productivity in unrelated tools like spreadsheets or word processors or even web browsers. I’m not willing to do everything in Emacs (and I know you can!) so I’m not willing, as a result, to do anything in Emacs. It’s just too alien.

Now the same can be said for my editor of choice: Vim. It is, in many ways, even more alien than Emacs. It is, however, significantly simpler (being a much more limited tool!) and thus easier to partition away into “I’m doing Vim so I do this; I’m not doing Vim so I do that”-style handling.

Eclipse is almost as bad. It uses standard GUI conventions, but it pretty much demands that you do anything development-related in it. You must use it for editing, debugging, project management and pretty much everything else. If you don’t, well, lots of luck figuring out where it tosses things. (Its configuration files are a nightmare and I’ve frequently found myself changing things that aren’t reflected in the IDE itself for reasons I couldn’t fathom.) It’s not as bad as Emacs, but it’s still sufficiently bad that I don’t use it. I’d honestly rather have a window for editing (vim), a window for a REPL (if applicable) and a window for executing my builds, tests, etc.

This is going on too long…

Let’s face it, the world of software development tools is in a really bad way. As much contempt as programmers show their end-users, they show even more for their own kind. (I’m pretty sure a good Ph.D. thesis is awaiting anybody who wants to study this self-loathing.) The real tragedy here is that everybody’s productivity would be enhanced if everybody worked only a little bit more on the usability aspects of their software development products. As things stand now, however, development tools are almost the epitome of user friendlessness.

Entertaining maths

There’s loads of talk about corporations and how they’re stupid and/or incompetent. There’s then wonderment expressed at how this can come to pass in a competitive environment that should filter out incompetence quite viciously. This amuses me to no end because I know that pretty much by definition any large, hierarchical entity—corporate, governmental or otherwise—must be incompetent. Any claim that this isn’t true is risible on the face of it.

The model

We will model here a fairly typical mid-sized corporation on its way to growing into a large-sized one. In such a corporation it’s not uncommon to have about 13 layers from the lowest worker to the CEO and/or board.

Flowing uphill

Information typically flows from observations of the workers to their supervisors. Supervisors collate that information and pass it up to foremen. Foremen summarize that information up for their lower managers. The lower managers compile it into reports for their middle managers. This process of communication goes up and up through twelve transitions before it reaches the CEO in some summarized form.

Flowing downhill

Now based upon this summary of observations communicated through the chain of the company’s hierarchy the CEO makes an executive decision. This decision is sent down to the vice presidents as policy. The vice presidents translate the corporate policy for their departments and ship it down ot the directors as, say, a road map. The directors take the road map and build an action plan from this. The action plan is further subdivided and elaborated upon with greater detail all the way down the chain until such a time as it turns into specific tasks for the workers at the bottom.

Stimulus/response

This whole system, in the end, is a stimulus/response chain passed through a hierarchical control system where each person is a node in said system that acts as both a sensor (uphill) and as an actuator (downhill). When a stimulus is sensed, the signal is transmitted up, a decision is reached and the actuators are activated on the way down to respond to it.

Only…

Signal loss rears its ugly head

Let us make a flattering caricature of corporate communication. This is the kind of caricature where the flabby jowls of a chubby guy are turned into the chiseled jawline of a romance novel cover model or where the frumpy and slightly plump middle-aged woman is drawn as a Hollywood hottie.

Let’s assume, in short, that the accuracy of communication through each layer is 95%. I think anybody who’s ever spoken with another human being one-to-one knows that this is a ludicrously high percentage of signal fidelity, but I’m going to pretend that I believe that corporate communications is just that efficient. This means that each transition layer that information flows up the hierarchy has a 95% chance of being transmitted correctly. Now with a 13-layer entity, that’s 12 transitions up the chain. 1 layer has a 0.95×100=95% chance of getting the information right. 2 layers has a 0.95×0.95×100≅90% chance of getting it right. (To keep things easier this is will be written like 0.95↑2×100 from now on.) 3 layers is 0.95↑3×100≅86% and so on until we go 12 layers at which point the odds are 0.96↑12×100≅54% chance of the information being transmitted correctly to the CEO.

Read that again. The odds of the CEO getting the wrong information from the hierarchy below him are nearly half. Almost half the time the CEO is getting bad information from his employees, and this is with the Hollywood hottie/romance novel cover model version of communication! (Remember, 95% accuracy is ludicrously high for human communication!)

But it gets worse. The CEO, based on this information (which, recall, is wrong almost half the time!), makes his decision and communicates it downward. This downward communication works at that same 95% accuracy. That means the round-trip stimulus/response has a 0.95↑24×100≅29% chance of providing the right response to a given stimulus.

The scary implications.

Again, look at those numbers. Almost three quarters of the time the company will do the wrong thing. And this is with the pert-breasted/chiseled-jaw version of corporate communication!

Now let’s have the breasts sag just a bit and the jowls get just a bit flabby. Let’s assume that corporate communication is 75% accurate (which is still ridiculously optimistic, note!). The odds of the CEO getting the right information from the hierarchy under him are now 0.75↑12×100≅3% and the odds of a proper response to a given stimulus now drop to 0.75↑24×100≅0.1%.

That, my dear friends, is why large companies always come across as clowns who do the dumbest thing possible whenever they’re given a chance to. They simply cannot be any other way.

“Solutions”

Now I can already hear some of the objections. “But Michael,” some are saying, “no company forces all information to flow from the bottom to the top and back down again. Managers are given leeway and discretionary power. This offsets the problem.”

While it is true that some managers in some companies are given some measure of independent authority (it’s a lot less common than you think, however!), this doesn’t solve the problem. It just means that the incompetence is focused on precisely the most important problems. Such a corporation will, on minor issues, make what looks like smart decisions frequently. This will lull them into a false sense of security while important decisions that do travel the whole stack fall through undetected. This could be even more devastating than having all decisions done stupidly because nobody would expect that degree of stupidity. (I personally suspect that a lot of sudden corporate deaths stem from this kind of false security.)

Other objections include the objection that not all miscommunication causes things to be diametrically wrong. It could be a small mistake. While this is true, the larger the drop in signal (70% is a pretty huge one, not to mention 99.9%!) the lower the chances are that the response to a given stimulus will be even close to what’s required.

“But Michael,” others will now comment, “any mistakes can be corrected as the corporation assesses and reacts.” Well, any such assessments and reactions also travel the communication stack. They’ll be just as incompetently executed. The company is just going to lurch from crisis to crisis and getting farther and farther away from anythign resembling balance.

Also, another factor enters here that I haven’t yet addressed: communication lag. There’s not only signal loss to contend with, there’s retransmission delay. Even if by some fluke the CEO gets the right information, by the time it reaches him it’s days to weeks to even months too late.

There’s one final objection that has a patina of truth to it: CEOs don’t rely just on their hierarchy to get information. They have consultants, the press, etc. as well. That sounds good in theory, but think about it for a minute: that’s more layers and more signal loss. This is going to exacerbate the problem (especially when the CEO starts to get conflicting information!), not mitigate it.

Nope, I’m sorry, but if you’re a large, hierarchical organization you’re just doomed to be stupid.

And evil. But that’s a rant for a different time.

A tale of two Prologs

It’s conventional wisdom in Prolog circles that you should, if you’re using free tools, do your development work in SWI-Prolog (because of its wonderful environment and tooling) and then grit your teeth and port your code to YAP for performance because YAP is just faster than SWI-Prolog. Now I have no reason to doubt this conventional wisdom, but being the kind of guy I am, I always have questions like “how much faster?” and “how much teeth-gritting?” These questions are actually astonishingly hard to answer.

I’ve looked under chairs / I’ve looked under tables

The problem, you see, is that finding a decent set of benchmarks for Prolog is fiendishly difficult (for me, I mean: I’m sure there’s people out there whose search-fu is so good they can find a million detailed Prolog benchmarking suites in only a few seconds). There are two main reasons for this:

1. Prolog and portability are still not really on speaking terms after all these years. You know, despite there being an ISO standard for it.
2. Much of the Prolog space is taken up with commercial offerings whose marketing literature boasts of performance; actual hard data could be embarrassing to these claims, so there’s not a lot of incentive to make a comprehensive suite of benchmarks.

Still, I did eventually find a benchmark suite for SWI-Prolog and decided to test two things with it:

• How hard is it to port code from SWI-Prolog to YAP?
• How much faster is YAP than SWI-Prolog?

I had my own guess for that second one: I figured that YAP was likely going to be about 50% faster than SWI-Prolog on average with perhaps a range of 10% faster to 70% faster. I thought, in short, it would be along the lines of various C compiler differences.

Porting

The benchmark suite is about 7000 lines of Prolog code doing a variety of micro-benchmarks. It turns out that to add YAP support, 11 lines of code have to be changed. (The code in question deals with the guts of the implementation—peformance statistics—so it’s quite reasonable to expect these not to be compatible.) To be specific, the following code:

 
ntimes(M, N, T, GC):-
    statistics(gctime, GC0),
    statistics(cputime, T0).
    ntimes(M, N),
    statistics(gctime, GC1),
    statistics(cputime, T1).
    ntimes_dummy(N),
    statistics(gctime, GC2),
    statistics(cputime, T2).
    T  is (T1-T0) - (T2-T1),
    GC is (GC1-GC0) - (GC2-GC1).
 

…had to be changed to:

 
ntimes(M, N, T, GC):-
    get_performance_stats(GC0, T0),
    ntimes(M, N),
    get_performance_stats(GC1, T1),
    ntimes_dummy(N),
    get_performance_stats(GC2, T2),
    T  is (T1-T0) - (T2-T1),
    GC is (GC1-GC0) - (GC2-GC1).
 

 
get_performance_stats(GC, T):-
    current_prolog_flag(dialect, Dialect),
    get_performance_stats(Dialect, GC, T).
 

 
get_performance_stats(swi, GC, T):-
    statistics(gctime, GC),
    statistics(cputime, T).
 

 
get_performance_stats(yap, GC, T):-
    statistics(garbage_collection, [_,_,TGC]),
    statistics(cputime, [TT,_]),
    GC is TGC / 1000,
    T  is TT / 1000.
 

That is a rather impressively small change (a change, incidentally, that makes testing other Prolog environments that much simpler). Add one simple bash script and I’m ready to go:

 
#! /usr/bin/env bash
scale=1
time yap -l run -g "run($scale), halt."
time swipl -s run -g "run($scale)." -t "halt."
 

Note that I wrapped the whole execution up in a call to the time utility as a redundant sanity check.

Results

I found the results rather surprising, to be honest. As I said, I expected the average improvement to be perhaps 50%. Look at the real numbers:

Benchmark	SWI-Prolog (s)	YAP (s)	Ratio
boyer	0.685	0.190	3.6
browse	0.562	0.220	2.6
chat_parser	0.949	0.430	2.2
crypt	0.682	0.130	5.2
fast_mu	0.736	0.250	2.9
flatten	0.771	0.300	2.6
meta_qsort	0.627	0.290	2.2
mu	0.641	0.260	2.5
nreverse	0.517	0.110	4.7
poly_10	0.637	0.250	2.5
prover	0.737	0.380	1.9
qsort	0.657	0.230	2.9
queens_8	0.608	0.110	5.5
query	0.641	0.160	4.0
reducer	0.758	0.330	2.3
sendmore	0.755	0.120	6.3
simple_analyzer	0.797	0.330	2.4
tak	0.792	0.340	2.3
zebra	0.690	0.310	2.2
Total	SWI-Prolog (s)	YAP (s)	Ratio
user	13.320	5.438	2.4

BU2B

There’s a lesson to be learnt from all this, I think. For starters the big take-away lesson for me is that assumptions can be bad. I would never have guessed that SWI was on average 2.4 times slower than YAP with a range of 1.9 to 5.5 (!) times the speed by just switching compilers. I also have to take away the lesson that the talk of YAP and SWI working together to ensure as much compatibility as possible is not just empty talk. This porting job was miniscule.

Now I know these are microbenchmarks and thus not really worth much, but this leads to take-away lesson #3: TEST before you make any assumptions. It will save you unpleasant surprises in the future.

So it seems that conventional wisdom was right. For this set of benchmarks at least.

How to ride a bus

I’ve ranted about how to drive a bus before. Today is different. I’m going to teach you how to ride a bus. Specifically I will teach you why you need to erase the preconceptions that you have—and that insidiously poison every thought you have if you’re not careful—if you wish to truly learn (or teach!).

Back-story

A friend of mine has lived in Japan for many years. We’ve spoken (online) at length of the differences in living in “the East” vs. life in “the West”. This is always a fruitful area for exploration because things are often done differently in “the East” and those differences can often be surprising to those brought up in “the West”. In conversations with him I found the perfect illustration of these differences and, at the same time, an important lesson to all who think themselves smart. I’ll let him take over here.

Benjamin’s story

Outside of Tokyo, the buses are not yet westernized. They have the entrance at the back, the exit at the front. There is a ticket you pick up when you enter (at the back). Just like when you visit a doctor, a number is on the ticket. The number increases each stop so the ticket shows which station you entered at. Next to the bus driver at the top there is a large display that shows all the tariffs for each number and they go up as the bus continues on its journey from stop to stop. So if you leave, you can see exactly what your fare is at the time, as can the driver when you get off.

The arrogance of western visitors

The first time most westerners encounter a system like the above the reaction is “how odd”. Then the mind closes and it’s viewed as “silly” or “stupid” or whatever. The assumption is that the western system is superior in some way that is “obvious” yet… well… indefinable. And the reason that it’s indefinable is because it’s not superior in any way, shape or form.

Consider the advantages:

• it’s cheating-resistant;
• you always know what you’re going to pay;
• the driver’s handling of payments is easier;
• if you change your mind mid-journey, you pay only for what you’ve used of the system up to that point.

Ottawa’s alternative

The system I had access to in Ottawa was inferior on all counts:

• you had to pay the same fare whether you were going one stop away or one hundred;
• if you had to change buses a complicated (yet easily-gamed) system of transfers was used;
• the driver had to know every possible transfer code—and these changed daily because of the gaming possibilities—that could work on his bus.

Wuhan’s alternative

The system here in Wuhan, apparently based on some French city’s system, is bad on other grounds:

• you pay per bus, period;
• the fare is fixed, like in Ottawa;
• there is no transfer system, however, so if you have to travel ten stops but change buses twice, you’re paying three times the fare of someone who’ll be travelling a hundred stops on one bus.

At least you can pay by waving a card at a plate, though. That’s way ahead of Ottawa.

Both systems, in short, are obviously inferior to this “pre-modern” Japanese system, yet the reaction of people encountering it is to deride it reflexively as wrong and somehow “silly”. This has extended to the point that the Japanese, to be “modern”, are actually switching to an inferior system just to be more “western”!

This applies to more than buses… or to Japan

There are some lessons to be taken away from this bus system. We carry an awful lot of baggage into any situation. In most cases (like bus systems) this isn’t a problem, but when it comes to some very important circumstances (social activism and engineering both spring to mind here) it is very important to be wary of that baggage.

The lesson is to follow three simple steps.

1. Yes. It’s different. Believe it or not, things can be done differently from how you think they should be.
2. Don’t dismiss things out of hand because they’re different. Stop. Think. Analyze.
3. After your analysis, be honest: consider the very real possibility that what you’re used to—what feels “natural” to you—might, in fact, be inferior.

The lesson for software

Yes, this is another one of my software rants, albeit in muted form. This kind of baggage is especially prevalent in software, of all “engineering” disciplines (remind me to explain at some point in the future why I laugh at the notion of “software engineering”), where there’s more cargo cult nonsense than actual thought than in any other industry I’ve worked in or alongside. (Evidence: the whole curly brace blight.)

When you hit a new programming language, for example, don’t suddenly decide that begin…end means that a language is useless because it involves typing six more characters (or four more keystrokes) than {...}. Even more importantly, when you hit a language that’s confusing at first glance, say one in a paradigm you’re unfamiliar with (which, if you’re like the Chamber’s Constant of developers means any paradigm other than imperative/OOP), don’t discard it because it doesn’t use loops and explicit branching. See if maybe the approach used isn’t perhaps superior in some way in at least some problem domains.

The lesson for social activism

Social activists in particular are arrogant in their assumption that what they “know” is superior to what others may know. I see them coming to China all the time to “teach the locals”—they “teach” Christianity, they “teach” democracy, they “teach” Objectivism even (!)—and they find out that their messages fall on deaf ears because what they think of as “obviously superior” is thought of as obviously inferior to the locals. These people come to China to “teach the locals how to do it right” and leave frustrated and angry at the Chinese because the imagined mass flocking to their idology they were expecting didn’t happen; because the “intransigent” locals just won’t do things the way the visitors think they should be done.

It never seems to dawn on them that the locals don’t want the new way because the old one is, at least in their eyes, superior.

To be a successful social activist you’re going to have to remove your ideological blinders at times and try to see things from the others’ perspective. And you’ll have to, on occasion, and likely far more frequently than you’re comfortable with, confess that perhaps the other side’s approach is superior.

The take-away lesson

We all have blinders. Every last one of us. These blinders lead our thoughts in well-travelled ruts that are often so deep we can’t even see over them. In many cases—like bus systems—this doesn’t matter much. A stupid bus system isn’t the end of the world. In some important cases, however, like engineering or social activism, these blinders can be a problem that leads to failure. We can’t avoid the existence of these blinders, but we can at least try to mitigate their impact. When you encounter something different:

1. Recognize that it’s different and move on.
2. Analyze the differences instead of reacting with knee-jerk negativity.
3. If the different way is better, admit it, even if it’s only to yourself.

Brewing tea for fun and consumption of copious quantities of time

The occasional visitor to or expat in China gets called something that translates to “more Chinese than the Chinese”. Context and tone are very important when you hear this phrase. It can be used in a derisory sense as it would be directed at, say, a westerner who walks around in Hanfu and tries, too hard, to act in the way he perceives the Chinese to act. It can, however, also be used in an admiring sense in which a non-Chinese demonstrates deep understanding of and appreciation for Chinese culture.

I have twice been called this in the admiring sense. The second time was in relation to making tea.

“Kung fu” tea

The term “kung fu” in China has little to do with martial arts (except that martial arts is one area in which one can show kung fu). Any skill that takes effort to master is, upon mastering it, showing “kung fu” (or, rather more accurately, “gongfu”). Making tea the right way is a skill that takes effort to master. There is, thus, a ”功夫茶” (literally “kung fu tea”) ceremony that is used to make, in my experience, the best tea even conceivable.

I thought I’d spend a bit of time today teaching how to make this tea. Of course, as with anything worth learning, this is going to be complicated, difficult and entirely worth your time to learn.

The tools of the trade

No, that photo is not of a set of torture utensils. That is a very complete tea set. For the full experience you will need a set about that complete. You can ditch the three figurines; you can lose the three-part cup at the right rear of the tray if you’re not making green tea; you can omit the bowl at the front left (although you’ll be happy if you have it).

In addition to the tea set, you will also need:

• a supply of good-quality water; it can’t be soft water and it can’t be too hard (spring water is ideal)
• a decent quality oolong or pu’er tea¹
• a near-infinite supply of patience

The pots

There are two pots in the picture: one with a lid, one without. The one with the lid is the brewing pot. The one without a lid is the serving pot. You will need both for the full ceremony. Both pots should be made of terra cotta (Yixing’s “purple clay” is traditional, but any terra cotta should do fine). In a perfect tea-drinking world you’d have a separate pair of brewing and serving pots for every tea you brew. Realistically this isn’t likely to happen. Keeping a separate pair for each major grouping is fine: one pair for oolong, say, one pair for pu’er, etc.

The cups

There are six cups in this picture (the ones with tea in them). They’re very small. This is intentional. The kung fu tea ceremony is predicated upon frequently-flowing small doses of tea.

The cups should be made of terra cotta to help preserve some of the heat, but the best cups are terra cotta on the exterior and white ceramic on the interior. This permits people to more easily assess the colouration and quality of the ensuing liquor.

The drip tray (and optional waste bowl)

There will be drippings. You need the slotted tray to catch them. Some are just a tray and need to be frequently emptied. Others have a discretely-placed hose to redirect drippings into a larger container. If possible you’ll want one of those.

The waste bowl is an optional component. It is used at the beginning as part of the process of cleaning if present and it is used during the ceremony proper to dump tea leaves if you make a lot of tea and have to recharge.

The filter

Even the finest teas and pots will leave some unsightly grit in the liquor. The metal thing in the coiled stand is a filter. You generally pour the tea from the brewing pot into the serving pot through the filter.

The aroma cylinders

The mysterious cup-like objects that are narrower and taller than the actual cups themselves are aroma cylinders. Use of them is optional (but highly recommended for the full experience). If you choose not to use them, simply skip the steps that mention them.

Assorted tools

The brush is used before starting to dust off the equipment. The towel is used during the ceremony to help keep the workspace tidy and free of unsightly puddles. Both of these are optional.

The wooden container with the implements of destruction sticking out are the tools of the trade. You’ll find a tea scoop (looking for all the world like an elongated baker’s scoop), a handle with a flattened end, another handle attached to a needle-like shape and a pair of oversized tweezers. The two middle components are for cleaning out the teapot (the flattened end is used to shovel out the tea leaves while the spike-shaped one is used to clean the spout). The tweezers are used to handle the components during the initial stages of the ceremony when all the equipment is going to be too hot to touch.

The ceremony itself

Lay out the tools in a way that facilitates access. Make sure you have a steady supply of hot water (for oolongs you want it to be ~90°C give or take a few, and for pu’er as close to boiling as you can get without actually boiling). You then follow these simple steps:

Phase 1: cleaning

Fill the brewing pot with hot water. Put on the lid and pour hot water over the outside of the pot. Transfer the water from the brewing pot to the serving pot, swishing it a little as it drains to help dislodge any grit, etc. Again spill a little bit of the hot water over the outside. Fill the aroma cylinders from the serving pot, spilling, as before, some of the hot water over the outside. (Arranging the cylinders in a ring and pouring in a circle is a good way to do this.) Using the tweezers, pick up each aroma cylinder and pour its contents into one of the cups. Again using the tweezers, pick up each cup in turn and dump the water, into the waste bowl if you have one.

As an option, in place of transferring the water from the serving pot to the cylinders, place all of the cylinders and all of the cups into the waste bowl and douse them with the hot water from the serving cup, supplemented, perhaps, with fresh hot water if necessary. After this you would recover them from the waste bowl (using the tweezers) and place them on the drip tray after ensuring they’re empty.

Phase II: washing the tea

Using the tea scoop, fill the brewing pot roughly 1/3 full (!) of tea leaves. Pour hot water into the pot about half-way and then place the lid on top. Douse the exterior of the pot with more hot water (I told you there’d be drippings!). After about 15–30 seconds follow the procedure as for cleaning the tea set above, steps 2–5, only this time you will definitely have to transfer from the serving pot to the cylinders to the cups and out. Don’t use the waste bowl for this.

When you transfer the wash to the serving pot you will have to use the filter to catch the inevitable grit. Get in this habit because you’ll be doing it for all future steps.

Do not assume the wash is optional. Failure to wash the tea will lead to a very unpleasant brew and to future brews that will be more unpleasant as the tannins soak into the clay of the pot!

Phase III: Brewing and serving

Open the brewing pot and fill it with fresh hot water. Fill it to overflowing. Any unsightly foam or floating tea leaves should be swept aside by the lid as you put it on. You then once again douse the exterior of the pot to keep it hot (and wash off any excess leaves or unsightly foam). Once again you transfer from brewing pot to serving pot (via the filter).

Phase IIIa: First-time serving

This step is only performed if you use the (highly advised!) aroma cylinders.

The first time you serve, pour from the serving pot into the aroma cylinders. Fill each aroma cylinder about 3/4 full of tea. Place a cup on each cylinder as a lid and then turn the whole assembly upside-down so the cup is on the bottom and the cylinder, with tea inside, is face-down. These should then be given a minute to build up steam inside. When the time is ripe, the imbibers should pick up the cylinders (the tea drains quickly into the cup) and quickly transfer them face up to their noses. The concentrated aroma of the tea is then available for assessment and enjoyment.

Phase IV: Enjoying the tea

Once the tea is in the cup, the drinkers should take the cups and start with a small sip, their noses practically in the cup for the aroma. Experiment with swishing around the liquor and breathing deeply to get the aromas all over the sinuses. Any decent tea will give a large complex of flavours which will change as more sips are taken and swished around before being swallowed.

Rebrewing

While the tea still has life (a decent oolong should last at least five brews!), repeat phases III and IV, bypassing the aroma cylinders. Generally drinkers will probably find either brew #2 or brew #3 (accounts will vary) the best, but all of the brews before the flavours grow flat will be a delight. Once the flavours do grow flat, if more tea is desired simply start over again from Phase II.

But what if I’m lazy?

The kung fu tea ceremony is simply the best way to make tea that has ever been created. Every step along the way enhances the experience of drinking the tea in some way. It is, however, as should be obvious, an extremely involved process that isn’t suited to a casual cup (nor to lazy people). So what do you do if you’re in a hurry or if you’re lazy? Well, do what I do. Buy yourself one of these and make your tea with it instead. It comes a fairly close second to the full kung fu tea ceremony in terms of tea quality, but is a whole lot more convenient.

Just remember that the key to good tea is in the wash (phase II above).

¹ You can use this ceremony with some modification with black teas or green teas, but I’m trying to keep this under novella length.

War of inner spaces

It’s rare that I have a dream whose contents survive the transition to consciousness. It’s even rarer for a dream of mine to not only survive mostly intact but remain compelling hours after waking. This is one such dream.

I think this idea, at its kernel, could make a really interesting science fiction story, role-playing game or even possibly a board/card game.

It all begins…

The story begins in the past. Decades of research into neurology, sociology and psychology leads to a new discipline: socionomy. (The neologism in question is coined by way of relating it to sociology in the same way that astronomy is related to astrology.) In socionomy we don’t just study societies, we know enough that we can shape them; we can program them, in fact, like we program computers.

A new world of joy, happiness, freedom and peace is heralded.

Fast forward a hundred years

The claims of socionomers are legitimate. Socionomy can allow you to program human societies with the traits you desire. The discipline is sufficiently difficult, too, that there are very few practising socionomers out there. Of course any software developer has already spotted the flaw. Sadly the sociologists, neurologist and psychologists who created the new discipline didn’t take enough CS courses to spot it.

Writing code, for mere computers, even, is hard and bugs creep in despite the best efforts of the coders. When you have a computer you’re programming, a bug is an annoyance that you curse under your breath over, then fix. When bugs are created in social “software”, however, you’re fucking with people. You only have one computer—society—and if it crashes, you’ve really got your work cut out for you.

Society has crashed at many times since socionomy programs were launched, at least portions of it have. Socionomers have been working frantically to patch the programs of their forebears, but…

Coding styles evolve

…there’s this problem. Socionomy being a new discipline, there were no known “coding standards” in place. Initial socionomy programs, although simple and limited in scope, were written in what essentially amounts to spaghetti code. Newer, more structured styles (the most popular being “Structured Memetics”) were quickly created, but by then it was too late: damaged, buggy and undecodable social programs were already out in the wild and causing damage. Increasingly new programs were built to just patch up the vital, but buggy, older programs.

Coding wars begin

On top of this, religious wars have started to divide the formerly close-knit circles of socionomy. Structured Memetics has reached its limits of complexity, but social problems requiring code have exploded combinatorically, helped onward by undetected bugs in earlier social programs. The already-small community of socionomers is fracturing into several new, untested approaches each with its own strengths, weaknesses and, most importantly, interactions—both good and bad—with other styles.

The entry point

Whatever form this idea takes (story, game, other), this would be where it begins. The world is a nightmare of badly-designed, badly-executed, badly-patched social programs written by increasingly desperate and frantic socionomers. In an RPG it would be up to the GM and players to decide how things progress. In a story, the next step would be the focus of a protagonist or two. In a card game the game play would focus on trying to stabilize an unstable system on the brink of total existence failure.

OK, that’s out of my system. I can return to sanity.

How to drive a bus

Many, many years ago, early in my tech career, before I could afford a car, I took the bus to work every day. This being Kanata (then a separate city from Ottawa) it meant a four-block walk to the bus stop followed, typically, by waiting at least 35 minutes for the bus that came every half hour. (Yes. 35 minutes or more for a twice-hourly bus. Did I mention this was the beginning station for the route? I didn’t? Consider it mentioned now.)

The bus driver

There was one bus driver who I seemed to be cursed to always get. The bus I needed to catch to get to work on time was his bus. This bus driver hated the public as far as I can tell. If you greeted him as you got on the bus you got ignored stonily on a good day; on a bad day you got a hateful glare that made you wonder if you were going to be next in a bizarre series of ritual murders.

This guy was a real piece of work. He glared at you if you got on the bus. He glared at you if you got off the bus. He turned beet red, looking like he was just this side of an aneurysm if you dared to signal your intent to get off at the next stop. Heaven help you if you delayed him for three seconds at a stop!

That’s what I did once, you see. I showed up late (for the first time since he’d started the route) and came running to the bus stop just as he’d thrown the bus in gear. He literally had to stop for three seconds to open the door, let me on, and carry on moving. This was too much for him.

“Next time come to the bus stop on time!” he scolded me.

My back went up. I turned to him and said, in a clear, loud voice, “Listen, asshole. You’ve got a real attitude problem given that without riders you don’t have a job.” I then proceeded to my seat.

To cut a long story short (too late!) he slammed on the brakes and refused to move the bus unless I got off. I refused to get off. The other passengers supported me, however, so his peer pressure thing didn’t work out. (We’d been comparing notes on this guy for several weeks, you see, and nobody liked him one iota.) The passengers filed off the bus, several of them giving me supportive pats on the shoulder while glaring defiantly at the driver. They just waited for the next bus while I waited for what came next.

His error

The rest of what happened (I got driven to work personally by the guy’s supervisor and he was never on that route again) is irrelevant to my point. My point is that this guy had the wrong attitude as a bus driver. He thought—incorrectly—that his job was to drive the bus. It wasn’t. His job was to drive people and the bus was the means to that goal.

By confusing what his job actually was he focused on the wrong things. Delays were interfering with his incorrectly perceived job instead of being, you know, the whole point of his job. I’m sure he thought he could do his job perfectly if it weren’t for all the customers…

And here’s where I get personal

This error is typical of people in the software industry. (That’s software industry. If you’re a researcher the status is obviously different.) Indeed I’d guess that the Chambers Constant (that’s 99.44%) of software developers share this guy’s attitude. “If only the users weren’t so stupid then I could do my job!”

Don’t believe me? Ask yourself this question: when was the last time you used a term like “luser” non-ironically when talking about the people who use your software? When was the last time you sat down and listened to a user complaint (say in a bug report) without rolling your eyes at the injustice of having to support such a dunderhead? If you’re like most of the people I’ve worked with or interacted with over the years—and that number’s well into the multiple hundreds scattered across three continents (North America, Europe and Asia)—you’re probably rolling your eyes now because you know what’s coming next and you want to get your defensive eye-rolling and dismissal going before you see the punch line.

What comes next

Because, you see, just like the job of a bus driver isn’t driving a bus, the job of a software developer isn’t developing software. The job of a software developer is solving user problems using software. Those “lusers” who irritate you so? They’re the entire reason you’re in the fucking industry. If you lose sight of that you’re no better than that asshole bus driver. Indeed, since you’re presumably educated and/or intelligent, you’re likely a worse asshole than that bus driver.

If you can’t cope with this fact do yourself, your co-workers, and your customers a favour and find another job. Software development just isn’t for you.

日本人 versus 中国人 cage match

There’s a famous pair of pictures found on a blog post at The Atlantic‘s web site. The first of the pictures shows refuelling operations at a Japanese airport:

The second shows refuelling operations on the same aircraft at a Chinese airport:

Both of these approaches are also reflective, in my opinion, of philosophical approaches to software development which I will call the 日本人 approach and the 中国人 approach respectively. (I will also refer to these approaches as “Order” and “Entropy” for reasons which will become clear at the end.) It is my considered opinion that both approaches, incidentally, are utter bollocks and create shit software.

Why the 日本人 approach sucks

Look again at the picture of the Japanese refuelling operation. Every little detail is accounted for. The two workers are wearing coordinated outfits right down to the bloody shoelaces for crying out loud! Surely they must have a well-thought-out system that is a marvel of efficiency, safety and effectiveness!

Well, no, actually. I mean I don’t know about aircraft refuelling, but I do know that in software a focus on fine-grained process that monitors every little detail of the creation of software is damaging to the outcome. Brooks noticed this too over 35 years ago. The statistic he cited was that 75% of software projects are deemed failures by the people making it. (I’d wager that the percentage is closer to the Chambers Constant—99.44%—if you were to ask the users of the software.)

There’s no end of methodologies for software development out there and every single one of them (even, perhaps especially, the ones that claim to free the developers!) lose the big picture in favour of a never-ending plethora of ever-irrelevant details.

I’m sure anybody who’s ever coded in an enterprise environment has seen the software equivalent of colour-coordinated shoelaces. (Mine was being told I wasn’t allowed to use Python scripts to generate my boilerplate code because not everybody in the company knew Python. Note that this was a code generator I wrote for my own use that was not part of the build process in any way, shape or form…) I’m also sure that such people will admit, grudgingly or openly, that the software quality wasn’t appreciably improved, but the quality of the workplace was noticeably degraded.

So it’s the 中国人 approach

Not so fast. Coding anything non-trivial is hard work and it’s complicated. Human beings are limited in their cognitive capabilities and, as a result, we suck at complexity. This is why we make teams for things. And once we have teams we have a need for constancy, regularity, communication and all the other things coders suck at. I’ve worked in shops that had, for all practical purposes, no controls and the results were easily as ludicrous as—sometimes worse than—the shops that had B&D-style processes.

The problem, in the end, is that cowboy coding is like that guy in the second photo. Sure you may metaphorically get the plane fuelled, but you’re getting a metaphorical mouthful of avgas in the process. If you’re lucky. If you’re unlucky you’ve got burning fuel spilled all over you instead.

Cowboy coders produce shit. It’s often inspired shit, but it’s shit nonetheless (in terms of reliability, expandability, utility, etc.). I submit the aforementioned Chambers Constant of projects at SourceForge and Github as my evidence of this.

So the solution is…

You think if I had the solution I’d be writing a blog about it? Fuck no! I’d be making millions fixing the software world and letting computers finally live up to their potential! I have no answers, but I do have an idea what the answer would look like.

I like the works of Michael Moorcock. (Yes, this was a jarring segue. Deliberately so. Suck it up.) While he’s weak at plotting and his characterization can be charitably termed weak, his strengths outweigh these weaknesses enough that I’m a fan.

One of the concepts I’m a fan of is his cosmology of the eternal struggle between Order/Law and Entropy/Chaos. In Moorcock’s writings too much order is as bad as too much disorder. Excessive Law results in tyranny and stagnation. Too much Chaos results in continual destruction. The solution in Moorcock’s world view is a metaphysical “Cosmic Balance” that keeps the two opposing forces in check.

We need some kind of software development balance to allow us to combat the creeping control freaks behind things like the Booch method or its ilk whilst at the same time not having our software fall into the more typical anarchy that is the norm in projects (despite the increasing clamps placed on the work by management). When a business leans too much toward process, it needs a boot back to the middle. When it leans too much toward cowboy coding, someone needs to rein in the cowboys and crack the whip a little.

And we need an instance of the Eternal Champion in every software shop to boot things around until balance is finally achieved.

Profanity amuses me

Really, it does. And I don’t mean in the sniggering internal 12 year old sort of way. (OK, I don’t mean just in the sniggering internal 12 year old way.) I mean the very concept of profanity amuses me at a profound level.

Picture, if you will, a party full of ordinary, middle-class people. Boring, isn’t it? And I think that dip is from a can. But I digress…

Got that picture in your head? Good. Now picture what happens if, in one of those sudden lulls in conversation that happens on occasion, your voice is heard loudly and clearly saying, “I heard that John had vigorous sexual intercourse with Mary last night.” What would the reaction be? There might be some nervous tittering. A bit of the old “Ew! TMI!” reaction perhaps. What you wouldn’t get much of, however, is raw shock.

Now let’s rewind to that lull, but this time instead you say, “I heard that John fucked Mary last night.” What’s the reaction this time as the tape moves forward? My prediction is that you’d get a rippling ring of shock spreading through the room with you at its epicentre. The shock would be utterly visceral and it would be focused purely on the specifics of your word choice, not on what you chose to communicate. Despite, you know, you having just said exactly the same thing!

It is this which amuses me. It amuses me when others react with such stark negativity to certain key words and it amuses me even more when I catch myself reacting that way. I have a pet crackpot theory to explain this, too: I believe that the notion of profanity—the notion that words have intrinsic properties, divorced from the meaning communicated, that are somehow directly harmful—is the last vestiges of magical thinking left to a culture that claims to be rational.

Sometimes it’s just fun to believe in magic.