Checkbox

Hi xkcd visitors! Today we released "Checkbox", our 2021 April Fool's project. I proposed the original concept for this one and built the frontend. The backend and editor tooling was made by davean and Kevin Cotrone, with content written by a collection of amazing folks credited in the header of the comic and the one and only Randall Munroe.

This year was a doozy. We specced and scrapped several different ideas in the months leading up to today. We finally settled on today's concept just 3 days ago. The need to do something simple was a really useful constraint, and we leaned into the idea of making something primitive but deep. I'm really proud of how it turned out!

Here's a few stories from development.

Morse all the way down

If you take a peek at the JS code, comic.js, you'll find that it's written in morse code!

".---- -.-. -.. - .. ... ...-..- -.-.-- ..--- . ---.. ..- -.-- -. ----- -----
----- ---.. ---.. ..-. ..--.- -.... .---- --... ---.. .--- -..- ..-. ....-
-..-. -....- .-.-.- --- ... ..--- --. -.-.-. .---- .-.-. -..- -. .--.-. -----
.-. ...-- -.--. -.- -.-. --.- ..- -....- .--. ...-..- .... --.. .---- ..-.
-.-.-. ---.. . -.-. -... --. ----- --- ...- .. ...- .-... ..--.- ---... -.-.--
.. ..... .-- -.--. ----. -.-- -. .--. .---- .--- -...- ... -..-. ---.. -..
<snip>
-.-.-- .... -... -. ..... -..-. . - .-..-. --..--".split(";D").map(morse.run);

When you paste the code into a morse translator, unfortunately the result is not very helpful:

1CDTIS#!2E8UYN00088F#6178JXF4/-.OS2G#1+XN@0R3(KCQU-P#HZ1F#8ECBG0OVIV&#:!I5W(9YNP1J=S/8...

What's going on here?

Initially, I'd hoped to faithfully translate the JS code to morse and back. When I started implementing this, I noticed that my character table did not contain curly brace characters. Hmm, this was going to be a problem.

I considered adding new characters to the morse table, but this seemed like it could become tedious, and ran the risk of overlapping with other more obscure morse sequences I wasn't aware of. My next idea was to interpret sequences of 8 dots and dashes as binary bytes, which seemed like a pretty good solution, though a bit verbose. I took a break from thinking about this, and then had an idea: it's pretty common to encode arbitrary data into letters and numbers, so would something like base-36 work?

Now we were getting somewhere! As a slight enhancement over base-36, I pulled in the excellent base-x library, which accepts an arbitrary list of characters to encode/decode binary data into. I threw our 56 character morse code table into it, and we were in business!

So, the JS code is actually encoded twice: once in base-56, and then in morse code. If you count UTF-8 – which actually came into play when I needed to add an emoji to the source! – it's running through three layers of encoding. :)

A UI challenge disguised as a statistics problem

A major design challenge from the outset of this project was accurately interpreting users' morse code input. Morse code is based on tapping out a rhythm with a base tempo for dots, multiplied by 3 for dashes, and 7 for spaces. Humans are not great timekeepers, especially unpracticed ones – and our assumption was the majority of our users would be tapping out morse for the first time. Given a sequence of checkbox presses, how do we determine the tempo they intended to be morsing at? This is a pretty interesting and hard to solve problem!

My initial prototype of the checkbox interaction was in the form of an Observable notebook. I wanted to get something put together quickly to make sure the experience of tapping out a conversation with a checkbox was compelling, and most importantly, actually usable! The experience would be ruined if it wasn't possible for a new user to learn how to communicate with it.

This initial prototype uses the average time period between presses and releases to determine the base tempo. With a little tuning, it worked, but it had two significant issues: firstly, it has no way of knowing whether the initial presses it's seeing are dots or dashes. If the input skews towards dashes, the threshold will be off, and dashes will be interpreted as dots. The second problem is that as more dots and dashes are observed and the threshold is adjusted, the morse decoding changes over time (you can see this in the Observable prototype).

The first thing I tried to increase input accuracy was tune the prototype with really lenient thresholds. I increased the pause time allowed between characters because in my own testing, I found myself looking back and forth between a morse chart and the screen, and I was feeling rushed. I later discovered this is called Farnsworth Timing, and it was invented in the 1950s! Even so, the prototype was undeniably unreliable and a little frustrating to use, even though I could watch tne debug data as I tapped.

Originally, I didn't envision anything in the comic besides the checkbox – the utter minimalism was a part of the gag. However, actually using the checkbox convinced me we'd need to somehow show users their input. Otherwise, it'd be too easy to typo a letter unknowingly and get a wrong response from the right intent – a super frustrating experience. I didn't want users to have to reach outside the confines of the comic (e.g. looking at the console, or a user script) to use it. Whatever we did, it had to be ergonomic. The fact that we might not be able to interpret a morse input until more characters had been entered made giving users feedback on their typos much more difficult.

In the final day as I prepped the JS code, this problem remained unresolved. I asked davean for his algorithm wisdom on a quick and adequate way to calculate the time period. He suggested using k-means clustering to find the dot, 3x dot, and 7x dot clusters in the input data. He had also considered implementing fuzzy matching on the server side to allow for input errors, but there simply wasn't enough time to do that justice in a couple days.

It was a couple hours before midnight on Mar 31, and I was finally ready to start on a k-means implementation. I felt that the prototype algorithm would be good enough for the first few hours of the comic being up, and I could follow up with a better recognizer when it was ready. And yet, the issue of indicating typoes still daunted me. Even with a better way of analyzing the timings, until the user had input at least a dot and a dash, the best we could do is apply a default tempo and guess. If we couldn't reliably show users the impact of their actions, we couldn't effectively teach them morse code.

With an hour or so left I decided to focus on adding some kind of "HUD" that would indicate to users if they had just input a dot or a dash. To do that, I needed a deterministic recognizer. So in the end, I just set a fixed tempo. These two choices together were a breakthrough! The problem all along was that users might not have a consistent timing in their input – but the UI could show exactly when a dot became a dash, or when a pause between presses formed a space. Even the animations could help reinforce the rhythm, by making them multiples or divisions of the dot period. The characters slide to the left in sync with the ideal tempo, so it feels very natural to tap in sync with that.

I'm really proud of how this final interaction turned out. The HUD takes an incomplete sequence of presses, extrapolates the effect of the user pressing at the current time, and encodes as morse. A crucial win is this not only makes it crystal clear when your dot becomes a dash, but also when a gap between presses becomes a space, which we can indicate before the user touches the UI again. This works great for users composing their messages using a code table or pencil and paper, because they can see the screen fill in with their intended patterns.

Another cool outcome: the comic is far more lenient than traditional morse code. A user merely needs to wait until the preview displays what they want. It's not necessary to have consistent timing: you can input consecutive dots much faster than equal timing, and dashes can be any length as long as they reach the minimum threshold. This flips the task from keeping time to a WYSIWYG model, of sorts.

I think it's really interesting how this initially appeared to be a statistics and data analysis problem but turned out to be solved in a much better way by UI. This is particularly exciting to me as a generalist, because I often have an internal dialogue running like:

"What if I was better at this specific discipline? Does knowing a little about a lot of things really make a difference when the complexity I can tackle in any particular field is limited?"

This is an example where working different angles of the same problem helped me to understand which problem was really necessary to solve, which resulted in a better solution.

Unable to send the letter "E"?

Shortly before launching, we discovered a really strange bug: if you tried to send just the letter "E", you wouldn't see the intended response. The server did not recognize the input, even though we specifically handled "E". Also, two "E"s, or any other combination of letters, was working normally.

So we started digging into both the client and the backend to see what was going on. We took great pains to make the API for this project use morse code in the transport. If you take a look at the network inspector, you'll notice that the URLs requested have morse code in them:

chromakode@atavist:~$ curl 'https://xkcd.com/2445/morse/.../...._..' -v
> GET /2445/morse/.../...._.. HTTP/1.1
>
< HTTP/1.1 200 OK
< Content-Type: text/x-morse;charset=utf-8
-.. -... ...-- ...-- ....- .- .- -... -....- ----. ..--- .- .---- -....- .---- .---- . -... -....- ---.. ----- ----- .---- -....- ---.. -.-. .---- -.... ....- ..... ....- ..-. -... ----- ..--- .- / .... . .-.. .-.. --- -.-.-- / .- -. -.-- -... --- -.. -.-- / --- ..- - / - .... . .-. . ..--..

The letter "E", as it happens, is the simplest character to enter in morse code: it's a single "." – hence how we discovered this relatively obscure bug so quickly.

Back to the problem at hand. We noticed that the client wasn't sending the morse for the letter "E" along. Instead, the comic would request as if nothing had been entered at all: an empty path. Investigating a step further, I verified that the client was generating and requesting the right URL with the morse letter "E" in it! What was going on?

Then, an even stranger thing happened. I copied and pasted the correct URL into my browser and pressed enter, and right before my eyes, it deleted the "." from the end of the URL and returned a different result. This immediately set off alarm bells in my mind, thinking that this period at the end somehow made it a malformed URL. Could it be a quality-of-life feature to correct typos when a URL is at the end of a sentence? No, that didn't add up, because it had originally failed when requested by JavaScript, a context in which URLs are not human input.

Ok, next idea. Was there a spec somewhere that described URLs ending in a period? My googling eventually led me to this StackOverflow post by user "speedplane" (which at the time of writing has one vote):

"As other answers have noted, periods are allowed in URLs, however you need to be careful. If a single or double period is used in part of a URL's path, the browser will treat it as a change in the path, and you may not get the behavior you want."

Evidently, a bare . or .. in a URL will be interpreted by browsers similar to UNIX filesystem paths: as referring to the current and parent directory respectively. This appears to be particularly relevant for relative URLs. So in addition to "E", the letter "I" (morse: "..") would also be affected.

We fixed this issue by adding an extra separator character before the period, making the urls /_. and /_...

If you're interested in learning more about how this was made, we'll be releasing the code on https://github.com/xkcd when it's ready. You can also "check" out the comics from previous years there. I'm partial to Alto, our infinite menu system. You can find more of my work on github.com/chromakode and @chromakode.

Slow news

For the past 3 years, I've been addicted to the news. My usual days are punctuated by checking the NPR Newscast (via the wonderful NPR One) every couple hours, reading articles linked through Twitter, Facebook, and Reddit, and listening to topic-focused news podcasts throughout the day as I exercise or do chores.

I've been reflecting on what motivates me to devote so much time and attention to news. I want to know what is going on so that I can feel connected to and orient myself within the world. The past few years have brought a dismaying array of social, economic, and political challenges that seem to only accelerate. Often, catching up with the news compensates for a feeling of restlessness from the mounting problems my country and planet faces.

In America, the dominant sources of news (media corporations and social media) are monetized based on advertising. Advertising demands attention. The more frequently and longer you engage with a broadcast medium, the more ad impressions result. Likewise, on social media, the more time you spend scrolling through feeds, the more sponsored content you'll consume.

I recall hearing a story about how food science is engineering delicious-tasting, empty food. Bland crackers and chips are enhanced with seasonings that create addictive, habit-forming snacks. Our taste buds love the flavor of these snack foods, but they provide little nutritional value. Reflecting on social media, I fear a similar effect is happening: we have engineered our information streams to be addictive streams of interesting but worthless information.

From 2011-2014 I worked at Reddit. I recognized the role addiction played in the product, and rationalized that we were channeling that addiction into teaching people things and exposing them to new ideas. I was addicted myself, and often interrupted my coding or thinking to feed my own info addictions. When I catch myself boredly scrolling through my feeds searching for something interesting, I sometimes wonder how I have come to so devalue my attention and focus.

Whether the addictive qualities of social media were intentional or an emergent quality, they are now actively cultivated because they produce business value. Designs and algorithms are optimized to increase the amount of attention users give to media products, and changes that reduce engagement are rolled back.

I believe that this is causing us to create information streams that are fundamentally unhelpful. Social media doesn't benefit if it provides information that is useful; the incentive is only to take our time. The design of infinite, randomly assorted feeds engages us all in the sisyphean task of trawling for what matters to us. There will always be more to see. There is no goal to satisfy that will end our search. We have optimized ourselves out of a sense of agency.

A similar dynamic exists with broadcast news. The more frequently we tune into our news sources, the more ad impressions will result. To keep us coming back, there has to always be a new story, a new update, new information to disseminate. So there always is. Similar to how an audio compressor makes everything sound loud by increasing the volume of the quiet parts, news keeps our attention by providing information with a steady level of significance.

As I've reflected on this, I've come to refer to this kind of news as "fast news". Fast news is Twitter, hourly radio broadcasts, cable news, and frequently newspapers. Fast news is pieces of stories as they develop in real-time, constantly becoming superseded by new information. Fast news is zoomed in on somebody somewhere else right now. Similar to social feeds, it's addictive because it's endless and inconsistent: the next important event may be just around the corner.

There is a serious problem with fast news: it turns us into spectators. With a focus on the immediate, fast news presents us with information that we can barely react to before it changes again. Fast news surfaces facts before they can be processed into patterns. Every story has a long tapestry of events and dynamics that preceded it, which we miss when we constrain our view to what's happening now. Similar to engineered foods, fast news feeds our desire for information without serving the reason we consume it: a need for information that is actionable and enables us to direct our lives.

Being able to receive and act on news as it happens can have tremendous value, such as in the case of natural disasters or organized protests. However if we look at the amount of time we invest in consuming fast news compared to how many choices it presents us with, there is an obvious disconnect. Ironically, by the time we receive fast news, it's usually too late to do anything about it.

An alternative could be news that exposes opportunities we can participate in. Behind every disaster is a years-long recovery effort. Behind every political move are larger influences, inspirations, and public drifts in sentiment. Slow news changes the focus from individual events to ongoing topics. By weaving events together into continuous stories about larger causes, those causes become things we can debate, influence, and change. Slow news is not about describing the present, it's about understanding the future.

Unlike fast news, the amount of slow news to consume is finite. If little in the big picture has changed, there is not much new to speak of. This presents a challenge: without a constantly-changing perspective, why will an audience return to it? A successful slow news source must justify itself by the practical value of the information it provides, not by how interesting it is.

Changes in the world, even from acute events, are usually the result of a long ongoing effort. It’s not possible to orient this effort at the pace of fast news. An ongoing value of slow news is to provide focus and context in the face of change. When our perspective is zoomed out, and not changing so quickly, more attention can be paid to the actionable aspects of ongoing, slower transformations.

How we think about ourselves and the world fundamentally changes how we function, and our information sources play an outset role in this. I am seeking to reduce the priority of fast news in my life while staying informed during these changing times. What sources are you following to accomplish this? Are there any that you would consider to be slow news?

I'd be interested to hear your thoughts. Drop me a line at slow.news@chromakode.com.

Isolated Core

Today I released Isolated Core, a technique for updating web apps in-page using iframes. I originally developed this technique while working on Euphoria because we needed a reliable way to push updates to long-lived clients. It's a simple architecture pattern with a lot of interesting benefits. Try it out and let me know what you think!

Backing up 18 years in 8 hours

This winter, while home visiting family, I took the opportunity to gather up all of my old hard disks and archive them. This amounted to the majority of my digital footprint for the first 18 years of my life. I'd been putting the task off for a few years, but the chance to explore the data sitting on these old drives (and the cherished computers they came from!) helped motivate this project.

When I was a teenager, whenever a hard disk needed replacement, I'd pull the old drive and shove it in my closet. There they sat, some for over a decade, until I turned them back on last month. Amazingly, across ~350GB of data, only about 500KB proved unrecoverable. Quite fortunate, considering that some of these drives were 15 years old, and one was removed with failing health checks.

In the process of recovering this data, I resolved to preserve it for the rest of my lifetime. Why go to all this trouble? Well, in my explorations of these old drives, I discovered far more meaningful memories than I expected. Old music and 3D renders, chat logs, emails, screenshots, and tons of old photos. The disks from the early 2000s were a reminder of the days when computer use gravitated around "My Documents" folders. Then I learned about Linux and always-on internet access arrived. I took a peek at my first homedir and found all of the little Python scripts I wrote as I learned to work on the command line.

By today's standards, the breadth and fidelity of these scraps is rather... quaint. A kid growing up today should have a fine pixel-perfect view of most of their digital trail as they grow up. That was another reason this project proved interesting: it was not just a record of how computers changed; it revealed how the ways I used computers and what they meant to me changed over time.

Here's a brief rundown of the tools and backup process I used, both because they will be useful to refer back to decades from now, and because they may perhaps be useful to others in their own backup tasks:

Archival process

IDE HDD -> USB -> Laptop -> External USB HDD

I used a Sabrent USB 2.0 to IDE/SATA Hard Drive Converter (USB-DSC5) to connect the drives to my laptop. I've found this to be a really handy (and cheap!) swiss-army knife for recovering old hard drives, especially since it works on both 3.5" and 2.5" drives. To store the recovered data, I used a 2TB WD My Passport USB Hard Drive (WDBBKD0020BBK-NESN). I've had good experiences with these drives in the past, and they have a great form factor. I ordered both items from Amazon and received them a couple days into my trip.

Reading data from the drives

To recover data from the drives, I used ddrescue. This is an imaging tool like dd that will record read errors and exhaustively retry the surrounding areas. Recovering a drive looked like this:

Copy data from /dev/sdc to disk.img (outputting a log of errors to ./disk-log):

$ ddrescue -d -n /dev/sdc ./disk.img ./disk-log

One of my favorite features of ddrescue is that you can re-run it at any point to resume where it left off or try to recover more data. In the initial run, I passed -n to skip the slow exhaustive scraping process, in hopes of getting as much data off the drives as possible in case they stopped working after running for while. Thankfully, no issues cropped up. If there were read errors during the initial sweep, I re-ran the process with a retry count:

$ ddrescue -d -r 3 /dev/sdc ./disk.img ./disk-log

In addition, I saved the partition table and S.M.A.R.T. data separately:

$ smartctl --all /dev/sdc > ./smart
$ fdisk -l /dev/sdc > ./fdisk

With the holidays over and all disks archived, I flew back home with the external HDD in my carry-on bag.

Cold storage in the cloud

Thanks to the advent of cheap cloud cold storage options like Amazon Glacier, Google Nearline, and Backblaze B2, it's now very affordable to dump a bunch of full disk images in the cloud. I chose Google Nearline for this task. Amazon Glacier is a bit cheaper (Glacier: $.007 / GB, Nearline: $.010 / GB), but retrievals are complicated to execute and price. Backblaze B2 is even cheaper, but only uses a single datacenter.

Before uploading my backups, I was able to shave off ~100GB (almost 30%!) by compressing with lrzip, which is specialized for large files with repeated segments. I also experimented with compressing one of the disk images with xz, but (as predicted by lrzip's benchmarks) xz took 22% longer to produce a file 10% larger.

After compressing the images, I encrypted them with AES256 using gpg. While I've typically used the default CAST5 cipher in the past, for this project I chose AES256 based on this guide. I considered generating a keypair for these backups: my plan was to create copies of the private key encrypted with a couple different passwords given to family members, etc. I decided to defer this because I didn't fully understand the crypto details and wanted to get uploading, so I ended up symmetrically encrypting the files. I may revisit this later and re-upload with a more granular key system.

Putting it all together, I assembled everything into a pipeline and ran it overnight:

for n in $(ls); do
  pushd $n
  lrzip -vv $n.img
  tar cv $n.img.lrz $n-log fdisk smart | pv | gpg --passphrase="$PASSPHRASE" --no-use-agent --symmetric --cipher-algo AES256 | gsutil cp - gs://$BUCKET/$n.tar.gpg
  popd
done

Waking up to ~250GB of memories neatly packed up and filed away was a lovely sight. I've been sleeping better since!

At my friend davean's suggestion, since lrzip is a less common program, I also uploaded a copy of the git tree to my Nearline bucket.

I also encrypted the files on my local HDD: while I used the out-of-box NTFS filesystem on the My Passport drive for the disk images in transit, once I had a copy of the files in Nearline, I reformatted the drive to ext4 with dm-crypt.

Update: an important final step (thanks to khc on HN for mentioning this): it's critical to test a full restore flow of your backups before leaving them to rest. In my case, I tested downloading from Nearline, decrypting, un-lrzipping, and reading. Similarly, for my local HDD copy, I tested mounting the encrypted filesystem and reading the images.

Finally, a word of advice when handling disk drives (and other objects you would not like to fall): objects that are already on the floor cannot fall further. Treat any object that is elevated from the floor like it will fall. You can increase the odds of this happening massively by haphazardly arranging your backup drives on swivel chairs and assorted hardware. ;)

A character encoding gotcha with React

Tonight I noticed that in a React 0.12 codebase of mine,   entities were rendering as "Â " in Mobile Safari. After a quick search I came across this StackOverflow answer which identifies the "Â " output as a UTF-8 formatted non-breaking space character being interpreted as ISO-8859-1.

To resolve this problem, putting...

<meta charset="utf-8">

...after my <head> element did the trick. While explicitly marking your webpages as UTF-8 encoded has been a best practice for a while now, I learned the hard way today that it's a requirement when working with React.

Interestingly, this problem was apparent in Mobile Safari on OSX but not Chrome on Linux. This made it present much later in QA. Another good reason not to leave the choice up to the browser!

Fun with the Myo gesture controller

Myo is a wireless armband that uses electromyography to detect and recognize hand gestures. When paired with a computer or smartphone, gestures can be used to trigger various application-specific functions.

When their marketing video made the rounds in 2013, I remember one specific demo made my jaw drop: touch-free video control. The video shows a man watching a cooking instructional video while cutting some raw meat. Being able to pause and rewind the video simply by raising his hand was a solution to an interaction problem I've had countless times, such as listening to podcasts while doing chores, or watching videos while eating a sandwich.

I ordered a Myo back in March 2013 and deferred shipment until their consumer design was ready. It was a nice surprise to return home from holiday travels to find a Myo waiting for me. :)

Unfortunately there is no official Linux support yet (though there's a proof of concept from a hackathon). On Windows and OSX, there's a pretty elegant Lua scripting environment in the SDK which is used to write "connector" integrations. Lua scripts are selected based on the currently active app to trigger mouse/keyboard actions from gestures. This is a neat approach. It enables developers and tinkerers to do a bunch of the legwork writing and designing integrations, while wrapping the complex parts (gesture recognition / mouse control / keyboard automation) in a cross-platform manner.

I was happy to see some web browser integration already built, but upon further inspection there were a few different behaviors which would be more to my liking. I was delighted to discover that I could simply open up the web browser connector and hack the high-level Lua code into the controls I wanted. I added a gesture to take control of the mouse, as well as some special cases for controlling video playback.

While the gesture recognition doesn't always work perfectly (probably a matter of training both myself and the armband better), when everything works properly, the results are pretty sublime:

I'll be posting my scripts and future tinkerings in a myo-scripts repo on GitHub.

Running Amazon Instant Video on Android tablets

In September, Amazon finally released an official Instant Video app for Android. Unfortunately, this app is not downloadable on tablets via their app store – only phones. Yet, Amazon Instant video has been supported for years on Amazon's own "Fire" Android-based tablets. It really bums me out that Amazon Instant Video continues to snub their competitors' products, but if you're like me and have a lot of purchases locked up in their store, there is a way:

I'd read in a couple places that if you install the APK manually, it works fine on tablets – as long as the phone version of Amazon's store app is installed as well.

I used adb (apt-get install android-tools-adb) to transfer both apps from my Nexus 4 to my Nexus 7:

First, Enable USB debugging on both devices.

On the phone:

Find the app ids:

$ adb shell pm list packages | grep amazon
package:com.amazon.avod.thirdpartyclient
package:com.amazon.kindle
package:com.amazon.mShop.android

Locate the relevant APK files (your paths may vary):

$ adb shell pm path com.amazon.mShop.android
package:/data/app/com.amazon.mShop.android-1/base.apk

$ adb shell pm path com.amazon.avod.thirdpartyclient
package:/data/app/com.amazon.avod.thirdpartyclient-1/base.apk

Fetch the APKs (using the paths we got):

$ adb pull /data/app/com.amazon.mShop.android-1/base.apk mshop.apk
5364 KB/s (29897881 bytes in 5.442s)

$ adb pull /data/app/com.amazon.avod.thirdpartyclient-1/base.apk avod.apk
3134 KB/s (1960268 bytes in 0.610s)

On the tablet:

$ adb install mshop.apk
5139 KB/s (29897881 bytes in 5.681s)
    pkg: /data/local/tmp/mshop.apk
Success

$ adb install avod.apk
3166 KB/s (1960268 bytes in 0.604s)
    pkg: /data/local/tmp/avod.apk
Success

Now you should be able to launch Amazon Instant Video on your tablet, watch some Star Trek TNG, and dream of a world where tech companies don't play silly games about which kind of PADD you use.

Notes from the development of xkcd's "Pixels"

Over the years, I've had the pleasure of hacking on the frontend code for a bunch of xkcd's interactive comics, including: unixkcd, xk3d, Umwelt, Time, Externalities, and Lorenz. This weekend, I was pinged about making something to coincide with the release of What If?: Serious Scientific Answers to Absurd Hypothetical Questions. The process of building "Pixels" was even crazier than our usual April Fools rush, and had the extra intrigue of being live during Randall Munroe's Colbert Report interview.

Here's a few anecdotes from the development of Pixels and a quick explanation of how it works. I hadn't worked with some of the graphics programming patterns (coordinate systems!) for a while, so I ended up making some classic mistakes – hopefully you can avoid repeating them. :)

background

Pixels is an infinitely zoomable black-and-white comic. As you zoom in, the pixels that make up the image resolve into smaller square comic panels – dark ones for black pixels, light ones for white. Depending on which panel you are looking at, the set of pixel panels will be different. These comics can be further zoomed into, revealing new pixels, ad infinitum.

view the source code on GitHub

Our schedule for the project was pretty absurd: we had 3 days from the first discussions on Saturday to going live Tuesday night. I was traveling between 3 different states during those days. With this crazy schedule, there was very little margin for error or engineering setbacks. I ended up sleeping for a total of 1 hour between Monday and Tuesday, finishing the zooming code on a plane flight home to SF via Gogo Inflight Internet. My plane landed in SFO a half hour before our rough goal of launching midnight EST. While my cab was hurtling me home at 80mph, the folks at xkcd were putting the final touches on the art and server code. We cobbled it all together over IRC and went live at around 1:30am EST.

On the art side, we decided on a panel size of 600x600, with pixels only pure black or pure white. Without grays for antialiasing, the images look a little crunchy to the eye, but this makes the zoomed pixels faithfully match the original image (we also thought the crunchiness would be a nice cue that this comic was different from the usual). On the tech side, HTML5 Canvas was the obvious choice to do the drawing with decent performance (I also tested plain <img> tags and found they were significantly slower). Unfortunately, browsers were still too slow to draw all 360,000 individual pixel panels via Canvas, so we had to compromise for fading the individual pixels in once they were 500%-1000% zoomed.

For this project, I chose to use vanilla JavaScript with no external dependencies or libraries. In general, we strive to keep the dependency count and build process minimal for these projects, since keeping the complexity level low gives them a better chance of aging well in the archives. Browsers recent enough to support Canvas (IE9+, Firefox, Chrome, Safari) are superficially consistent enough to spec to not require too much time fixing compatibility issues.

to infinity and beyond

One of the core challenges to Pixels was representing the infinitely deep structure and your position within it. As you zoom in, the fractal pixel space is generated lazily on the fly and stored persistently, so that when you zoom back out, everything is where you first saw it.

Each panel has a 2d array mapping: a pixel stores the type of panel it expands to, and possibly a reference to a deeper 2d array of its own constituent pixels. Appropriately, this data structure is named a Turtle, a nod to the comic and "Turtles all the way down".

To represent your position within the comic, it would be convenient to store the zoom level as a simple scale multiplier, but you'll eventually hit the ceiling of JavaScript's Number type (as an IEEE 754 Double, that gives you log(Number.MAX_VALUE, 600) ≈ 111 zoom nestings). I wanted to avoid dealing with the intricacies of floating point precision, so I decided to represent the position in two parts:

• pos: a stack of the exact panel pixels you've descended into
• offset: a floating point position (x, y, and scale) relative to pos

Here's a couple examples:

To render the comic, we locate the Turtle that accords to pos and draw the pixels described in its 2d array, offset spatially by the values of offset. As you scroll deeper, pos becomes a long array of the nested pixels you've entered, like [[305, 222], [234, 674], [133, 733], ...]. When you zoom far enough to view the pixels of a panel for the first time, the image data is read using a hidden Canvas element, and a new Turtle object is generated with the panel ids for each pixel.

One complexity of relying on pos for positioning is that it needs to update when you zoom into a panel / out of panel / pan to a different panel. When a pixel panel is zoomed in past 100% size, its location is added to the pos stack, and offset is recalculated with the panel as the new point of reference. Some of the hairiest code in this project came down to calculating the new values of offset.

There's a small trick I used to simplify the handling of the various cases in which pos needs to be updated. If the current reference panel is detected to be offscreen or below the 100% size threshold, pos and offset are recalculated so that the point of reference is the containing panel, as if you were zoomed in really far. This then triggers the check for "zoomed in past 100% size", which causes a new reference point to be chosen using the same logic as if you'd zoomed to it.

corner cases

Working out the browser and math kinks to simply position and draw a single Turtle took way longer than expected. It took me deep into the second night of coding to finish the scaffolding to render a single panel panned and zoomed correctly. Then, I needed to tackle nesting. Because the sub-panels are pixel-sized, you can't see the the individual pixel panels until you zoom above 100% scale. Since a panel at 100% scale takes up the whole viewing area, I initially thought this implied I'd only need to worry about drawing the pixels for a single panel at a time.

I then realized a problem with that thinking: if you zoom into a corner, you can go past 100% scale with up to 4 different panels onscreen:

This thought led me to make the worst design decision of the project.

TurtlesDown.prototype.render = function() {
  // there is no elegance here. only sleep deprivation and regret.

Brain-drained at 3am, and armed with the knowledge that there could be up to 4 panels onscreen at any time, I began to write code from the perspective of where those 4 panels would be. I decided to let pos reference the panel at the top-left-most panel onscreen, and then draw the other 3 panels shifted over by one where appropriate.

For me, programming late at night is dangerously similar to being a rat in a maze. I can follow along the path step-by-step well enough, but can't see far enough down the line to tell whetcher I've made the right turn until it's too late. With proper rest and a step back to think, it's clear to see what's going on, but in the the moment when things are broken it's tempting to plow through. Once I got 4 panels drawing properly, I realized the real corner case:

Consider that the top left panel has position [[0, 0], [599, 599]]. How do we determine what the other 3 panels will be? We can't just add one to the coordinates. We have to step back up to the parent panel, shift over one, and wrap around. In essence, we have to carry the addition to the parent. And, if necessary, its parent, and so on...

At this point, I needed to get something working and was too far down the path to reform the positioning logic into what it should have been: a descending recursive algorithm. Walking the pos stack and doing this carry operation iteratively ballooned into 40 lines of tough to reason about code. It's necessary to carry the x and y coordinates separately – this is something I forgot to account for in an early release, causing some fun flicker bugs at certain corner intersections.

I'm not proud of the render() function or how it turned out – but I'm really happy and somewhat bemused that when all is said and done, that nightmare beast seems to work properly.

"I'm not sure how this works, but the algebra is right"

Like many computer graphics coordinate systems, Canvas places (0,0) in the top-left. Early on I decided to translate this so that the origin was in the center, in order to simplify zooming from the center of the viewport (centerOffset = size / 2). A while later, I discovered that the simple ctx.translate(centerOffset, centerOffset) call I was using didn't apply to ctx.putImageData(), the main function used to draw pixel panel images. I considered two options: either I could figure out the geometry to change my zooming code to handle an origin of (0,0), or manually add centerOffset to all of my putImageData() calls and calculations. I did the latter because it was quick. That was a mistake.

What I didn't foresee was how much splattering centerOffset everywhere would increase the complexity of the equations. The complexity arises when centerOffset is multiplied by offset.scale or needs to be removed from a value for comparison. For example (from _zoom(), which ended up needing to know how to origin shift anyway!):

// ugh.
this.offset.x = ((this.offset.x - centerOffset) * this.offset.scale - scaleDeltaX) / this.offset.scale + centerOffset

In general, it's a good idea to do your translate operation as late in the chain as possible. Eliminating or externalizing as many operations as possible helps keep things understandable. I knew better, but I didn't see the true cost until it was too late...

Eventually, the complexity of some of my positioning code reached the point where I could no longer think about it intuitively. This led to some very frustrating middle-of-the-night flailing in which I sorta understood what I needed to express mathematically, but the resulting code wouldn't work properly. The approach that finally cracked those problems was going back to base assumptions and doing the algebra by hand. It's really hard to mess up algebra, even at 5am. However, this had the amusing consequence of me no longer grokking how some of my own code worked. I still don't understand some of the math intuitively, sadly.

TV audiences are mobile

All told, our launch went quite well. While there were some timing and performance issues we noticed the following day, it seemed to work for most people – a huge relief after the last 2 days. I spent Wendesday working through my backlog of minor fixes and improvements in preparation for the Colbert Report bump. Two tasks I triaged for release were proper IE support and mobile navigation. For IE, due to the lack of support for cross-origin canvas image reading, we needed to do some iframe silliness to get IE to work properly. Regarding mobile, I didn't think that phone browsers would perform very well on the image scaling, so I deferred it to focus on the best experience.

We anticipated a higher proportion of the Colbert Report referrals would be using IE, so we sprinted on that, getting it working just in time before the xkcd folks needed to leave for the interview. However, rushed nerds as we were (and ones who usually do not watch much TV), we didn't consider a very important aspect:

People watching TV don't browse on computers. They use their phones.

As I watched the traffic wave arrive via realtime analytics, my heart sank. The visitors were largely mobile! A much, much larger proportion than those using IE. Our mobile experience wasn't completely broken, but if we'd been considering the mobile traffic from the start, we'd have focused on it a good deal more.

berzerking works (sometimes)

When working fast, you have to resign yourself to make mistakes. Trivial mistakes. Obvious ones. Some of those mistakes can be slogged through, while some will bring a project down in flames. There's a delicate calculus to deciding whether a design mistake is worth rewriting or being hacked around. Having a hard 2 day deadline amplifies those decisions significantly.

I, like most developers I know, hate the feeling of a project running away from me; hacking blindly and not fully understanding the consequences. That's how insidious bugs are absently created, or codebases that need to be scrapped and rewritten from scratch. I prefer to take the time to recognize the nature and patterns to my problems and realize them with elegance.

Yet – sometimes, for projects small enough to fit fully in the head, and rushed enough to not weather a major time setback, brute force crushing works. For fleeting art projects that are primarily to be experienced, not maintained, that is perhaps enough.

When we work on interactive comics at xkcd, we take pride in experimenting with the medium. It's a privilege to combine forces with their masterful backend engineer + Randall Munroe's witty and charming creativity. Each collab is an experiment in what kinds of new experiences we can create with our combined resources at our disposal.

One of the fun aspects of working with comics is you don't expect them to think, to react to your behavior, to explore you as you explore them. That lack of expectation allows us to create magic. Every now and then, it's good to shake things up and push the near-infinite creative possibilities we have on the modern web. The health and sanity costs on these projects are high, but for me personally, novelty is the impetus to take part in these crazy code and art dives.

Overall, my favorite part of working on these projects is the moment after Randall's art & creative comes in: when I get to experience the project for the first time. Even though I know the general mechanic of the comic, when the backend, frontend, art, and humor all click into place simultaneously, it becomes something new. That first moment of discovery is as much a joy and surprise to me as it is to you.

The aptitude why command is awesome. At a glance, it tells you the whole dependency tree that led a package being installed, starting with the user's action:

chromakode@ardent:~$ aptitude why liboxideqtcore0
i   unity-webapps-service Depends webapp-container
i A webapp-container      Depends webbrowser-app (= 0.23+14.04.20140428-0ubuntu1)
i A webbrowser-app        Depends liboxideqt-qmlplugin (>= 1.0.0~bzr490)
i A liboxideqt-qmlplugin  Depends liboxideqtcore0 (= 1.0.4-0ubuntu0.14.04.1)

Every package manager should be able to answer that question so simply.

Leaps

There's a lesson I seem to repeatedly forget, applicable to both life and design, in the nature of leaps forward. By "leap" here, I refer to an evolutionary change in the quality of an experience or approach which simultaneously solves many problems at once. The big "ah-has" that transform entire problem spaces and change the ways we think about possibilities.

A leap typically has the ability to take things we knew would be good, but opens them up in ways we couldn't imagine before. For example, the potential of smartphones was long known before they became widely available. Authors and researchers imagined the possibilities of portable computers and ubiquitous connectivity decades before they became everyday utilities. Sci-Fi predicted things like the internet, Google, widespread social networking, and cryptocurrency. However, what it didn't predict is the things that come after. The results of the leap.

As designers and engineers, we are constantly asked to quickly sort through possibility spaces, finding the good or elegant options. When searching a possibility space, we often use heuristics, basing our perception of options on experience or existing data that models expectation. Leaps obscure the options that these models fail.

Sometimes you can't measure an experience after a leap based on what has come before it.

Why did social networking take hold when it did? Why did YouTube and Netflix start to work after so many video streaming services failed? Why are peer-to-peer sharing economies like Uber or AirBnB working now, rather than 10 years ago? Many people felt certain all of these things would eventually work, but why now? It seems easier to look back and remember the reasons these things wouldn't work, rather than notice the changes which allowed them to.

Perhaps the reason leaps are so difficult to pin down is that their effectiveness comes from a lot of little changes, rather than a few big ones that humans find easier to reason about. Many small reductions in frictions that touch our lives and the lives of others in little ways throughout our day.

I often make the mistake of forgetting leaps when I judge a new idea. While it is necessary to be able to rapidly weed out bad ideas when searching, I miss good ideas too because I didn't see the leap. And there's the crux of the problem: many of those little reductions in friction can't be noticed until you try.

A leap of faith? Or rather intuition?

Fixing hanging deja-dup S3 uploads

After configuring deja-dup to back up to S3, I hit a snag: the process seemed to hang during the upload phase.

To obtain more information, I found that you can enable verbose output via an environment variable (why it isn't a verbose command-line parameter is a mystery to me):

DEJA_DUP_DEBUG=1 deja-dup --backup

The first S3 upload would start and hang, eventually printing the error:

DUPLICITY: . Upload 's3+http://[...].vol1.difftar.gpg' failed (attempt #1, reason: error: [Errno 104] Connection reset by peer)

It turns out that this is a transient error for new S3 buckets while the DNS changes propagate through AWS (reference). Indeed, the full error contents of curling the bucket described a temporary redirect, which was probably not being handled properly by deja-dup/duplicity/python-boto. After waiting about an hour, the problem was resolved and my backup process went smoothly.

As a side note, after tinkering with the IAM profile a bit, this is the minimal set of permissions I could find for the duplicity account:

{
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket"
         ],
         "Resource":"arn:aws:s3:::BUCKETNAME"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObject",
            "s3:PutObject"
         ],
         "Resource":"arn:aws:s3:::BUCKETNAME/*"
      }
   ]
}

A hacky workaround for empty contents_pillar in salt-ssh

I've been really enjoying learning to use SaltStack to configure my servers and VMs. The relatively new salt-ssh transport is incredibly convenient for managing a small number of project cloud servers. However, there is one limitation I've discovered when handling certificates and private keys: file.managed's contents_pillar parameter outputs blank files.

It seems that pillar data is not sent to the minion environment when using salt-ssh. The contents_pillar pillar lookup then falls back to an empty default value (site note: a good example of why strict KeyErrors are helpful!). However, since the state datastructure is rendered on the master server, there is a hacky workaround relying on templating directives. For example:

/etc/openvpn/server.key:
  file.managed:
    - contents: |
        {{ pillar['openvpn']['server_key'] | indent(8) }}
    - user: root
    - group: root
    - mode: 600

(It is necessary to use the indent() jinja2 filter so that the inlined contents form valid YAML.)

Hopefully a future version of salt-ssh will support contents_pillar, making this unnecessary. In the mean time, this was the least gross hack I could find.

Downloading an image of a Linode drive

Recently, before rebuilding samurai, I wanted to download the old drive image as a final resort in case I'd forgotten to snag any files. I was a bit disappointed that there was no way to simply download the raw image using the web interface, but there are dark arts of dd that can fill this gap. Linode provides their own instructions for the procedure, but I discovered a few tricks worth saving:

1. Running a separate rescue distro per Linode's suggestion and then opening it up to root ssh seemed a bit ugly to me. However, imaging the system live and in-place could lead to corrupt images if the disk is written to. Remounting the live root filesystem read-only with mount was not possible, but there is an old Linux SysRq trick you can use to remount all mounted filesystems read-only:

   # echo u > /proc/sysrq-trigger
   

   While it's still a good idea to stop any nonessential services, this allowed me to proceed with the system online and using my existing ssh keys. I also lived dangerously and kept my soon-to-be-doomed nginx process running during the imaging. >:)

2. Since most of my drive was empty zeroes, passing the data through gzip was a massive savings in both transfer and disk usage at the destination:

   $ ssh samurai "dd if=/dev/xvda bs=64k | gzip -c" | dd of=./samurai.img.gz
   

3. Un-gzipping the image file the straightforward way leads to a problem: gunzip does not appear to support sparse files. In my case, I didn't have 50GB to spare for 1.5GB of uncompressed image data, so I needed a way to decompress to a sparse file. There's a nice trick using cp and /proc's file descriptor mapping to pull this off:

   $ gunzip -ck ./samurai.img.gz | cp --sparse=always /proc/self/fd/0 samurai.img
   

So there you have it! Gzipped online disk imaging with a sparse output file. Note that the image itself won't be bootable, as Linodes boot a kernel specified outside of their drive storage. You can read the data by mounting the image loopback via the instructions Linode provides.

I'm sure this is all common knowledge around sysadmin circles, but I wasn't able to find all of the pieces in one place during my googling. Hopefully this recipe may prove useful. :)