At the Black Hat conference in 2010, an ATM designed and built by my employer was setup on stage, and a security researcher demonstrated an exploit which emptied out all its cash. For many months prior, I had been my employer's designated point-man in responding to this attack...
The anti-hero of this tale is a heinous moustache-twirling villain whose hobby is tying damsels to train tracks. His unlikely name, Barnaby Jack, evokes a swashbuckling mischievousness that surely destined him to be either a hacker or a pirate.
As a depraved evil-doer, Barnaby is of the civil and gentlemanly ilk, reminiscent of Moriarty, Hannibal Lecter, and Count Dracula. Thus he courteously delivered to us a detailed technical report about his attack on our ATM. One can imagine that The Joker or Lord Voldemort would have published without first informing us, so mad props homie.
I read Barnaby's document. There should be a word in English for the pall of gloom that gradually descends as one reads a damaging report that will soon be made public, slowly shaking one's head and increasingly muttering obscenities.
It was immediately clear that before anything else our first task was to implement a defense against the attack. So to know mine enemy, my colleague Keith and I traveled to San Jose to debrief Barnaby, also with a mind to assassination by wooden stake through the heart should the opportunity present.
In person, Barnaby was charming and likeable, belying the heartless evil within. (Ok, ok, I'll stop with the wicked villain schtick.) He was very forthcoming with technical details, and helpfully clarified the nuts and bolts of what he had done. And what he had done was an extremely competent reverse engineering exercise, demonstrating very strong technical skills and determination.
With a full understanding of the attack, our urgent task was to then implement our defense. My primary role was to design and implement our solution, working with my fellow programmers Richard, Andrew, Keith and Clark. This "project" consumed most of my time for months, because aside from the software development, I was also responsible for preparing bulletins to our customers, drafting our post Black Hat press release, liaising with Barnaby and his employer, managing the deployment of the patch onto several different model ATMs in various territories, and discussing our legal options with our lawyers (my favorite of which was to have a U.S. Attorney frog-march Barnaby out of his office workplace in handcuffs, but alas this magnificent image was never realized).
Meanwhile, my friends and colleagues in our trade group, the ATM Industry Association, setup a workgroup to respond to Barnaby's upcoming presentation. Barnaby hadn't announced which manufacturers were the subjects of his "research", but like NATO, an attack on one is an attack on all. Usually I am quite active in the ATMIA security forums, but given that we were still treating the matter as confidential, I discretely recused myself.
Barnaby and his colleagues had planned to present his work at the Black Hat conference in 2009, and his presentation was listed in the Black Hat 2009 program. But this was relatively soon after we had learned of his attack, so we insisted that they delay their presentation until we had sufficient time to roll-out our patch to more ATMs. Barnaby's employer ultimately acquiesced, albeit grudgingly, and his Black Hat 2009 presentation was cancelled. Apparently this cancellation was something of a minor scandal among some Black Hat participants, who condemned his employer as cowardly caving to The Man. Of course I am biased, but I truly view his employer's decision as an example of conducting security research responsibly. Anyway, the up-side for Barnaby was that the delay gave him time to add a second ATM to his presentation.
Black Hat is probably the world's premier infosec conference, featuring the most prestigious security researchers, attended by tens of thousands, and extensively covered by the media. (Review of Black Hat by Obi-Wan Kenobi: "You will never find a more wretched hive of scum and villainy.") So seeing Barnaby's presentation highlighted on Black Hat 2010's home page as the one "Featured Briefing", we knew that this was going to be a circus.
Thus it was that my boss Bob and I attended the Black Hat conference in Las Vegas in late July 2010.
The auditorium at Caesar's Palace is enormous. I later heard estimates that the audience for this presentation numbered 5000, and was overflowing into the corridors. They reserved us seats in the very front row, like the jester's throne at a celebrity roast.
I won't describe the details of Barnaby's presentation, you can find it on YouTube, other than to say that he had two manufacturers' ATMs on stage, and successfully attacked them both. His attack on our ATM was local and required physical access to the internal electronics, while he remotely connected to our competitor's ATM. He didn't disclose technical details of how to perpetrate these attacks, it was more a demonstration that it was possible.
When our ATM began dispensing its cash, the crowd erupted into wild applause. It's an interesting experience to have 5000 people on their feet cheering with triumphant glee at your demise. Bob and I seemed to be the only people who remained seated, arms folded, unsportsmanlike killjoys.
Unforewarned, and still somewhat shell-shocked, we were then invited to join Barnaby at a press conference. This was a dangerous breach of the "golden rule of tech company media liaison": never let your engineers talk to journalists. But Bob fielded the reporters' questions with great aplomb. Despite the precariousness of our situation, the bottom-line was that we had released a patch many months beforehand, and thankfully there were no real gotcha-media moments for us. I did delight in schadenfreude as Barnaby squirmed trying to evade a series of questions about the status of the solution for our competitor's ATM.
Bob and I then phoned our office to provide a quick summary and to green-light the publication of our press release, and we then proceeded to "quietly have a few drinks".
In the days that followed, we received a few media inquiries, and our customers and partners wanted to clearly understand the status, but we were well prepared with bulletins etc., and the noise died down after a week or so. To me this mild response reflects just how desensitized to such attacks the public has become, and people's familiarity with the endless hack/patch life-cycle. Hacking an ATM may be entertaining because it is the fulfillment of a fantasy, but with successful attacks on targets such as RSA's SecurID tokens, it is difficult to imagine a target that is considered so secure that anybody would be truly shocked were it successfully breached. Though having said that, Barnaby's presentation was widely reported in the mainstream media, not just tech journals, it was perfect grist for the tabloid mills. My friends in Australia heard about it.
My great mistake, and a huge gift to my antagonist, was during one early discussion with Barnaby, to casually use the phrase "jackpot the ATM", which caught his attention, and I then foolishly elaborated that it was our in-house term for an attack such as his which empties out the cash. Fast-forward to the Black Hat conference, and when I visit Barnaby's employer's booth I am appalled to see it overflowing with marketing material – banners, stickers, brochures – all based around a "jackpot" theme. The coincidence of "jackpot" with Barnaby's surname "Jack" was too good to be true. Later, Barnaby did buy me a beer to acknowledge his debt for this branding windfall.
In truth, Barnaby and I started as adversaries and ended as friends (which is surprising because Barnaby is from New Zealand and I'm from Australia, and trans-Tasman friendships are regarded as treasonous).
Barnaby chose my company's ATM arbitrarily, it was just the most conveniently available to purchase on the web and be delivered to his home. Note to our salespeople: for security purposes, please make it more difficult to purchase our product.
It may be impolitic for me to recount the following anecdote, but I shall document it for completeness. At an ATM Industry Association conference some months later, there was a panel discussion about the repercussions of Black Hat. I mentioned it to Barnaby and he attended, sitting in the audience. The panel included my boss Bob, and a gentleman named Bill from the other victim ATM manufacturer. Bill stated that Barnaby's attack was an "inside job", and this compelled Barnaby to interject. The panel moderator then invited Barnaby to stand up, and some in the audience politely applauded, though I jeered and booed. Continuing, Barnaby explained that he had no inside knowledge and that the ATM was a black-box to him when he began, but Bill unwisely stood his indefensible ground and chose to debate the matter with Barnaby, and kept digging himself into a deeper and deeper hole.
Looking back on it all, I'm kind of proud of how we conducted ourselves: we dropped everything to immediately implement a technical fix; we engaged Barnaby and worked with him, despite the temptation to be adversarial; and we were open and honest with our customers, uncomfortable though it was. There are right and wrong ways to respond to security breaches, and we could have done worse.
But overall, it was a bitter pill. No matter how I spin it, having one's ATM so publicly attacked can never look good. But the actual fallout was far more benign than our worst-case fears, and people were surprisingly unmoved, no doubt because we had already released a patch. The enormous benefit is that our ATM is now significantly more secure, because we didn't just plug the hole that Barnaby discovered, but we took our defense a thousand times further, and implemented a comprehensive security scheme which includes the cryptographic verification of all files as they are installed and as they are executed. Ain't no nasty software runnin' on our ATMs ever agin. Barnaby got his 15 megabytes of fame, and we improved the security of our product, which I guess is how this ruthless Darwinian process is supposed to work.
If nothing else, it was an adventure.