How the 60-Year-Old IRS Computer System Failed on Tax Day


April 17. In 2018, this was Tax Day. It’s the deadline for American taxpayers
to submit their income tax returns to the federal government. Yet on Tax Day 2018, the IRS electronic filing
system was down. Millions were attempting to e-File on that
day alone and nearly every one of them were denied. Turns out the IRS’s computers were still partially
relying on old assembly code meant for mid-20th century IBM mainframes, and a buggy update had caused their modern systems to come crashing down. What happened? This is LGR Tech Tales, where we take a look
at noteworthy stories of technological inspiration, failure, and everything in-between. This episode tells the tale of the 2018 Tax
Day outage and the cold war-era technology that contributed to it. In 1959, the Internal Revenue Service had
some big problems. 260 million of them, actually. That was the number of tax documents they
had to check from 1958, with over 60 million individuals and 975,000 corporations all submitting
their tax returns that year. Even though the IRS employed 50,000 people,
checking 60 million documents by hand was unfeasible, much less the millions of tax
audits on top of that. And that’s where International Business
Machines came in, with the latest and greatest of their so-called “brain machines,” “electronic
brains,” or simply “computers.” And in 1959, the new hotness were these gigantic, room-filling mainframes from the IBM 700 and 7000 series. Specifically, the IRS made use of the IBM
7074 starting in the early 1960s, capable of calculating word lengths of up to ten digits
in addition to a sign, with a total memory capacity ranging from 10 to 20 kilobytes in
its standard configuration. Now, the entire tax return and history of
each American citizen could fit onto a single four-inch long, half-inch wide strip of magnetic
tape, accessible on equipment costing the IRS just $4,000 in daily rental fees. Or $33,150 per day adjusted for inflation. By 1966, the IRS had gone onto invest in enough
IBM hardware that the entire country’s taxes could be checked by computer every year. And their latest machine was an absolute monster:
The Martinsburg Monster, as it was known, both for its size and its location in Martinsburg,
West Virginia. And also for its reputation as Uncle Sam’s
cold, calculating tax monster, expected to catch billions of dollars in unreported income. In reality, the monster was an IBM System/360,
one of a family of mainframes that IBM delivered to governments, businesses, universities,
and anyone who had a few hundred thousand dollars from 1965 to 1978. And for the time this was absolutely state
of the art, thanks to an impressive tape storage solution: the Individual Master File. The IMF was stored on a couple thousand reels
of tape and held the data for each individual taxpayer, corporation, legal entity, and whatever
else the IRS needed. When an employee needed to look something
up, the appropriate tape could be loaded using the Martinsburg Monster and checked against
a selection of punch cards to verify everything from an individual’s name, address, money
owed, marital status, tax credits, deductions, and so on. And for a while this was rather disconcerting
to the general public, with stories in the news describing it with such phrases as “un-American,” “Orwellian,” “an ultimate weapon,” and “frightening.” – [CBS reporter] There are those at Internal
Revenue who say that if this building in Martinsburg, West Virginia were filled with hay instead
of computers it still would put the fear of god in us all so long as it said National
Computer Center on the outside. This is where we are at, the end of the line
for all of us, lined up together on the shelves of internal revenues National Computer Center
in Martinsburg, West Virginia. The Martinsburg Monster. The very idea of computer checking returns
conjures up images of frightening efficiency, which guarantee either that you can’t cheat
or that the other fellow can’t cheat, depending on how you look at it. – [LGR] Of course, the US government did what
the US government does and released a series of short propaganda films to try and explain
exactly what the Monster was doing, and how the computer wasn’t evil at all, but merely
super good at its job. – [IRS narrator] This is the real heart of
the Martinsburg Monster. Nearly everyone in the United States has some concern with this mechanical marvel and its electronic relatives. Electronic computers have made it possible
for IRS to process millions more returns than would have been possible by hand. Returns go through a carefully planned cycle
of processing operations. When information from each return is later
transferred to the punch cards, the machines of course check the math. – [LGR] Whether or not the public embraced
this new faceless overlord though, it didn’t really matter. What the federal government really cared about
was its proficiency doling out refunds and catching more tax cheaters than humans could. And considering that by 1968 the IRS was bringing
in an additional 25 billion dollars in taxes year over year, the government’s continued
investment in tax computing was justified. By the mid 1970s, there were mainframes, terminals,
and now minicomputer systems in all of the most important regional IRS centers across
the country. Allowing them to collect an additional 5 billion
dollars annually by 1977 due to the larger number of successful audits, with computerized
detection having become so accurate that the odds of an audited taxpayer losing a case
were 4 to 1. Meaning that you’d have a better chance
succeeding at a game of Russian Roulette than against the Martinsburg Monster. By the 1980s, the microcomputer revolution
was in full swing, with machines that packed unprecedented performance in a form factor
that fit onto a desktop. Not only that, but modem usage was increasing
alongside them, allowing microcomputer users to dial into far-off computer systems remotely. Tech-savvy tax practitioners in particular
got an early jump onto the personal computer bandwagon in order to help them prepare taxes,
even if the IRS still required them to print out and physically mail in each return. It wasn’t until the mid-80s that the IRS
Research Division started experimenting with a new Electronic Filing System. The first e-filing in the US happened in 1986,
performed by just five tax preparers in three metropolitan areas: Cincinnati, Phoenix, and
Raleigh-Durham. The process went like this: a tax preparer
would use their personal computer’s modem to dial into the main IRS Cincinnati Service
Center, and an IRS employee would pick up the call, plug the phone line into a Mitron
magnetic tape terminal, and the completed digital tax return was received. It was then processed by the IRS on a minicomputer
system using a newly-created e-File program written using COBOL. This final step was one of the trickier parts
of the process, requiring the assistance of retired programmers to develop software which
could interface with the aging IRS computer systems from the 60s and 70s, which also ran
COBOL. After 25,000 digital tax returns were processed
successfully in 1986, the e-File system was deemed a success and work began on expanding
its usage. Nationwide e-filing commenced in 1990, and
although the system still only allowed returns that were due a refund, 4.2 million of them
were electronically filed that year. In 1999 electronic payments through credit
and debit cards were introduced, along with the ability to sign returns electronically
instead of by mail. The e-File system continued to expand alongside
the explosive growth of the internet in the 2000s, with Free File and Modernized e-File
debuting in 2003 and 2004, respectively. This resulted in a record 68.4 million returns
filed electronically in 2005, and in 2011, e-filed returns crossed 100 million that tax
season, meaning that approximately three out of every four US tax returns were now filed
electronically. With all this in mind then: what happened
in 2018 that caused the whole system to fail? Are they still using all those old mainframes
and minicomputers running COBOL or what? Well, in a word, yes.
Kind of. While the old IBM machines themselves are
long gone, the actual programs they ran remained in use throughout the emergence of the modern
e-filing system. And as of the making of this video, the IRS
still relies on those old programs written in COBOL and IBM assembly languages. Remember the Individual Master File, the giant
database of all individual taxpayers? Well, the IMF is still relied upon to reference
all that taxpayer data, meaning that even though the computer hardware itself has been
upgraded, they still have to emulate the computer systems from the Kennedy Administration. Not only that, but every time the US tax code
changes, the old programs have to be updated, and this has caused a massive headache for
the IRS. According to a Government Accountability Office
report in 2016, some 20 million lines of code are still used that date back to the creation
of the IMF in the 1960s. And this ancient programming is only going
to result in greater challenges as time marches on, something the government has been aware
of for decades. But despite ongoing efforts at modernization
and hundreds of millions spent since the late 90s, the Individual Master File remains begrudgingly
in use. And there were copious problems with its planned
replacement, the Customer Account Data Engine. Despite lofty ambitions and nearly half a
billion in funding, CADE was only ever used as a hybrid system tied to the old master
file, delivering on only 15 percent of its promised capabilities before being canceled in 2009. And this brings us to the outage of April
17, 2018. According to a Treasury Inspector General
report later that year, a known firmware bug caused a Tier 1 high-availability storage
array to fail at around 3 AM on Tax Day. This was an 18-month-old piece of hardware
installed to support the Individual Master File. That morning it detected a deadlock condition
after a warmstart due to cache overflow, causing 59 systems in total to fail. Since almost all other IRS services and systems
ingest data from the IMF mainframe, they too failed, and e-File systems were offline for
11 hours during Tax Day. It was later determined that buggy firmware
on their IBM hardware was to blame, something that IBM had known about since June of 2017. And IBM released a firmware update fixing
the bug that November, five months before Tax Day. But after a December 2017 meeting between
them and Unisys, the agency’s storage contractor, the IRS decided not to update on the advice
of Unisys, since the older firmware was thought to be more stable in supporting the all-important
Individual Master File. And so, millions of people were freaking out all over the United States, unable to pay their taxes. And droves of IT specialists were freaking
out in government computer centers, scrambling to get everything back online while the higher-ups
breathed down their necks. All over a bad piece of firmware on a storage
device holding a bunch of primordial software programmed before the moon landing. While there are currently plans to replace
the Individual Master File with a new system, CADE 2, that still hasn’t happened either. Will the system continue to hold together
until the IMF is finally replaced? That remains to be seen, but until then, I
hope you’ve enjoyed this episode of LGR Tech Tales! Never thought I’d do a video about taxes,
but whatever, I just like computers, stories about computers, and using computers to make
videos about computers. If ya like this kinda thing then stick around,
I’ve got new videos every week right here on LGR. And as always, thank you very much for watching!

100 thoughts on “How the 60-Year-Old IRS Computer System Failed on Tax Day

  1. Not getting the point of that stupidity, why so much bullshit on new failed software and the cost? For basically database program. Just because it is government or something? Did they gone crazy and they must support old master file as it always was or something? That masterfile is some kind of obscure ibm database that nobody uses for a good reason?
    I bet upgrade is never instated because everyone hates taxes and managment can bathe in money of those failed upgrade contracts 🙂

  2. A half of me is concerned that my tax info is stored on an ancient reel on a shelf, but my other half is glad the government isn't using scarily hyper efficient nasa computers for this.

  3. I predict that within 10 years computers will be twice as powerful, ten thousand times larger and so expensive that only the 5 richest kings of Europe will own them.

  4. America. The most technologically advanced country in the world, we boast an OP military, we have the most technologically advanced valley, yet we still use tech from fucking 60 years ago and even still use fucking windows 98.

    Please kill me.

  5. I also oversee old calculation programs (dating from the 1950s!) where I work. While we don't actually run the code on the old mainframes or Cray minis anymore, you would be unnerved at just how awful coding practices were back then. People didn't think their code would last 60 years! So they didn't document that well…

  6. So many large firms out there still using IBM mainframe. Lots of migration to newer systems but it's slow work! I love playing around with them, such awkwardly specific rules to work with but you gotta love it!

  7. Every modern IBM CPU for giant systems/servers (Z-series) has a full IBM 360 implemented in HARDWARE on the chip, so that it can run legacy mainframe programs (usually in COBOL or 360 assembly) from the 1960s/70s natively in hardware without having to do emulation (and therefore not risking bugs). It's not just the IRS, a lot of the major banks / banking infrastructure is like that. It's funny looking at pictures of the dies b/c the entire 360 architecture is like a tiny little spec in the corner of the chip all by itself, compared to the hundreds of millions of transistors elsewhere that are all interconnected. I remember discussing this in a class in grad school.

    And they're not alone; a lot of businesses (Amtrak, airlines, auto parts chains) use DOS software for their user interfaces and try to train the people on how to use it. I remember laughing at seeing an Amtrak rep on a brand new Dell computer (the era of their black boxes with the circle logo), running a DOS emulator window with their custom app running. That's why they seem to be typing so much… it's an all keyboard interface (no mouse). Infrastructure is the hardest, most painful, and most costly thing to update; whether that's a bridge or software.

  8. Doesn't the us have a paye system? I haven't filed a return since the 90s because it's all worked out for me as I go.

  9. I work with a team of young webbies who ribs this old timer for maintaining some code that is 20+ years old.

    The difference – I point out – is that “my old shit” still serves the company day after day regardless of hardware advances. “Your new shit becomes unmaintainable every five years because your frameworks evolve past your app’s codebase – essentially, you guys are stuck in Park.”

    Long live big systems – THAT’S the backbone of stable long-term businesses.

  10. If the government is anything like the facilities it funds (my high school) I bet they still have a few dozen tape reels that they actively use for tax processing

  11. You made a video about the IRS that was actually fun to watch. How the hell did you manage to do that!!??

    😵 😵 😵 😵 😵 😵 😵

  12. Weird Unisys did not want to perform a software update on an IBM system. When worked for them we would do the fixes for A Series and OS2200 when required. Both flavors have mostly been made virtual now, but that only happened in the last ten years or so.

  13. I work with witj an ibm mainframe every day. Almost every old fortune 1000 company that is old seems to still rely on ibm mainframes. You can make a lot of scratch if you can deal with old terminals and or cobol

  14. The Marvinburg Monster is a great stage name for a new WWE character. Could wear 60s style computer engineer outfit (thick rim glasses, white short sleeve business shirt etc).

  15. I always wondered why you can't simply convert the databases of an IBM 360 system to work with more modern programming languages?

  16. As someone who works at a DMV in my state, this is relatable. One of the systems we use is an emulator of a IBM terminal from the '70s! (TN3270 It crashes every few months.)

  17. Lets come back to this at the next crash in 2050, and see if they're still emulating that same IBM COBOL system. Technical debt, so wonderful! 🙂

  18. Fascinating, and very well done! Very much appreciate all the obvious hard work that went into researching and creating this.

  19. ITs funny because when I do my usual research into orders of magnitude and the histories of when things were implemented technology wise, even other parts of the government are so behind, they were asking google years ago if hundreds of terabytes would be enough to replace their aged systems. Google said no, because a few hundred petabytes, even 2 or 3 hundred would be enough for even the FBi or Cia. When asked if thats what google would use themselve,s they replaied that they accumulated hundreds of petabytes by the late 2000s, slowly needing dozens of exabytes by the end of the next decade just to consolidate and ready themselves for the 2020s, when a few hundred petabytes might eventually not be enough to see them into the 2030s, and so on. Yottabytes on the way in the next few years, less than 10 in theory. lol

  20. Those programmers were not payed enough for that software! XD Imagine if they instead licensed it out to the IRS every year.

  21. Screw an upgrade, it does the job, and nobody I know can hack it. Keep it until it doesn't work…

  22. The upgrade requirements are insanely complex. Forgetting for a moment about upgrading the code, which is a complete rewrite, the data wouldn't even be transferrable! Those systems used a text format called EBCDIC which is not readable on modern systems. To do any work they would need to access prior data after the upgrade. That means all those tapes you saw would need to be translated into ASCII or unicode and stored somewhere. The sorting code that depends on EBCDIC will have to be redesigned. It would be a complete redesign from the ground up including data. This isn't an upgrade, it's a replacement! Millions of lines of code and 100 BILLION tax returns and a billion audits converted! Basically, the IRS will have to be replaced!

  23. Marriott also uses IBM Mainframes and cobol, people at the check in still uses the old MARSHA software

  24. I'm not an American, nor i live in America but your voice, you made it a great experience watching it
    CheeRz

  25. In Norway some doctors still receive patient data on floppy disks. Government is apparently decades behind the rest of us

  26. When I seen this video I said oh , the government was screwing ppl out of there money with dinosaurs .

  27. Thats why Trumps tax records can't be released. They are stored on a 8" floppy that nobody else can read.

  28. A storage array failing sounds like something that would happen with modern software too. Especially since the hardware in question was only 18 months old. And it was a firmware bug – that sort of thing doesn't happen on old systems because if there was a such a bug, you're more likely to have found it ages ago. So unfortunately that problem validates their decision to continue using the known working software.

  29. Did this episode remind anyone else of the Siemens 4004 from Willy Wonka & The Chocolate Factory? Actually, a rebranded RCA Spectra 70 machine designed to work with the IBM system/360! All credit for this information to www.starringthecomputer.com, an archive of various machines used in movies and television over the years with some surprising detail

  30. Tech tales are some of my favorite videos on the channel! You always talk about really interesting stuff, and everything is thoroughly researched!

  31. My Dad who would be over a hundred now,worked for the Dept of Agriculture as a data systems analyst for payroll!,He was one of the very first and was over his dept! He received calls constantly in the evening because he understood the code better than anyone else !

  32. It's the same all over the world, and thinking closer on it it isn't that surprising either. This is also common in the industry. Many legacy systems around there ad well…

  33. So basically, the old software and the old data in its old file format work perfectly fine and have done so for decades. Yet the whole system fell over due to a bug in NEW hardware that was not patched because SOMEONE thought it might not be worth doing.

    Solution: High availability and redundancy.
    Implemented by: Creating a second mirror system that is ready for rollover at a moments notice.

    Everyone who has critical systems should be doing this.

  34. My dad’s company have a similar situation. Literally, they rely on radar machines from the 1940’s. Most of their systems come from the 1950’s and 60s, but lots of 1970’s systems are still in use. It’s honestly incredible. Apparently they are experts at maintaining old hardware, and tend to prefer keeping 1960’s systems running rather than their more modern replacements, as if you break it down enough the 1960’s equipment is simply transistors and switches, and dad’s company are really good at replacing these components to keep it running.

    The other day, I received from them a bunch of components from a 1982 286 DOS machine, which had been doing important work for 40 years nearly. It also came from a 2005 monitor. It’s only being replaced now, and I think it might well be being replaced with another 286 rather than a modern PC.

  35. someone should incript all the computer files used by the IRS then delite all of the files and over write each file 50 times with asky junk good by IRS.

  36. For the government itself, it's not a big deal. System crashing ? Just delay the deadline with 1 week. Done.

  37. The social security building here in So Cal, was using older beige boxes for ages……and their systems would slowly update data over the course of months and months. The US Government charges more and more taxes and tariff's but they wont update their aging data processing infrastructure!! lmfao!!

  38. Or, now hear me out, OR we could just revert back to them using computers that played nicely with the IMF. Why fix what wasn't broken.

  39. Gotta love those decisions that favor older firmware instead of upgrading it. Dammit, if IBM said the bug was present and it would crash, it's a gamble from there onwards.

  40. This really is the best content youtube has, I really wish Clint had the time and resources to do more content likes this.

Leave a Reply

Your email address will not be published. Required fields are marked *