Worst Tech Catastrophes

<p>New contract, re-calculating an OLAP cube. The quickest way was to export the bottom row data, clear the database, load the bottom row data back in and calculate – doing that would take about 2 hours, just re-calculating, without clearing the existing data would take 12 – 14 hours.</p><p><br></p><p>A week into the contract, I dropped the database and… Realised I hadn’t exported the data first!</p><p><br></p><p>I asked my colleague what they usually did – he said restore the previous export (from 4 hours earlier) recalculate it and blame it on the users. I was horrified. I too the long lonely walk to the head of finance, explained the problem and said I could restore the previous dataset and then try and replay the transaction log. He said, fine, try it.</p><p><br></p><p>It took an hour longer than usual, but in the end, only 4 edits to the data were missing. I got praise for being honest and the head of finance trusted my judgement after that, for giving him an honest answer, as opposed to the answer people thought he wanted to hear.</p><p><br></p><p>I guess turning a disaster into a victory…</p>