Hidden things, lack of fries and pointless data

Ah Friday’s, it’s always a good day in the office, everyone (at least in the dev team) know it’s Friday, and most weeks, Dan’s behaviour is a tell tale sign. Friday also means pub lunch day, something to look forward to…usually – however today was a sad day, our usual pub, the Wellington, let us all down. The waiting time for our fries was going to be at least 45 minutes, and I do mean just fries, chips, potato cuts, whatever you want to call them. It was almost enough to ruin the day, almost! However the pint went down well and I watched a bit of Bruno which helped…but still, no fries… Dan decided that to entertain himself (I can only assume it was due to the lack of fries on the couch of happiness*) he’d take Ben’s cigarettes off him and not give them back, and then hide them in the office allowing an honest “I don’t have them” statement. Deep down, Ben was happy about this, he said he wasn’t, but he did it with a smile 🙂

Now the important rant is over, databases! We have a collective big one so to speak, it’s huge (for what it stores.) We’ve known all along a lot of data is pointlessly duplicated and indeed some of it is pointless (invalid); however the extent of that was found out today – an estimate anyway. I wrote a script to de-dupe one of our tables, a table that contains hundreds of millions of rows, at the moment, we stand to loose 1,000,000 rows that are completely useless. We also have almost 16,000,000 rows that are duplicates and can be removed and that’s before the data is normalised (the normalisation process is affecting about 16% of the data [yes really, 16% of hundreds of millions]). Once that is done, I am anticipating the duplicate row count to go up to more like 25-30 million. The best thing about all of this though, was the resulting conversation, the same conversation half of the dev team have pretty much weekly that begins with one member saying “We only have that much data because we’re storing it wrong” – you know who you are! It then moves into sharding (nothing to do with a toilet) and archiving. Dan also broke one of our jobs even more than he did yesterday but that’s a daily expected situation so isn’t worth more than this one sentence.

And because it’s Friday, here’s a cat, thanks to icanhascheezburger.com

*the couch of happiness is opposite the couch of joy in the pub. Not that it helps you in any way for me to drag your eyes down here to know that, but now you do and can scroll back up, more confused than you already were.

Comments

comments