A career in software development: Expectations vs. Reality

Whenever I read a post or a blog entry about some new technology (React, Ember, WhateverJS), I think about my career, and what I’ve done in it. Being a (frankly, line-of-business) developer is never about using something new or doing something different. A good chunk of what I’ve done over the last 12 years is really just CRUD (create/read/update/delete) apps – things that display data, or let you change that data.

One project (or really, a four-alarm tire fire) always sticks out in my mind, and has been recently – the time I had to build a caching system. In 2009, I was working for a company (I’m changing some details for anonymization purposes) that built and maintained a website – and all the backend processes to support it – for a client. You’d probably call it an app these days. Their employees (okay, a subset of them) used it on a fairly regular basis for ordering certain products or subscription-like products.

Well, we were running on 2 website servers and a single MS-SQL cluster for probably 10k, 12k users. And we had been building an entirely new version of the site. But this one day I came in, and we were having horrible database connection issues, and performance was in the gutter. Of course, that happened about every other day, for different reasons. But this time, a cursory glance at sp_who2 showed lots of deadlocks on a single table – the table that stored the users’ transactions.

You see, since this was an internal application, everyone was given a certain amount of money to spend on these products – but it could only be spent up to a certain percentage of the cost, i.e. 30%, 40%, 50%. Instead of storing a balance of that amount and calculating against it, or updating it after transactions, this app only stored each debit or credit. If you wanted a balance, it had to be calculated. Every time. And the website was doing that on every page load, because their balance was in the page header.

When you’re a developer on a team, people come and go, and you never end up being the person who built an entire website, or even a feature. When you’re in a company with lots of turnover – and this certainly was that company – lots of people have had their hands in that code. By the time I left, I think our team had turned over 3 or 4 times entirely.

So I can’t say who in the world made the decision – or if it was even a conscious decision – to store transactions this way – or rather, to never store a balance. (Thank God Entity Framework didn’t exist when this site was built or it would have been a seven-alarm tire fire.) But I was the most-senior database developer, and it was up to me to somehow resolve it. No one could use the app at all, and we were getting hit from every direction. I literally had three or four people coming to my desk asking what was going on. We needed a solution immediately.

So there was no time to investigate redis, memcached, all that stuff that people talk about these days. I don’t even know what would have been the option in those days. I had zero hours to research and even less time to implement.

So here’s what I did. We had a stored procedure getting a user id, and returning an amount – their current balance. So I created a table to save the results:

ID (PK) User ID Balance Date Modified
int varchar(32) double DateTime

Yep, balance is a double, not money, not even an int (or a long) stored in cents (there’s a famous project that does this, but I can’t remember the name). File this under “who in the world made this decision? or did anyone? sure wasn’t me”.

I threw an index on UserID, changed the existing select procedure to just select the record from this table instead of calculating every time, and then created a new proc to update it. That was easy – we already had the exact script I needed. I copied that into my new procedure, had it update the proper record with the balance, the current date/time, and hit F5 to change it (live. no time to test it when literally everything is broken, with people literally hovering over you, literally breathing down your neck.)

It worked great! Or at least good enough to get the app back up and people out of my space. But then over the rest of the day – and the next – I had to add in a call to that update procedure to the between 50 to 100 other procs where transactions happened, so the balance wouldn’t be out of date. No big deal for a system with about 4000 stored procs, eh? (Pinal Dave and AutoHotKey were my best friends. I had macros like Ctrl-Alt-1, Ctrl-Alt-2 that spit out premade 7 line SQL queries all the time.)

So I invented my own caching system, and it worked well. I didn’t research other options, and I didn’t read up about caching theory, or look at examples. I had the idea and threw it together – because I had to. That was the one option at the time. And no one really noticed, other than they could login again. Of course, when you make a change like that, you get the symphony of “you touched it last, so you broke it!” complaints. Even though I hadn’t changed the calculation at all – just delayed it a little. And in the ensuing months, I (or rather our users) found existing bugs in the pre-existing calculation that people knowing about the change simply exposed, rather than the cache actually causing it.

I’m not saying you should do any of the above, or even that it was great. It just simply was what had to happen. No choice, do it or – I don’t know what. Quit? We lose the client and all lose our jobs? But these kind of experiences are what my career has been founded on, and what I have under my belt. Instead of arguing about JSX or Microsoft or why JSLint transpiling in Jenkins Considered Harmful in comments sections, I was frantically making stuff work, with basically no help. I’ll leave you with a quote from Friends, when a stripper steals Ross’s wedding ring – a quote I also always think of when I see arguing over an unproven technology that’s a week old, and am reminded of the above day’s events:

Ross: (dialing the phone) All right-all right, fine! I-I’m gonna call the cops!

Joey: Dude, I screwed up, you don’t have to turn me in!

Ross: Not on you! On the stripper!

Joey: Oh, yeah, well I already did that! They said they’re gonna look into it right after they solved all the murders.