Well, that was a crazy work day.

One of our customers did something yesterday afternoon that they really should not have done. Had they asked me about it I would have warned them off. I wasn’t asked though and they did it and it blew up.

The messes fell like dominoes. The thing they ran caused this one thing to go bad, which in turn caused this other thing to go bad, which then caused a third thing to go bad.

First thing this morning I was asked if I know of a way to clean up the mess. I said no, it just needs to run its course. Half an hour later the problem turned into a teamwork adventure. I threw the question out to my whole staff. Any ideas? One guy says that this one thing includes this other thing that we could sort off of. I say that the issue is past the point in the process. Someone else asks if that one thing includes a link to this other thing that is the actual point where things are stuck? A third guy comes back with woah, there is a link. 

So loop through the second thing, get the linked data, link back to the first thing and check the sort item, if it is what we’re looking for, kill it off. Run that over the entire million and a half records and boom, problem solved.

Another guy figures out that there is a third thing that is feeding the first thing. We had been under the impression that the problem had moved beyond that point but it was still processing and making things worse. That third thing has a value we can sort off of as well. A second loop was written to kill off anything that is still feeding the problem. That loop had to be run first.

The only other problem came when the first loop finished and we had to start the second loop… everyone was at lunch. Shit. I had people pushing for progress and I had to stall a little so that the writer of the loop could come back from lunch and execute his code. I did some of his prep work and fortunately he came back before the customer started asking questions and everything went off without a hitch.

Teamwork, man. Three customer service programmers and one development engineer all working together to fix a problem that none of us even realized was possible. Fan-freakin’-tastic. All the while there was me, your humble narrator, getting on conference calls with the customer trying to keep them from freaking out. My job description doesn’t really include dealing with customers directly but it does happen now and then. Today was one of those rare days and I was the point person on most of these calls. I’m guessing that they could tell that the big red head introvert talking to them was not comfortable in the role, but I’m also guessing they could tell that I was getting a little excited at the prospect of my team doing something that had never been done before.

Of course, now that we know we can do it… here’s hoping we never have to do it again.