It’s probably imprudent on my part to be so candid, but I’m known for being earnest, so I’ll just plunge right ahead.
As most of you know, we had some severe downtime today. At about 10:50 a.m. PST, our server load went from a normal 2% up to 80%, which basically brings everything to its knees. Our database load was 486% (no, I don’t know how that’s possible either, but it’s bad). Ironically, this all took place on one of the best “down” days we’ve had in months!
It reminds me of the many days and nights I spent prowling around the bowels of server farms. They have a very distinct smell – – probably mostly because of the constant cool air being pumped in – – and unless you were installing something new and shiny, being down in a network center usually meant something was wrong and you were under the gun to fix it. It’s a miserable feeling until you’re sure you’ve figured it out. I think I subtracted a decade from my life dealing with those traumas.
The PAIX I am referencing in the title is the Palo Alto Information Xchange, which was one of the first major Internet exchanges in the United States. It’s actually housed beneath the site of Palo Alto’s first school from the 19th century, and it’s filled from top to bottom with whirring servers, network routers, and RJ45 cables.
There’s not a soul on the planet that likes downtime, neither user nor administrator. Slope actually went for eons without any downtime at all, but in recent weeks we’ve managed to snuff out spammers completely and optimize a bunch of back-end processes. However, is obviously still a beast or two lingering in the woodpile. I appreciate your patience with us, and I’d like to thank God almighty that Slope isn’t a brokerage, since such a “mission critical” failure would have given me a dozen heart attacks.
I only managed to do my tastytrade show today since a light bulb went over my head and I realized I could use our development server for the charts (which is also where I composed this post during our downtime). We might have more rough sledding today as we continue to sort things out, but hang in there. And again, thank you.
