TWC Down for 1.5 days
Late Sunday night/early Monday morning the site went down. There was another monster power failure in downtown Denver that included our building. Now this shouldn't have been a problem. We are supposed to be on backup generators. But after the fiasco last December (you really should read that if you haven't or to refresh your memory) apparently the idiot electricians in charge of this did not hook up anything above the second floor to the generators. I nearly lost my mind when I heard that, we are (were) on the 11th floor.
The building was hot at about 8 PM Monday night, but we still couldn't get out. Something happened to the Level 3 (owned by Century Link now as of last month) fiber routing equipment. That is another group of idiots now on my spit list so to speak. I have no idea what their problem was, and at this point I do not care. I did hear that they were back up at about 6 PM last night. But at about 1 PM on Tuesday I decided to go grab the web server and take it to a new spot.
If you recall in that other thread, I mentioned a deal I made with one of my clients about using Comcast fiber in his building, and a 90 day time period for the install. Well between construction delays and permits and all that crap, it took a bit longer than 90 days. We have had fiber live in that building since July 9th. This was a total building remodel and I put in hundreds of hours running network cable and doing all the network stuff for him, so that TWC could piggyback on his fiber connection and IP addresses. I split some costs with him on this as it relates to network equipment and labor, That building is now 99% complete, and I was planning on moving our stuff over the Thanksgiving weekend.
It got moved last night. Mostly anyways.
I didn't move any battery backup stuff, and that buildings generator still isn't hooked up. But if we have a power issue there its on the ground floor and its a 30 minute drive, so I can manually hook up a small generator easily rather than carrying one up 11 flights of stairs. And every piece of network equipment we own (me and my client) and have extra stuff just sitting right there. Everything except the actual Comcast fiber interface can be replaced in about 10 minutes, and Comcast guarantees a 4 hour response time.
I have hundreds of hours and thousands of dollars tied up in this project over the last 10 months. I haven't been posting about it because I have been working on it. The last of the offices was laid out in October and cabled, now there is some painting and stuff like that going on. I am not sure why the generator install is delayed, but eventually we will have a generator big enough to power the server room and phones and certain other parts of the building that fires automatically. So all we have to have is enough battery power to last 3-4 minutes while the generator spins up and levels out.
The entire point of this project stems from the problems we had last December. Neither James (my client) nor I was impressed with having to rely on other people. As bad as TWC was hit in that incident, he was hit far worse. He has 200ish employees that could not communicate with his customers or with each other because of that problem last year, and both of us decided that relying on ourselves would be better in the long run. He would rather pay someone to do nothing except sit there putting diesel fuel in a generator all day than go through more BS like that, and I agree with him.
His stuff has been live on the Comcast fiber connection since July 14th or 15th, and has had no problems at all. My stuff (other than TWC) has been on since mid Sept. I moved some of the TWC servers (Vault, mail) the first part of October. I didn't move the main TWC server because of the no generator issue. I guess I should have.
I still have to rebuild Thor (database server) and Olympus (test server) that got fried in that incident last year. I think Thor needs a new motherboard ($1200) and not sure what Olympus needs. It wont boot at all. So still a bunch of work to do.
At some point over the next couple of weeks I will have to take the site down to install battery backups and get the generator hooked up. But at least I should be able to plan for that.
Reply With Quote
Originally Posted by Lifthrasir
.
