09-12-05 Server Crash Explanation

A place to rant about politics, life, or just anything you damn well feel like telling others.
Post Reply
User avatar
Apoptosis
Site Admin
Site Admin
Posts: 33941
Joined: Sun Oct 05, 2003 8:45 pm
Location: St. Louis, Missouri
Contact:

09-12-05 Server Crash Explanation

Post by Apoptosis »

At 3:15pm on 9-12-2005 many of our readers noticed that Legit Reviews went down. The crash was directly related to the Los Angeles power grid problems. According to CNN News worker error killed the power to half of L.A.
LOS ANGELES, California (CNN) -- About 700,000 electric customers in Los Angeles lost power Monday afternoon after a worker mistakenly cut a wrong line, triggering a cascade of problems in the city's power grid, a spokesman for the Los Angeles Department of Water and Power said.

About 50 percent of the department's 1.4 million customers were affected by the outage, which began at about 1 p.m. (4 p.m. ET).
Our web servers just happen to be located in the city of L.A. Our servers are backed up by a diesel reserve that is capable of seamlessly supporting the IDC for 28 days. The Power to the building is drawn from two substations and is on a preferred service grid of the Los Angeles DWP that only provides service to one other building. The facility is also served by a second power grid for redundancy.

Both power grids failed and so did the diesel power generators. The servers crashed when more than 1,000 people were on the main page and another 100 were on the forums. Since data was being pulled from the database when the server crashed the database was corrupted.

The site is back online (Duh you are looking at it) and the dbase has been repaired. Sorry for the site being down, but the problem was thousands of miles away and due to the power outage in LA.

Let me know if any "issues" are seen on the site or forum.
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

lol, in the words of Milk Chan (if you watch adult swim, and I'm talking about the person who inadvertantly cut the power): "You dumbass", lol. Thanks for the clarification there, and I did notice where there were database errors later on in the day, so right there I knew someone screwed up.
Image
User avatar
dicecca112
Site Admin
Site Admin
Posts: 5014
Joined: Mon Mar 01, 2004 10:40 am
Contact:

Post by dicecca112 »

I like the DB errors here, doesn't take 5 years to get to the page or have it time out. You long on and you know its down
Image
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

Oh yeah, it only makes good programming sense to implement a feedback/diagnostic module into a site and site software.
Image
User avatar
Apoptosis
Site Admin
Site Admin
Posts: 33941
Joined: Sun Oct 05, 2003 8:45 pm
Location: St. Louis, Missouri
Contact:

Post by Apoptosis »

you a coder killswitch83?
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

fraid not, I just know some of the basic rules of programming from when I took a Turbo PASCAL class (pretty much the "plain english" of programming languages). I'm more hardware-oriented, and I'm developing my knowledge of networking and Cisco products. After I finish CCNA 1 and 2, it's go time for that CCNA, and a nice cushy secure job, lol.
Image
kronchev
Legit User
Legit User
Posts: 20
Joined: Tue Sep 13, 2005 3:16 pm

Post by kronchev »

Repaired? My user name was completly gone and this huge post I made was gone :(
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

some of the data got corrupted in the database when the power went and those servers crashed, so that's probably why your post and username got wiped. I'm pretty sure it happened to others as well.
Image
User avatar
Illuminati
Site Admin
Site Admin
Posts: 2378
Joined: Mon Oct 06, 2003 8:48 am
Location: Wright City, Missouri, USA
Contact:

Post by Illuminati »

kronchev, sorry that you lost your username and a contributing post... As killswitch stated, since our forum tables were corrupted, we did lose a little data.
Justin West
Server Admin & Forum Moderator
Follow me on Twitter | Find us on Facebook
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

I have one question about that Justin: did the host offer data redundancy? Because that could have saved a lot of headaches early in the game. Just curious 8)
Image
deadly-app
Legit Extremist
Legit Extremist
Posts: 307
Joined: Fri May 07, 2004 3:23 pm

Post by deadly-app »

Well seeing as it kept much of the data intact, I'm guessing it saves a copy of itself fairly often so if it does go down you don't lose much information.
Image
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

Yeah, more than likely. It probably didn't fry the storage drives or the servers, so they probably just used a restore point to put the data back online.
Image
User avatar
Apoptosis
Site Admin
Site Admin
Posts: 33941
Joined: Sun Oct 05, 2003 8:45 pm
Location: St. Louis, Missouri
Contact:

Post by Apoptosis »

actually no data was lost when the server crashed. When the site came back online the database tables were corrupt. I ran some repair commands and got the forums back online. I missed a couple tables and the forums crashed again. I then made a script to run a full dbase repair on the forum tables and the data between the repairs was lost. Was less than an hour worth of posts and I didn't notice a new member joined. Wasn't data loss due to the server crash
User avatar
killswitch83
Legit Extremist
Legit Extremist
Posts: 1747
Joined: Tue Jun 21, 2005 3:45 pm
Location: South Carolina

Post by killswitch83 »

oh ok. Thank you for clearing that up for me Natedogg 8) ;at least I knew there was some data loss somewhere, lol :P
Image
Post Reply