PLAYERS TASKS PRAXIS TEAMS EVENTS
Username:Password:
New player? Sign Up Here
Professor Møbius
Ranger
Level 6: 1251 points
Last Logged In: July 3rd, 2012
TEAM: The Disorganised Guerilla War On Boredom and Normality TEAM: SCIENCE! TEAM: Level Zerø TEAM: INFØ TEAM: Silly Hats Only TEAM: The Adherents of the Repeated Meme BART Psychogeographical Association Rank 2: Trafficker The University of Aesthematics Rank 2: Dealer Humanitarian Crisis Rank 5: Diplomat Biome Rank 4: Ranger Chrononautic Exxon Rank 3: Historiographer Society For Nihilistic Intent And Disruptive Efforts Rank 2: Trickster


140 + 30 points

Craaank by Professor Møbius

January 7th, 2011 12:21 PM

INSTRUCTIONS: Don't stop moving for a full 24 hours.

(Write-up started back in November, and was ironically interrupted by a sever crisis.)

Really? "Don't stop moving for a full 24 hours."? That's the best you can throw at me, SF0? (For some parts of this completion, I loosely interpreted "moving" as being mental and dexterous movement as well)

Try this on for size: I'm an IT Administrator for a real estate group, I have the most temperamental little server farm on earth, and I have literally no problem putting all the time into keeping that network up that I need to.

devotion_to_duty.pngYeah, like that.

In the past 2 weeks, I've worked almost 210 hours, including no less than four 40-hour shifts where I didn't stop moving for a moment. Let me take you back to last Sunday, though, as I was laying in a tent as water was leaking through it's poorly constructed ceiling and onto the screen of my smart phone, which was displaying for me in vibrant color the fact that my network was - much like a drunken peregrine falcon - diving at the ground at 200 MPH, and was not going to pull up in time not to end up so much feathery, bloody insanity.

(31 hours, starting Sunday 5:30AM) -

Phone buzzes, panicky message from server telling my it's dying and wishing me it's best with whatever server we pilot next. I'm not so pessimistic. I shout at Mantis and Tirius, getting them up, we proceed to scramble around in the rain, packing up our campsite, running up and down the hill to truck, all the while getting drenched and loving the warm water coming down on us. Once in the truck, I plug my phone in, dial into my PDC at work and get to work getting to the bottom of the problem, fingers flying, mind focused so sharply that I didn't realize that we were (a.) at burger king, (b.) that I'd ordered food, and (c.) that I was eating - until I tried swallowing a bite I'd forgotten to chew and started choking. Coughed it up in the trash can, went back to eating slower, and working on my network. At this point email was safely routed, phones were back on line, and I was working on getting one of the other Domain Controllers to take up the load.

We get home, I hopped in the shower, scrubbed down, talked to Pixie a little, hopped on my bike and took a small nap during my ride up to the office (It's 2 miles away, and I like the exercise), got in and started yelling and pacing as I saw the culprit: my branch's managing broker had ripped the switches out and stuck them in a box to send to Cisco. Why? Because they were being slow and that wasn't acceptable. After averting security, who'd come to investigate the initial ripping and my subsequent yelling, I decided to go with it, put the switches in the shipping boxes and ran them downstairs. I'd already missed pickup time for our UPS box. Called UPS, found out that a truck was two blocks away. So what did I do? What any admin trying to get back to his dying network would do: ran.

After putting the switches in the hands of the UPS guy, I darted back to my office, got Staples on the phone and proceeded to arrange for a covert delivery of more hardware - something to get by until I can clear the MB out and get my Cisco stuff back. Staples doesn't deliver. After a two and a half hour conversation, alternating between pleading and yelling, I finally get a regional manager who used to do corporate IT, and in a moment that I can only remember being able to compare to Ford Prefect psychically overwhelming a bartender with his grief at the loss of Betelgeuse, I imparted to him my grief. Fifteen minutes later, the store manager (different bloke) was on my doorstep, handing my a crate full of networking hardware, asking me how I pulled off getting it delivered.

At this point, words seem unnecessary. Co-workers walk up, as I'm frantically dancing my fingers across keyboards, look around at the wires strung across the server room going to monitors and keyboards (because my KVM accidentally got shipped to Cisco) and ask if everything is going to be ok. The cleverer ones take my silence as the only response I can give, the less bright ask me when facebook is going to be available again, one even going to far as to point out in some misguided guilt trip that his farm was dying. Despite my instincts to chuck something - probably a monitor or a hot soldering iron - I calmly got up, closed my door and turned on some music.

Pizza. The smell, like some goddess' perfume, pulling me out of my trance. Got up, dashed to the conference room and started eating. My stomach's response was a stern reprimand not to go so long without eating, as I was thinking it could only have been a few hours, I look at my watch and see that it's nearly 10pm. Choke down another slice and duck outside to call Pix, venting some of my frustration as she reminds me that killing co-workers is bad and that remembering to eat/sleep is good. As I do this, I walk around the building, waking me up a little bit.

Get back in, sit down to start working, and as though in answer to my unspoken question of the morning "What more shit could roll down this hill today?" my modem is sitting in it's mounts, lights all lit up in an over-bright kind of way that lets you know that something is wrong with the LEDs. The modem manual says nothing about this phenomena. Call vendor tech support while pacing my office; they are less helpful than the manual. Upside: I know better than to call them again in the future for ANYTHING. Call Comcast. "We can have a technician out in 3 hours." Look at watch: It's just after 1. Right.

Crack another wifi connection from the first floor of the building; if it wasn't meant to be public they wouldn't use WEP, and I wouldn't be that callous about it usually, but damnit my network needed me. Get back to work sifting through backups and trying to isolate the original problem, now that we're something close to stable.

Just. As. The Comcast technician is walking in the door with my new modem, this high-pitched grinding noise starts coming from the server cabinet, and is promptly followed by nine more identical noises. I look at my primary, my backbone server that I was just getting ready to raise from the dead using a backup cobbled from the flesh and entrails of six other backups, and saw harddrive LEDs stuck on, go from green, turning that sick-yellow, then orange as they cooked one-by-one.

Ripping the power cable out of the back, I stood for a moment and fought back tears as the Comcast guy tried to get a grip on what hellish battlefield he'd walked into. I directed him over to the rack, where he spent the next half hour installing the modem, as I sat nearby, sobbing. He left, and I called an associate of mine from school, at this point ready to try anything to get back the data that would save the network. Drew says to do a transplant. I ponder this for a moment, steeling myself for the pain that's about to come my way.

As I'm rooting around a supply closet, pilfering some painter's plastic, duct tape and a vacuum cleaner, I call a slightly seedy friend of mine, a Russian who lives in our Chinatown and always seems to have a black eye and whatever computer crap you need. He has ten of the exact model drive I need.

I send a co-worker down there to grab those, while I rig together the messiest clean-room known to man. It's a five-by-three-by-three mostly-sealed box with a hole for the vacuum hose and two more for my hands, shod in yellow rubber cleaning gloves. A quick sweep with my phone clears the area of magnetic disturbances, the door is locked, and I begin carefully opening drives, and transplanting platters.

Three hours later, it's done. I look at the clock, it's just around six in the morning. I can hear the first people trickling in to get the office moving. Someone must have noticed the "Sorry for the inconvenience, we're working as fast as possible" sign on my door, and had the good sense not to knock.

Slapping the drives into their bays, I fire up my backup and once that's firmly underway, I begin alternating between cleaning my office (which looks like a tornado hit it) and just pacing while trying to figure out what could possibly go wrong next. Grabbing the receptionist/intern, I tell him to stare intently at the screen and call me if anything but the little blue bar moves, and proceed to run outside, across the street and grab breakfast. Get back, it's all ok. 8:45 rolls around, my office is PRISTINE, and my restoration is complete.

Boot it up, clean up the minor misalignments from my frankensteining, and breathe a moment as everything seems to work. Until I look back at my error logs. It's pissed because it can't resolve itself across the internet. Internet's functional. Spend 45 minutes trying to figure out the problem, before realizing that not only did I get issued the wrong static IP when I got the new modem, I got issued someone else's.

Call Comcast, yell for 30 minutes. Not because they're being resistant, but because now I understand - despite my confusion in school - why hostile users exist, and it's these idiots. A technician finally makes it out, an hour later, resets my IP to the correct one, apologizing profusely. I say nothing, only glare like some jungle-cat staring at a mouse that should be eaten on principle, but isn't worth the effort.

He leaves, it's good. Functionality is back, for the most part. I spend two more hours going mad, editing logs to trick the machine into thinking nothing ever happened, then convincing email that it's ok to come back and function again. Just as I'm typing in the last command to bring it on line, my finger reaching for the enter key, I pass out. During my time asleep, it seems someone had the wherewithal to press "enter" and finish my work, as I woke up a few hours later and everything was back up and running.

The rest of that week was bad too, some days where more interesting, or insane, but that one stands out, probably because I was most awake for it. I may redo this at a later date, but when I saw this task, I knew my next network issue would make a decent completion.

- smaller

Sysadmins

Sysadmins



6 vote(s)



Terms

(none yet)

5 comment(s)

(no subject)
posted by rongo rongo on January 7th, 2011 3:19 PM

I'm glad I don't have your job...

(no subject)
posted by Pixie on January 7th, 2011 3:31 PM

I remember this. It wasn't the most fun...

(no subject)
posted by Pixie on January 7th, 2011 3:45 PM

I wasn't sure if i should vote or not at first, seeing as this is not the traditional way of doing this task. However, i think the challenge of it has earned you the hella points. Congratulations for doing something epic! though admittedly unpleasant

(no subject) +1
posted by Selahsaurus on January 7th, 2011 9:35 PM

Bitches dunno. Try doing it with kids. IN HEELS

(no subject)
posted by Ombwah on January 16th, 2011 2:54 PM

gods save the ITa