It is amazing what a new hat can do for your outlook on outside work. I was having a &*(^ day in a bucket truck and my wife came by with a hat that had been sitting on the dash of her truck with the warm defroster running on it. I think just that hat recharged me for another 6 hours! :) That and the fact that someone cares about you...
As for the very long night.. I had an issue where I was not getting BGP updates from neighboring ISPs when working for a colo. It was an all nighter and I finally had the wherewithal to call an alpha geek friend of mine. (his license plate is his ARIN NIC). After asking me a few questions to see if he could fix things quickly, he told me to go take a nap and not call him until I had slept for at least 2 hours. I protested, but he assured me we could fix things faster and more effectively if I had some small amount of sleep. Well, 3 hours later, and well rested, I noticed an ACL blocking incoming route announcements from my peers. I swear had I not taken that nap it would have taken me 5 hours to notice! ryan On Mon, Dec 7, 2009 at 6:58 AM, Robert West <robert.w...@just-micro.com>wrote: > Are you sure you didn't eat spicy food before bed and this was just a > nightmare? > > At least you got a new hat. That's the bright side out of all this, I say. > > > > -----Original Message----- > From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On > Behalf Of MDK > Sent: Monday, December 07, 2009 6:50 AM > To: WISPA General List > Subject: [WISPA] One long @#$%& day! > > It's 2:30 AM... > > I just drove up to the house, and walked VERY fast inside. After all, > weather.com says that it's 12 degrees and an 8 MPH wind. In their head, > maybe. Outside of the little valley I live in, the wind's more like 25 > mph. There's no snow on the ground. > > I don't make a habit of staying up late, but last night, I was doing one > of > > those "let's just have some fun" looking around sessions on EBAY. Next > thing I knew, it was nearly 3AM. Without shutting anything down, I just > crawled into bed. At 8:45 my cell phone rang... I didn't answer it, but > I > > did get up. Looked at my computer and Peer Monitor says... nothing is > connected. Now, I have had some issues with one of the dist points a few > miles outside of town. It had randomly locked up 3 times last week. > Each time, I thought I had found the problem and not worried about it. > > The first time for instance, it was extremely dense fog, and I found the > ventilation fan running in the box. Thinking I had sucked in too much > damp, I just shut it off and rebooted. The locked up system is a mini-itx > board and RB 14 adapter card W/4 radios... > > Obviously, I was wrong. Something was wrong. It had run since Friday, > but now it's Sun AM and PM says I've been off for 3 hours. It's died 2 > other times since the fog incident, so... Houston, we have a problem... > > I quick yanked on some clothes and drove up the mountain to the site, used > the step ladder to get to the box lid and looked in. Restarted and > everything went off just fine. But, it's now done this several times. > And that's not good or right. I look in the van. Spare mini-ITX board, > licensed. Spare RB14, 2 spare radios, including an XR5, just like what's > up there. Anyway, morning zips by, and I have an appointment in the > afternoon to switch a family friend's computer out for her. So, I go to > do > > that and she's not home. That's odd. I could have sworn she said she'd > be > > there at 2... I wanted to be home, nice and warm and setting up the new > board so I could change in in daylight tomorrow. > > So, I go to the workshop and do some stuff I've been putting off and ... > fall asleep, waiting for an OS install to finish. When I wake up, it's > after 5. Must have slept at least 15 min... Sheesh. So, I get up, drive > over to house, start the project. 30 min later, my phone beeps. Text > message... Site's down. ARRRGGH. > > 25 minute drive to the site and I reboot it. Go home, pull out the parts > and start to assemble the whole thing. It goes down again . Drive back > up, restart. This time, nothing will coax it into running. Finally, I > pull everything out, and take it home. Now the phone's going nuts. I > just > > put 60 customers down. I take it home, cause it's COLD out there on the > hillside... and everything runs flawlessly. Just to be judicious, I grab > a > > config backup off of it, 'cause it's changed lately and my last backup is a > few weeks old. > > I haul it up the mountain, put it back in place... No run. > So, I go home, grab the spare mini-itx, use the backup config and haul it > up > > the mountain. > > Won't boot. Doesn't even beep. Power comes on, but no beep. > > Haul it down the mountain, back home (this is .7 miles of rocky pasture I > drive through at idle in 1st gear, then 5 miles of paved road. Takes 20 > min > > round trip) with all the parts. Runs flawlessly. Haul it back up the > mountain, plug it in, boots, but locks 5 to 60 seconds after booting. > > Inspect EVERYTHING (it's dark, so had to do it all by flashlight) again. > I > > see nothing. So, I grab the spare RB14, switch the radios over and plug > it > > in. Boots. Lights flash. Fire up the laptop and no. It's not > working. > > log in through ethernet port... NO RADIOS DETECTED. > > Put old RB14 back in, change one radio. Boots up. Logs in. all radios > detect, data flowing to customers. > > Drive home. Try to thaw out. The wind up there is 25 mph and it cuts > like a knife through you. Log in from home and watch everything, > suddenly > > noticed one of the ap radios has no clients. Odd. Look at config, looks > ok, reset radio, clients begin to associate. As they slowly hop on, I'm > sitting there wondering why this... and just then, PM suddenly starts > popping up red. Sure enough. System locked. > > Drive up the mountain, put in second spare radio in place of XR5 and boot. > (Spare XR5, was used for 2 weeks in temp site). System boots and runs > and > > ... No link to the XR5. Try new pigtail. Dead. How strange. It > worked fine just 9 days ago, and it's been stored in the ALIX board it was > installed and working in all this time. > > As I'm sitting and watching while sitting in the van, trying to stop > shivering with the heat on full blast, It locks up again. I reboot. > Everything comes up, I had put the old XR5 in, and everything starts > working - apparently it wasn't the pigtail, just the replacement XR5 that > had no output. Waited 15 min, still running. Go home. Look at Peer > Monitor. All Green. Suddenly, a bunch of stuff goes red, and it's all > far away. Call partner. He's a whole state away, forgot he wasn't > coming > > back from his relatives until Mon night... But the red stuff's near him. > As is the workshop. > > It's now almost 11 pm. The van has almost no gas left, the nearest open > gas station is 23 miles away. So, I drive to the workshop, to find out > why on earth the shop and some other stuff was red, and everything's fine > when I get there. Finally decided that I must have removed a routing > statement earlier that I needed instead of the right one. Started going > up > > the chain, looking at all the routing. No routing errors located, but > the > > very last device before the backhaul to the site that feeds my town.... > shows no link. I call home, son verifies... Site's down again. > > Said a few angry outbursts... Looked around found another new RB14, put > ALL new radios in it. Determine the new RB14 doesn't look anything like > the > > old one. Determined which slots are numbered which, so I can know which > pigtails go where... Go to gas station, get gas. Stop at Walmart on the > way back, get REALLY WARM HAT for the head, and drive back to the mountain. > Somewhere in there I stopped and got junk food to eat too. Hadn't had > food > > since 1:30 PM. Was feeling faint. That took over an hour. > > Drive back up the mountain, remove old RB14 and old radios, put in new. > System wont' start now. > > Pull board and everyting, look at it in the light. See no problem. Put > it back in, plug everything in, then I found I had bumped the "off" switch > on the #$%&^ power strip. Flip switch, plug in PS, lights come on. > > Since no XR9 is in use now, a pigtail is wrong, so I get a new piggy and > replace the old one with a UFL/NFBulkhead... Piggy is cold and it takes > about 30 tries to get the UFL connector on, even tried "warming" it with my > hands. Err, my hand shaped chunks of ice. It takes 3 sessions to get > the pigtail in place on the bulkhead end. It took several to get the UFL > end > > on. I was able to stand outside just long enough to push the threaded end > through the hole. Go sit in van. Get out, get nut properly threaded. > Get back in van. Tighten nut with fingers. Can't find tools I had > earlier. Get in van. Finally, get out of van with gloves on, and hand > tighten the n connector. Get back in van. Can't feel my hand touch my > face. My hand can't feel the difference betwene inside the glove, in > front > > of the incredibly hot air coming out of the heater, or touching the cold > glass window beside me. I could not even get 2 minutes outside before pain > so intense from the cold wind drove me back inside. Drive home. > > EVERYTHING IS UP. Oh, wait. Several sites are going > yellow/red/yellow/green/yellow/red repeat adnauseum... > > So, I start looking. Finally, discover that the 3.65 backhaul has 120 to > 400 ms ping times across it. Even though it's passing just 40 to 100KB of > data. Restart both ends. No help. Then, I noticed that quality on > end is zilch, but the other end is running 18 to 36 speed, instead of 54. > Lock back faster end to 36 and pings drop to 2-8 ms. Great. Another bad > XR3... This one's up on a pole, about 18 feet off the ground, on top of a > hill at a wind powered site. I think it'll wait until tomorrow. > > Oh, wait... I ##$%%^ forgot to close the danged gate, so I drove back up > the > > mountain. Closed gate. Wind is 5 mph stronger AT LEAST and 2-4 degrees > COLDER than it was last time I was up there. The moon is pretty. > Stars > > are awesome. I can't stand outside to admire them. > > Ok, the system's been up for long enough. I'm going to bed. Now it's > 3:30 and I can't keep my eyes open. I hope I'm forgiven if I turn off > the > > cell phone and hope to sleep in.... And I still DON"T KNOW WHICH PARTS > ARE GOOD OR BAD!!!!!!! > > I'm going to have nightmares. > > > > > ---------------------------------------------------------------------------- > ---- > WISPA Wants You! Join today! > http://signup.wispa.org/ > > ---------------------------------------------------------------------------- > ---- > > WISPA Wireless List: wireless@wispa.org > > Subscribe/Unsubscribe: > http://lists.wispa.org/mailman/listinfo/wireless > > Archives: http://lists.wispa.org/pipermail/wireless/ > > > > > -------------------------------------------------------------------------------- > WISPA Wants You! Join today! > http://signup.wispa.org/ > > -------------------------------------------------------------------------------- > > WISPA Wireless List: wireless@wispa.org > > Subscribe/Unsubscribe: > http://lists.wispa.org/mailman/listinfo/wireless > > Archives: http://lists.wispa.org/pipermail/wireless/ > -------------------------------------------------------------------------------- WISPA Wants You! Join today! http://signup.wispa.org/ -------------------------------------------------------------------------------- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
