Way back in late 2004, we came up with the systems architecture for Server North. We implemented 4/5ths of it by March 2005, but we were plagued with motherboard problems in the fall, and only recently recovered enough funds to put the final server, p1, in place. That was in early August. We’d been wanting to put it in for a very long time.
In mid/late August I put in the order to Computers + in Oshawa. By mid-September the case nor motherboard had arrived. Turns out the case was a very new model and not-yet stocked in North America, and the motherboard is EOL, so supply is very limited. ie: There’re two for sale in North America – and I just bought one of them, via MBC. Finally everything arrive about 2 weeks ago, and a window opened up for me to go to Toronto late last week.
Thursday
Drank too much caffeine while driving to Toronto. Picked up the rest of the parts for p1 at Computers +. After a delicious Mamma’s Pizza, I assembled p1 but it didn’t actually work at first power-on. I fiddled with the RAM and decided it was likely the trigger of the problem. Then I manged to remotely crash client’s server trying to kick off a back-up job, so I gave up for the evening, emailed him to reboot his server and then went to the hotel to crash.
Friday
0230 – The sirens started. And continued. All Night. (PDFs 1 2 3)
Somewhere around 0330, I finally fell into a fitful and futile sleep… until 0618 when the client with the crashed server called. Turns out that his server had eaten itself (/etc/fstab go >poof!<) so I spent till 9:30AM fixing that with him.
So now I'm up, and really not going back to sleep, I started calling PC stores to see if they'd let me test their RAM on my server before I bought it. Most shops wouldn't, but the one that did reminded me that I probably need registered memory (buffered.) Basically this is an extra layer on the memory bus that makes the electrical load less on the memory controller – this is so you can have large amounts/many sticks of RAM in a server. Oh – and it’s not cheap.
Oy vey. Nobody in Toronto stocks it! Actually, that’s a bit of a stretch, but while Registered ECC RAM is expensive, I could source a single 256MB stick for $378 downtown, and an outlying location of the same chain had a slower stick of the same for $280.
While eating spicey noodles, (from the nearby Korean bakery) I related this to Rebecca at EI Cat, when just then a colo-mate walked by while I starting to rant about not having any Registered ECC RAM and that this whole trip is about to become a total waste of time/money… and he says “Oh, I’ve got like 3 sticks of that. You want?” — !!
Turns out that he does, but they’re DDR2, doh. But he did check with a couple distributors and found some in Etobicoke. “Only 15 minutes away” he claims. So he orders a bunch of other stuff and I head off in the Element (with Mikes for company.)
15 minutes my butt!
2 Hours later I return, drop in the RAM… AND IT DOESN’T WORK!
Here’s a fun trick: Find bugs in your $500 motherboard’s BIOS. Sigh. It was crashing WHILE doing the memory test, which is further than I’d got it going previously… whatever that’s worth. I managed to get into the BIOS by pounding F2 and Esc. really fast right at power up. Poking around I discover that setting ‘Memory Interleave’ from ‘Auto’ to ‘Disabled’ solves the problem and I’m in business.
I call this a bug, because with one stick of RAM per CPU, one would expect ‘Auto’ to not TRY to interleave the memory… unless it’s trying to interleave it across the CPUs’ connecting HyperTransport buses – and failing…?
Install FreeBSD 6.2-BETA2 and cvsup to 6-STABLE. Things are looking good.
While that was happening, I walked next door to buy some Geek Fuel (aka Pepsi)… The distance from EI Cat to this store is half a block, no more than 250 feet. Last summer, while walking this stretch (only takes 30 seconds) a pane of glass on the building above exploded and showered the ground with millions of pebbles glass. This time, there was a 4×4 truck pushing a TTC streetcar down the street! I’d only noticed this because the ‘driver’ of the streetcar had gotten out and set the switch on the pavement with a big lever. Then he went back in and made a windmilling motion with his hand to go forward. That was amusing.
Bought my Pepsi and came back out to see ANOTHER streetcar tying up traffic because it’s power-pole had fallen off the wire and completely tangled in all the other overhead wires… Good light-show of sparks 🙂
What’re the odds? That’s like when in Amsterdam I asked Alia if they streetcars ever got misdirected, in her 3 years there she’d never seen it happen – and then on my last day there… guess what happened? Irony defined.
Sipping my Pepsi, safely back in the datacenter, I mostly finished with p1.
It’s about 8PM at this point so I have to pick up my mom’s friend and then my dotMom at the train station, as they’re going to spend the weekend together, doing whatever it is that they do. 🙂
We have a wonderful dinner at Spring Rolls next to the St. Laurence Market, just off Front St. Afterwards we tried to get to Hollywood Gelato, but sadly we got there just a few minutes too late. I dropped ’em off and went to my hotel. After all the noise and mayhem of the previous night, I decided to wear my E5s all night, and it was good. I slept reasonably till about 9AM.
Saturday
Not quite as eventful as the prior days. We (Dave and I) upgraded the Server North SAN with some success; it runs well, but not across multiple machines, so we’re RAIDed across all the drives but just in one server.
Had a wonderful steak dinner with dotMom and her friend. Went back to the hotel, watched a bit of Top Gear and read some Minette Walters – Acid Row. Slept reasonably well.
Sunday
Up early for 10AM checkout. Worked at the datacenter till 2PM… I installed Debian on our to-be Xen server… that was a big struggle given the time constraints and my lack of knowledge. But it is at least RAID1’d.
Had lunch with the girls again, then dragged dotMom back downtown with me to finish up at EI Cat.
Oh – I forgot to mention earlier – the DVP and Gardiner were closed all weekend, making downtown traffic pure hell. So I head up Lakeshore/Eastern/Kingston and just got to Scarborough when we got a call from my mom’s friend – apparently we’d forgotten some things at her house, so we went back to Avenue Road. Of course, getting back on the 401 was no treat because the collector lanes were closed too!!
We stopped in Oshawa to try and return the useless RAM to C+, but they were closed and I lost Jon’s cellphone number. Dinner at Pizza Hut took about 90 minutes and service was horrible… and when it came time to pay the bill – the staff started bickering about how and who should process the payment.
Oh – and we’ve determined there there’s on average 2 Fish & Chips joints on every block in Whitby/Oshawa. WTF?
Driving wasn’t great, the rain being extremely heavy at times, but we did make it home in one piece, thankfully missing this tragedy by only a few hours.
All in all – a lousy weekend.