The reason I’ve been so quiet the past few weeks… a client needs to move into a data centre. These guys are ramping for production in a few months, and they’ve got SLAs to meet at that. The Ikea racks just weren’t cutting it, and there’s a plan to move to rack-mount gear anyway… tho a 2U case is about $350 a shot. So I was off to Computer Recyclers again, much like last year when I was setting up Server North. Picked up a pair of these blue cabinets and had ’em delivered to the client’s site.
Nice thing about getting them at Computer Recyclers is that they’re cheap. A few hundred bucks… they’d be a few thousand if purchased new. The downside: They need cleaning. A lot of it.
This is the canvas I have to work with. Seems my tape-measure has gone missing from my toolbox, so I had to measure using the Mykes units. 6′ x 10′ appx (0.98 Mykes * 1.6 Mykes; 1 Myke ~= 6’2″) The company who occupied this cage previously had a half-rack SAN (… they called it a filer) that apparently tossed a disc a week. So, you figure out how expensive a 36GB UW drive is and cry. Not that SCSI HDDs that small are stupidly expensive, but they’re certainly not as cheap as a 300GB SATA drive.
The cabinets arrived Wednesday of last week… and I started cleaning ’em. Amusing observation while unloading: The air inside the cabinets was so hot, that once we brought them into the airconditioned building, the cooling fans in the tops started spinning due to air flow caused by the sharp temperature difference. (It’s about 18C in the data center, and the outside temperature was about 34C… and I’m sure the air in the cabinets was hotter than that.)
Took me almost 2 whole days to disassemble, clean, scrub and wash the cabinets. Then another half-day reassembling and rearranging the shelves to accomdate the non-rackmount equipment going in. I finished at 9PM Friday night. And I was tired, didn’t even eat a dinner. You can also see in this photo the UPS I’ve swiped from the refuse pile. It needs some therapy… and Dr. Myke plans to give’r. (Remind me some time to tell you about the UPS I picked up last week, if I don’t write about it later.) I’m hoping new batteries are all this unit needs. A 2U PowerWare unit for the cost of SLAs and some elbow grease is not a bad deal at all. There’s an derelect SUA2200 in another corner of the data centre, but it’s in a great many small and medium sized pieces and I’m not so confident I can repair it. You can actually see the back of it on the right of the photo.
After getting my driver’s license photo updated, picking up 4 more machines from MBC, delicious Pho dinner with BCRL, and about 12 hours of work, I can proudly present a rack full of equipment that’s doing nothing but drawing power, producing heat and lighting LEDs.
Err… doh. But at least it looks good… Right? 🙂
About 5 minutes after I snapped this shot, I was working behind the racks, connecting up the Ethernet patches; just as I plugged one of the servers’ patches into the switch – all the lights went out. Once my brain figured out what’d changed, I then had another WTF moment as I tried to work out how a GigE link could knock out the lights… Turns out that people leaving the building at 10PM on a Saturday night assume no one else is still there and turn off the lights. Note to self: Bring flashlight and always have something bright on the screen. If I ever have to run in during a power failure, it’s going to be very dark.
What’re you’re seeing here is a lot of redundancy. With the exception of the monitoring machine, which is duplicated off-site, there’s two of pretty much everything. For obvious reasons I won’t detail much, but the dual-Dells (ugh) firewall/routers are actually doing something really neat: running pfSense with CARP/pfSync so that they’re completely redundant. There’s an extra cross-over cable between them and they synchronize NAT states, firewall states, routes, rules and a bunch of other parameters, so that if one node goes down, the other takes over completely within a few seconds or less. Quite interesting how this works too because the software actually creates a virtual network interface with it’s own MAC address, this is what they share.
8-port KVM switch at the top of each cabinet. Linksys SRW2024 GigE switch below the patch panel on the back of one cabinet. Several medium sized APC SmartUPSes (Symmetra is the next step for these guys.)
There’s almost not an inch to space in these racks. What is ironic, or just downright annoying, is that I could consolidate all this gear into one rack, and that’s assuming 3U cases (use all standard parts)… if I could use 2U it’s be tinier. I’m just waiting for these guys to get a nice big influx of cash and then I’m just going to buy a S4ton of Sun X4200 machines 🙂