Ken, Thanks for the in depth response. My biggest problem is that the DC gave us cabinets on the smaller side which makes them almost impossible to work with. If I was building out fresh I would prob get a cage and build out with big cabs but my boss isn't offering such luxuries.....
On Mon, Nov 13, 2017 at 8:04 PM, Ken Chase <m...@sizone.org> wrote: > Some tricks I've learned managing multicustomer/shared cabinets over the > last > 20+ years...sorry it's long, but I think there's some good info on keeping > things clean and maintaining sanity. Please send your protips. > > Most of this is lower-end 1-4U sized mixes of gear specific and specific to > cabinets that have 2-6U+ flux per quarter with some rushed installs. Huge > one-time 12U blade installs of $1M appliances usually lend to gorgeous > cable > management schemes (and proper budgets) being included. No such lux here! > > TL;DR: thin premade ethernet of exact lengths and multiple random colours > (never black!), use min gauge required power cable thickness of exact > length, face A/B PDU's backwards on one side of rack cable management on > other side, never get less than 30" wide x 36" deep cabinet (if not, > wider > better than deeper), premeasure vert mount rail positions to be > compatible > with rail length/ front of server clearance, prewire front of rack > power/ether if needed (leave string too), practice tooless rail removal > while you can still get in above/below, rack similar-depth gear together, > switches face backwards (with front-to-back airflow switch config option > of > course) on rail-shelves not ears (that bend over time anyway) so they > can be > extracted out front and easily replaced in emergency. > > Details: > > Installing in 30" wide x 36" long cabinets makes all the diff over 24" x > 30". > A/B 0U PDUs on one side, cable wrangling ladders on other. More room = > more > flexbility. (If have no side panels and no neighbours, 24x30" is ok). 36" > deep > allows facing the PDUs backwards not sideways - cableheads extend > backwards, > not into the rail-tail path/airflow/etc. Worth getting the > 90degree-bent-head > cables too if you need the spare inches. (I ofset my PDUs vertically by > 1/2 a > plug-spacing distance so cable from left one fits between cableheads of > right > one.) > > Avoid racks that don't use cagenuts. Prethreaded holes get abused and > stripped. Try to get the right size of cagenut, there's a few standards out > there. Some will fit - poorly. (Either they fall out under weight or you > end > up trying to force them in with a thin screwdriver - I've seen people stab > themselves in the hand. Ask for a cagenut tool (J-hooked shaped piece of > metal that > looks like a bent desktop-case PCI slot cover.) > > Having many power cables of varying lengths is key (but why doesnt anyone > make > 15" and 21" power cables?). Not having ziptied loops of 12ga wire hanging > around made things much nicer (and better airflow). More $ but worth > sanity. > Esp. with varied coloured heads. Great for tracing (see ethernet below). > Wire > the gauge required - I find 10ga (6' long..) wire delivered with 100VA-max > server > configs often. Too thick to manage properly and usually unnecessary. But > check > your warranties and theoretical max power envelopes. > > Yes, full rack solns w/extendible arms exist but generally require vendor > compatibility. Expensive too. Great for one time well-funded installs. Not > practical for varied species installed over longer periods. > > Prewire any front-of-rack-powered gear when you first get the rack. I have > 5 > pairs (A/B) going to the front permanently ziptied and labelled - 3x2 in > use > for my back-facing switches, 1 for a small piece of gear (low watt > microtik), > others spare. Also prewire some proper length (multiple colours of) ether. > Fishing ether through the side can be impossible in a full cabinet in a > dense > row (we're in APC pods). I leave string in there too (probably will use it > for a twinax pull to the microtik soon, and pull more string with). > > Curse vendors for not picking a standard side (left vs right) for power > ingress! > (ibm and dell vs supermicro, sun and hp, IIRC?) > > Beware Dell's long fins/tails on their rails - won't fit in a 30" cabinet > if > your vert mount rails are too far back - or it blocks the power cord head > on > the pdu if it faces sideways/etc. And beware max/min rail extension - Dell > seems 'longest', with many min. rail lengths of 25.5". I think I saw min. > 26.5" once. > > Also had a cx jam a long Dell rail's tail into his fully assigned cabinet > - in > between a powercable head and the pdu body it was plugged into. BZZT! > Took 20 > min to get a monkey to reset at central panel. Thank proper cabinet > grounding > cables, right?) > > If you have an entirely empty cabinet to start with, grab a few different > rails and ensure your cab's vertical mount rails are all within spacing > spec. > and give door-closing clearance to server noses. (See reference tables.) > Moving them later can be impossible (though with sunfire rails that slid to > varying lengths, it worked out luckily!) > > Must admit Dell's tooless rail installs are awesome now. Better than > supermicro's and sun's (Sunfires). Learn how to derack them before you > install, and practice a few times while you can still get your > fingers/tools > in from above/below. Make notes on how it works. Trying to guess how to > derack > a single U rail sandwiched in with no other access can be nearly > impossible, > especially an old EOL one with no identifying marks to google. > > Try to rack similar depth gear together. Nothing like having 1U 27" long > Intels sandwiching 1U of 3/4 depth supermicro between them - cant get your > hands in to do anything. 2U minimum together, but 3+ is better. Stash a > headlamp > in the rack too to see into these wells. > > Having more switchports than necessary (3 redundant 48 port switches with > avg > 2xc/box, avg 1.5U per box, 42U cabinet) allows 4' ethernet cables to be > put in at > exactly 3.75' extension into a switchport - minimal extra slack, just > enough > to manage the cable and keep it unstressed. Folding spare cable into > management ladders makes manual tracing hard, and is bad form. Crimping > your > own rj45 ends for exact lengths is risky. Done it many times (and once saw > a tech do > it before a bgp session timed out on a damaged live cable! :), but results > are > poorer than factory crimps and dont resist stress well. ('Sides, do you > have 8 > different colour spools? See below.) > > Switch racking - I'd rather them on shelves not ears - then you can pull > out > your backwards-facing switch ass-first from front of the cabinet and slide > the > replacement in without moving cables too far -- trying to get the ears past > the cabling can be heinous (and most ears cause major cantilevering due to > deep heavy switches - after a few years I find things are bent from weight, > impinging on the U below). There are thin little 3" wide rail-edge > 'shelves' > you can get, but your switch may not give the spare vertical mm required > (about 3-4mm) - works best with 2u+ sized gear where there's spare mm play. > > Proper cable management trays/guides between the switches is great too > (though > eats U). Your density/financials will determine if that's viable. Its worth > the U/sanity usually. > > Out of U for cable mgmt but have spare depth? Bungee cords won't work as > anchor points to ziptie too - they sag in the heat over time. But rubber > bungees ('tarp straps') work great. > > Though not for full 100m ether lengths, narrow gauge ethernet is key. You > can > fit like 8-12 of them into a run of what 4-5 took previously. I've had zero > issues with them with rack-lengths of cable. Worth the $. > > I wish there was thin bicolour ethernet - it's sexy to have all your ether > coloured the same (or same per category - red mgmt, yellow public, white > internal) -- but then you need to know where the failed port 34 cable is... > tracing identically coloured cables can suck, your eyes start playing > tricks > looking at 30 cables in a twisting bundle that you ziptied too tight > trying to > be pretty - and under tension, yanking on them may not identify the right > cable properly -- even with labels (you stop trusting them anyway after the > first couple errors or when job security is thin). > > I tend to use as many random colours as possible -- and note it in the > switch > configs. If the switch config doesnt match, slow down and check things > twice > from scratch. (Keep many colours and lengths handy for new installs! Blue > is so > ugly - peach ftw, amirite?). > > Note that no pattern of colour use will work - wire all mgmt ports as one > colour, > or all ports of one server one colour - either way, you assume the colour > means > the cable is wired a certain way, and that leads to errors. With no > pattern, > no dangerous assumptions are made - must check with the switch config. > Maxing out > #s of different colours leads to easier identification/fewer chances of > neighbouring > cables of same colour. > > If the management layer of your company doenst have to slave over racks, > this > is likely not an option. They like the pretty and impractical stuff best > of course. > > Labelling only works so well - in 100-120F exhaust paths most glue just > melts > off. One client I had no input on the install for has 100s of little > silvery > labels littering the floor/lower U of their cabinet. So pretty! And goey > cables. And thank god they're all white! Makes tracing so easy! :P > > (Of course no one ever wires black ethernet, right? That's a capital > offence!) > > /kc > > > On Mon, Nov 13, 2017 at 05:05:35PM -0500, Chuck Anderson said: > >On Mon, Nov 13, 2017 at 01:30:25PM -0800, Seth Mattinen wrote: > >> On 11/13/17 12:49, Mike Hammett wrote: > >> >Keep the humans out of the rack and you should be fine. > >> > > >> >Where should I send the invoice?:-P > >> > >> > >> It's easy to keep a rack nice if you take the time. I've spent hours > >> removing and replacing cables in neatly dressed bundles because > >> equipment changes required a different length/type cable, but > >> sometimes that's what you gotta do to keep things neat and tidy. > > > >Exactly. Most people do not want to spend the time to do it properly. > > -- > Ken Chase - m...@sizone.org Guelph Canada >