Datacenter Cage // Engineering Guide

🧠Mental Model: What You're Managing

A datacenter cage is a locked, dedicated enclosure within a larger multi-tenant colocation facility. Colocation (colo) providers sell space, power, and connectivity — you own the hardware. The cage is your controlled micro-environment inside their building.

Think of it as a system with five distinct but interdependent layers. Failures in lower layers cascade upward; over-provisioning in one layer cannot compensate for a bottleneck in another.

SRE Mapping

If you think in pipelines, the datacenter maps cleanly:

Each "pipeline" has an input, a capacity, a failure mode, and a monitoring signal. Treat them accordingly.

🧱Cage Design & Layout

A cage is typically a welded or bolted mesh enclosure (floor-to-ceiling or raised) within a shared colo floor. Solid-wall cages offer more security; mesh cages allow easier visual inspection and air flow monitoring. Most enterprise deployments use mesh with solid-panel overlays for higher-security zones.

Core Design Concepts

• Cold aisle / hot aisle containment — servers face the cold aisle (cool intake air), exhaust into the hot aisle. Containment systems prevent mixing of cold and hot air streams, dramatically improving cooling efficiency (PUE impact: ~0.1–0.2).
• Density planning (kW per rack) — standard enterprise racks run 5–10 kW; GPU/AI racks can reach 20–80 kW. Know your density before placing hardware. Overcrowding causes thermal runaway.
• Growth planning — maintain 30–50% free rack space and 20–30% free power headroom at all times. Running at 90%+ capacity removes your buffer for planned maintenance or emergency rerouting.

Physical Components

Layout Principles

• Centralize network racks to minimize cable runs and latency
• Separate compute, storage, and network into distinct rack zones
• Isolate high-density GPU/AI racks — they require dedicated cooling circuits
• Keep critical infrastructure (PDUs, OOB switches) accessible without disturbing production racks

Uptime Institute Tier Classification

Colo facilities are rated Tier I–IV. Your cage's reliability ceiling is determined by the facility tier.

🗄️Rack Engineering

Standard Rack Specifications

Vertical Rack Layout (Top → Bottom)

This ordering optimizes airflow, cable management, and center of gravity:

U-Space Planning

Track U-space in your DCIM or CMDB (e.g., NetBox). Each device consumes a specific U count. Common examples:

⚡Power Systems

Power is the most critical physical resource in a cage. A power failure cascades instantly to every system; no redundancy elsewhere compensates for lost power. Understand every step in the power chain.

Power Chain

ATS = Automatic Transfer Switch; STS = Static Transfer Switch; UPS = Uninterruptible Power Supply; PDU = Power Distribution Unit

Redundancy Model

Enterprise standard is A/B power feeds — two completely independent paths from separate utility feeds, separate UPS systems, separate PDUs, to dual PSUs in each server.

Power Concepts

kW vs kVA

kVA is apparent power (what the PDU is rated for); kW is real power (what the server actually consumes). The ratio is the power factor (PF). Modern server PSUs typically have PF ≥ 0.95, so kW ≈ kVA × 0.95. Always plan budgets in kW (actual consumption), but size circuits in kVA (what the PDU/breaker must handle).

Phase Balancing

Three-phase power is standard in datacenters. Distribute load evenly across phases (L1, L2, L3) to avoid overloading a single phase. Target imbalance < 10% between phases. Measure at the PDU breaker level.

Power Measurement Tools

• Smart PDUs (e.g., Raritan PX3, APC AP8900 series) — per-outlet monitoring, remote switching
• DCIM platforms (e.g., Nlyte, Sunbird) — aggregate power dashboards
• IPMI/BMC — per-server power consumption via Redfish or IPMI 2.0

PUE (Power Usage Effectiveness)

PUE = Total Facility Power ÷ IT Equipment Power. A PUE of 1.0 is theoretical perfection; modern facilities target 1.2–1.4. A PUE of 2.0 means as much energy is wasted on overhead (cooling, lighting) as powers IT gear — unacceptable by today's standards.

🌬️Cooling & Airflow

Thermal management is the second critical physical resource. Servers tolerate brief power interruption via UPS; they tolerate almost no thermal excursion — CPU throttling begins at ~70°C and emergency shutdown typically triggers at 85–95°C.

Cooling Models

ASHRAE Thermal Guidelines

ASHRAE TC 9.9 defines environmental envelopes for IT equipment:

Best Practices

• Blanking panels — fill every unused rack U. Without them, cold air short-circuits from cold aisle through the rack to hot aisle without cooling anything
• Cable cutout seals — use brush strips or grommets on raised floor cutouts to prevent hot-cold air mixing under-floor
• Airflow monitoring — deploy temperature sensors at rack inlet (U1–U3) and outlet (top of rack). Alert on inlet >27°C or delta T >15°C
• CRAC vs CRAH — CRAC (Computer Room Air Conditioner) uses DX refrigerant; CRAH (Computer Room Air Handler) uses chilled water. CRAH is more efficient at scale, requires chiller plant

🔌Cabling & Connectivity

Network Cable Types

Power Cable Standards

• C13/C14 — standard IEC 60320, up to 10A/15A; most 1U–2U servers
• C19/C20 — heavy-duty IEC 60320, up to 16A/20A; high-power servers, GPUs, storage
• NEMA L6-20 / L6-30 — locking connectors, used for PDU feeds in North American facilities
• IEC 60309 — industrial connectors (16A/32A), common in European colo facilities

Cable Management Principles

• Use horizontal cable managers (1U arm) at every patch panel and switch level
• Vertical cable trays on both sides of the rack for power and data segregation
• Label both ends of every cable — source and destination. Use consistent scheme: RACK-RU-PORT
• Velcro ties, not zip ties — zip ties crush cable jackets and make changes destructive
• Color coding example: Blue = data, Red = management/OOB, Yellow = SAN/storage, Black = power
• Separate power and data cables in different trays to minimize EMI

Fiber Optic Handling

• Never exceed the minimum bend radius (typically 10× cable diameter for fiber)
• Use LC connectors for most server/switch connections; MPO/MTP for high-density breakout
• Clean fiber end-faces before every insertion — a single dirty connector can degrade link performance or cause intermittent errors
• Document fiber runs in your DCIM: port-to-port mapping, connector type, length, dB loss measurement

🖥️Server Components

Key Subsystems

Hot-Swap Components

These can be replaced without powering down the server (vendor-dependent — always verify):

• PSU (when redundant)
• Drives in a RAID array (with proper RAID rebuild procedure)
• Fans (in most enterprise-grade servers)
• NIC in PCIe hot-swap bays (rare; OCP 3.0 mezzanine cards on some platforms)

IPMI / BMC / Redfish

Every enterprise server has an out-of-band management interface separate from the main OS network. Vendor naming varies: iDRAC (Dell), iLO (HPE), IPMI/BMC (Supermicro, Lenovo). The modern standard API is Redfish (RESTful HTTPS, replaces legacy IPMI 2.0 LAN commands).

• Always put BMC/iDRAC on a dedicated management VLAN (OOB network)
• Set IPMI access to management network only — never expose to the internet
• Use Redfish for programmatic provisioning and sensor polling

80 Plus Efficiency Standards

🌐Networking Architecture

Three-Tier vs Leaf-Spine

Traditional three-tier (access → aggregation → core) was designed for client-server traffic patterns with most traffic going north-south (user → server). It is increasingly inadequate for modern east-west traffic-heavy workloads (server-to-server, distributed systems, microservices).

Leaf-Spine Architecture (Modern Standard)

Every leaf switch connects to every spine switch. No leaf-to-leaf links. This provides predictable, low-latency, and equal-cost paths between any two servers in the fabric.

• ECMP (Equal-Cost Multi-Path) — multiple equal-cost paths are load-balanced, increasing effective bandwidth and providing automatic failover
• BGP as the fabric underlay — eBGP is increasingly used as the routing protocol within the datacenter fabric (RFC 7938)
• VXLAN overlay — tunneling protocol that extends Layer 2 segments over Layer 3 underlay; enables VM/workload mobility across the fabric
• Typical oversubscription — leaf ports (server-facing) to spine uplinks: 3:1 to 6:1 for standard compute; 1:1 for latency-sensitive workloads

Key Protocols & Technologies

🔐Physical & Logical Security

Physical Security Layers

• Cage locks — electronic locks with audit logging; dual-factor (badge + PIN or biometric) for critical cages
• Access logs — every cage entry/exit must be logged with timestamp and identity; retain for 90+ days minimum, often 12 months for compliance
• Cameras — minimum coverage: cage entrance, all rack fronts. Motion-triggered recording; retain 30–90 days of footage
• Escort policies — vendor technicians and colo staff must be escorted by your personnel; never allow unescorted access to your cage
• Tamper-evident labels — on server panels and drive bays to detect unauthorized component access
• Asset tagging — RFID or barcode on every device; reconcile against DCIM inventory quarterly

Logical Security

• Dedicated management network (OOB) — BMC/iDRAC on a separate VLAN/subnet, isolated from production traffic, accessible only via jump host
• Jump hosts (bastions) — all SSH/HTTPS access to servers routed through hardened bastion hosts with MFA and full session logging
• Network segmentation — firewall between production, management, storage, and public network zones
• Firmware / BIOS passwords — prevent unauthorized boot device changes or BIOS configuration
• Secure boot — enabled on all servers to prevent boot-time malware
• Drive encryption — full-disk encryption (AES-256) on all drives, especially in shared or multi-tenant environments

Zero Trust Principles

Apply zero-trust concepts even within the physical cage:

• No implicit trust based on physical location (being inside the cage does not grant network access)
• Authenticate every access — network, management plane, and physical
• Least-privilege access — operators have access to only the racks they manage
• Audit everything — access logs, CLI session recordings, change tickets

📊Monitoring & Observability

Facility-Level Monitoring

• PDU power draw — per-outlet kW, amps, power factor; alert on >80% circuit utilization
• Temperature & humidity — rack-level sensors at U1 (inlet) and top-of-rack (exhaust); alert on inlet >27°C
• Airflow — differential pressure sensors on cold aisle containment
• UPS status — battery health, bypass status, runtime remaining
• Generator test status — last tested, fuel level (facility-provided, but monitor

Hardware-Level Monitoring

• CPU temperature — via IPMI/Redfish; alert if CPU package temp >70°C sustained
• Memory ECC errors — correctable ECC errors are warning; uncorrectable = critical, immediate replacement
• Disk health — SMART attributes (reallocated sectors, pending sectors, uncorrectable errors); NVMe wear indicators
• Fan RPM — alert on fan failure or RPM significantly below expected
• PSU status — fault/OK per PSU via IPMI
• NIC errors — CRC errors, input errors, drops; alert on sustained non-zero error rates
• BMC connectivity — alert if OOB/IPMI unreachable (cannot manage the server remotely)

Observability Stack (Common)

Per-Rack Dashboard (Recommended)

Build a per-rack view showing: power draw (A + B feeds), inlet temp, outlet temp, live alerts, hardware fault count, and top-consuming servers. This lets an on-call engineer assess rack health at a glance during an incident without needing multiple tool windows.

🤖Automation & Source of Truth

The Automation Stack

Ideal Provisioning Flow

NetBox as Source of Truth

NetBox should be the authoritative record for: every rack, every device, every IP address, every VLAN, and every cable. All automation tooling reads from NetBox as its source of truth, not from ad-hoc scripts or spreadsheets. Treat NetBox updates as part of the change process — no physical change without a NetBox update.

🔥Failure Domains & Risk Management

Failure Mode Matrix

Failure Domain Design

A failure domain is the blast radius of a single failure. The goal is to minimize the blast radius by isolating components at appropriate granularity:

• Per-server — dual PSU, redundant NICs
• Per-rack — dual PDUs on separate feeds; dual ToR switches via MLAG
• Per-row — in-row cooling, separate power circuit
• Per-cage — dedicated cage power feeds, separate cooling zone
• Per-datacenter — geographic redundancy, site-to-site replication

🧾Operational Checklists

Daily Checks

• Review active alerts (power, temperature, network, hardware) — triage any P2+ alerts
• Verify UPS status — all units on utility, no battery-only conditions
• Check temperature dashboard — no rack inlets above 27°C
• Verify all critical servers are reachable (ping / OOB connectivity)
• Review access logs for unexpected cage entry events

Weekly Checks

• Review power capacity — no circuit above 70% average utilization
• Review network bandwidth utilization — no sustained link above 70%
• Check disk health dashboard — any drives with elevated SMART errors
• Verify cable integrity — no loose or unlabeled cables observed during any site visit
• Review OOB (BMC/IPMI) reachability — all nodes responding

Monthly Checks

• Audit physical inventory vs NetBox — walk the floor, reconcile discrepancies
• Test failover paths — intentionally fail one PDU feed and verify A/B redundancy
• Review and rotate credentials — PDU admin passwords, BMC accounts
• Check spare parts inventory — drives, PSUs, cables, transceivers
• Review access list — remove departed personnel

Quarterly Checks

• Disaster recovery simulation — full tabletop or live failover test
• Firmware updates — server BIOS, BMC/iDRAC, NIC firmware, switch OS
• Capacity planning review — project 6-month power, space, and network growth
• Review and update runbooks — ensure documentation reflects current architecture
• Compliance access review — formal review of all physical and logical access rights

🧠Advanced Concepts

Capacity Planning

Capacity planning is the practice of ensuring your cage has sufficient headroom — in power, cooling, space, and network — to accommodate planned growth without emergency procurement. Key metrics:

• Power budget per rack — track committed vs available kW per circuit; alert when committed reaches 70% of rated capacity
• Network oversubscription ratios — leaf uplink bandwidth ÷ server-facing port bandwidth. Typical: 4:1 (standard compute), 1:1 or 2:1 (latency-sensitive or storage)
• Space (U) utilization — per-rack and per-cage tracking; maintain 30% headroom for emergency rack swaps
• Cooling envelope — calculate actual heat load vs CRAC/CRAH rated capacity; target <70% utilization of cooling capacity

High-Density / AI Racks

GPU and AI accelerator racks (NVIDIA DGX, AMD Instinct clusters) present fundamentally different engineering challenges than standard compute:

Cost Awareness

Reliability Engineering

• MTBF (Mean Time Between Failures) — average time between hardware failures. Higher MTBF = more reliable hardware. Use vendor-published MTBF as a planning input, not a guarantee.
• MTTR (Mean Time To Repair/Restore) — average time to restore service after failure. This is the metric you control operationally: good runbooks, spare parts on-site, and practiced procedures reduce MTTR.
• Availability = MTBF ÷ (MTBF + MTTR). A device with 10,000h MTBF and 2h MTTR has 99.98% availability.
• SLOs for infrastructure — define and track SLOs for power availability, network uptime, and provisioning time. Without SLOs, you cannot tell if your infrastructure is improving.