Recently, I was having dinner with some friends when somebody mentioned wanting to put a tray full of RaspberryPi boards in a server rack to use for random projects and shell accounts as an alternative to VMs. It sounded like a fun and impractical project, so I jumped right in!
First, I should explain that this is not meant to be a reasonable approach to a real world problem. If you simply wanted a server on the internet, Amazon Web Services and Google Cloud are great options, and likely far cheaper per CPU cycle than what I propose here. If you just want a bunch of RaspberryPis in a rack, it'd be easier to throw a power strip and some USB power bricks on a shelf and call it a day.
I want something more elegant. I want to supply 15 watts of power to each RaspberryPi, enough to be stable and run a USB device or two. I want to be able to remotely cycle the power on each RaspberryPi independently. I'd like to support at least eight RaspberryPis simultaneously and have it all fit in a 1U rackmount box. Ideally, this should all be done in a cost and power efficient manner.
With 8 boards at 15 watts, I'd need at least 120 watts of power. At 5 volts, that would be 24 amps peak load, which would require 15 gauge wire or 948 mil traces on a PCB with 1oz copper. While that's certainly doable, it seems a bit unwieldy.
Instead, I decided to start with a 48V power supply and step it down to 5v using "point of load" switching power supply modules. I designed this board (Fig. 1) around the GE Critical Power SSTW003A0A, which is a really nice isolated switching power supply. These modules come in a standard form factor, so if I wanted a higher wattage output in the future, I could just drop in a different module.
The 48V boards are definitely overengineered and overkill. I used Octopart's BOM Tool to get an estimate of the cost... $29.92 per port; and that's before adding in the cost of a controller board, wiring, enclosure, and the 48V power supply.
I went back to the drawing board and redesigned the board (Fig. 2) using a 12v supply, as 12v parts seem to be cheaper and more ubiquitous. I did away with the high end GE power modules and switched to a non-isolated Murata part instead. I trimmed down the power filtering and used just the minimum capacitors to get the job done. I added a PTC fuse for a bit of protection and changed the USB ports to be the dual-stacked ones, so that two devices could share a single 15W supply. All of these changes made the modules much cheaper at $10.57 per port in single quantities.
Next I focused on the controller board. I needed something that could manipulate the on/off pin of each port over an ethernet connection. I looked at using a microcontroller or low-end ARM chip, but quickly found that if I wasn't running a Linux variant, I'd likely be using some quirky network library or building a TCP stack from scratch. While that sounds fun, it's a bigger project than I want to tackle right now.
So if the controller board is to run Linux, the easiest and cheapest way I could come up with to get that going is to build my controller board as a RaspberryPi HAT (Figure 3). The HAT spec is pretty simple and just requires an I2C EEPROM for the kernel to enumerate at boot.
I could've just picked 8 GPIO pins and assigned one to each of the power control pins, but I wanted to leave open the possibility of controlling many more ports from a single control board. I dropped a 74HC595 8-bit shift register into the design and exposed the clock and shift out signals on a "chain" header that could be connected to another control board. In theory, this will allow me to control more ports by just shifting another byte out before toggling the latch.
I spent a couple days poking at the PCB layout and adding things like test pads, mounting holes for M2 screws, and nice silkscreen labels before deciding to pull the trigger on a batch of 3 control boards and 3 USB boards from OSH Park.
A 120 watt 12 volt power supply, 3 fans, 8 RaspberryPi 3 boards, 8 USB cables, 1 control board, and 8 USB boards comes out to $438.52 before shipping and taxes. That's $19.82 per RaspberryPi to supply it with a 15 watt remotely controllable power circuit.
I've ordered enough parts to assemble 3 control boards and a 3 of the USB boards for now. I expect that the PCBs won't be perfect the first time and I'll do another hardware revision before I get a full box of parts built.
Next I plan to build a Yocto based image for the control board and a kernel module to make the shift register a bit nicer to interact with. I still need to source a 1U enclosure for this project, although the idea of designing one using a service like Protocase or a local machine shop is appealing to me. I'm also not entirely happy with the 12 volt power supply I have picked out, and may try to find something nicer. Maybe an off the shelf server power supply that can run with load only on the 12v rail.Control board schematic
12 USB board schematic
Control board BOM
48V USB module BOM
12V USB module BOM
Full Assembly BOM