RogerBW's Blog

Building a File Server 2: hardware 25 September 2017

In part 2 of this series on building a file server, I'll talk about hardware selection.

I tend to make the big storage array completely separate from the main operating system drive; when I started doing this there was little ZFS support in grub, so it made booting much easier. I run the operating system off SSD (well, actually a pair of SSDs in RAID-1); 160GB is plenty, and those (from Sandisk) were the smallest I could buy at the time, so that's what I've used.

You know how many discs you want. What are you going to put them into? Even eight-bay chassis aren't all that easy to come by, and you may find yourself using bay adaptors (e.g. squeezing three 3½" drives into two adjacent 5¼" bays) and other such fixes. I've been very happy with my 24-bay cases from X-Case, 4U rackmounts which have 24 hot-swap bays against a backplane which simplifies power and data connections. (Some of these come with port expanders so that you only need eight controller channels to run 24 discs; that's cheaper but slower than getting a full 24-port controller.) As always, consider power and cooling, bearing in mind that discs packed closely together can get quite hot.

Some of the X-Case boxes have brackets to hold the SSDs; or you can bodge your own.

You'll need controllers for these discs. For a small system, you can use the controllers supplied on the mainboard; remember to allow for the operating system drives. Mainboards with ten SATA controllers tend to be expensive and specialist items, and you already want a mainboard and CPU that support ECCRAM - really, you do, you're doing checksums on your data with this stuff - and lots of it, as much as you can cram on. I've usually ended up with Xeon CPUs; I have 32GB RAM on the current hardware iteration, and 64 would be better.

For a bigger array than the mainboard can support directly, two of the mainboard channels go to the mirrored SSDs, while the big array is driven off an interface card. (This helps keep things separate at a bus level, leading to less confusion if you're trying to repair the array in a hurry.)

6Gb/s SATA discs really want one PCI Express lane per drive to run at full speed, so a controller that's running eight drives should be in a ×8 slot; it'll probably work in a slower slot, but you'll be giving up performance. (A 24-port in a ×16 slot seems to be fine.)

The first controller I used was an AOC 8-port SATA card using standard sockets. (Looking at the design, it had clearly been upgraded from an earlier 6-port version.) That worked very well, but even with eight discs the wiring had a tendency to get out of control.

More professional controllers have "8087" connectors, each of which carries four SATA data channels; this will be connected either to another 8087 on the backplane, or to a "breakout" cable that feeds individual discs. (Note that a "reverse breakout" cable, to combine four individual SATA port controllers into one 8087 connector going to a backplane, is wired differently.) I've had reasonable success with the HighPoint Rocket 2720SGL (now EOL, but you can still get them off Amazon), with two 8087 ports to control eight drives, though there was no support for it in the FreeBSD 9 kernel. Linux had absolutely no problem with it.

Or if you're getting really serious and want 24 discs… you can in theory gang together three 8-port controllers, but in practice getting a mainboard with three ×8 slots tends to be difficult, so bite the bullet and get a 24-port controller. I'm now using a HighPoint 2760A in each of my active servers.

Absolutely do not buy a hardware RAID controller. This is going to be done with software, so you need a controller that lets you address the discs individually (often known as JBOD mode, Just a Bunch Of Discs); hardware RAID adds hugely to the cost, and causes trouble down the line when the controller goes bad and you can't find a replacement. If one of these controllers goes bad, you can plug in any other cheap JBOD controller and it'll still work.

It would be nice to have battery backup, to complete disc writes in case of power failure, but it tends to be stupidly expensive.

I've had enough bad experiences with Adaptec controllers that I now won't buy them even if they offer a JBOD one. (Even their JBOD modes have a habit of putting proprietary gubbins on the disc so that nothing else can read it.)

Don't forget that each disc needs a power connection too (again, backplane configuration may make this easier). Get a grunty PSU, but more importantly one where you can get enough power feeds out without having to use lots of splitters.

Tags: computing

See also:
Building a File Server 1: planning
Building a File Server 3: software
Building a File Server 4: maintenance


  1. Posted by John Dallman at 04:28pm on 25 September 2017

    Shame that Adaptec have gone bad. Their SCSI controllers were really good twenty years ago, but that's several generations of "new product direction!"

  2. Posted by RogerBW at 04:51pm on 25 September 2017

    I see that the Adaptec brand name and RAID hardware business were transferred to PMC-Sierra in 2010. I think that would be consistent with the start of my bad experiences with them. Particular annoyances:

    • a need for the exact same model of card to get a disc set recognised when the card has failed
    • horrible binary-only software for talking to the card from Linux (needed an obsolete libstdc just to run)
    • a fondness for saying "array degraded, but all discs working perfectly"
    • a fondness for going from apparently healthy RAID6 directly to total data loss (not, bizarrely, immediately after the above problem)

    Back when they were good, they were potentially worth the price premium. These days hardware RAID is close to obsolete for many purposes (part 3 will go into this in more detail) so it's no longer worth putting up with the annoyances.

Comments on this post are now closed. If you have particular grounds for adding a late comment, comment on a more recent post quoting the URL of this one.

Search
Archive
Tags 1920s 1930s 1940s 1950s 1960s 1970s 1980s 1990s 2000s 2010s 3d printing action advent of code aeronautics aikakirja anecdote animation anime army astronomy audio audio tech aviation base commerce battletech beer boardgaming book of the week bookmonth chain of command children chris chronicle church of no redeeming virtues cold war comedy computing contemporary cornish smuggler cosmic encounter coup covid-19 crime cthulhu eternal cycling dead of winter doctor who documentary drama driving drone ecchi economics en garde espionage essen 2015 essen 2016 essen 2017 essen 2018 essen 2019 essen 2022 essen 2023 existential risk falklands war fandom fanfic fantasy feminism film firefly first world war flash point flight simulation food garmin drive gazebo genesys geocaching geodata gin gkp gurps gurps 101 gus harpoon historical history horror hugo 2014 hugo 2015 hugo 2016 hugo 2017 hugo 2018 hugo 2019 hugo 2020 hugo 2022 hugo-nebula reread in brief avoid instrumented life javascript julian simpson julie enfield kickstarter kotlin learn to play leaving earth linux liquor lovecraftiana lua mecha men with beards mpd museum music mystery naval noir non-fiction one for the brow opera parody paul temple perl perl weekly challenge photography podcast politics postscript powers prediction privacy project woolsack pyracantha python quantum rail raku ranting raspberry pi reading reading boardgames social real life restaurant reviews romance rpg a day rpgs ruby rust scala science fiction scythe second world war security shipwreck simutrans smartphone south atlantic war squaddies stationery steampunk stuarts suburbia superheroes suspense television the resistance the weekly challenge thirsty meeples thriller tin soldier torg toys trailers travel type 26 type 31 type 45 vietnam war war wargaming weather wives and sweethearts writing about writing x-wing young adult
Special All book reviews, All film reviews
Produced by aikakirja v0.1