I read a comment today that Storage Spaces was great for small/medium deployments. And yup, it is. I use Storage Spaces to store my invaluable photo library at home (a pair of Toshiba USB 3.0 3 TB drives). At work, we use a single DataOn Storage DNS-1640 24 x slot JBOD that is dual SAS attached to a pair of 2U servers to create an economical Hyper-V cluster. And we have sold 2U DataOn Storage CiB-9220 “Cluster in a Box” units for similar deployments in SMEs.
But most of our sales of JBODs have actually been for larger deployments. Let me give you an example of scalability using an image from my software-defined storage slide decks:
In the above diagram there are 4 x DataOn Storage DNS-1660 JBODs. Each has 60 x 3.5” disk slots. Using 6 TB drives (recently certified by DataOn) that gives you up to 1440 TB or just over 1.4 petabytes of raw storage. That’s with 7200 RPM drives and that just won’t do. We can mix in some dual chanel SAS SSDs (using 3.5 to 2.5 adapters) to offer peak performance (read and write).
In the above design there are 4 SOFS cluster nodes, each having 2 x direct SAS connections to each JBOD – 4 JBODs therefore 8 SAS connections in each server. Remember that each SAS cable has 4 SAS ports. So a 6 Gb SAS cable actually offers 24 Gbps of throughput.
Tip from DataOn: If you’re using more than 48 drives then opt for 12 Gb SAS cards, even if your JBOD runs at 6 Gb; the higher spec cards circuitry performs better even with the lower speed SAS disks/JBODs.
Now this is where you say that this is all great in theory but surely no one is doing this. And there you would be wrong. Very wrong. MVP Carsten Rachfahl has been deploying large installations since late 201 in Germany. The same is also true of MVPs Thomas Maurer and Michael Rüefli in Switzerland. At my job, we’ve been selling quite a few JBODs. In fact, most of those have been to replace more expensive SAN installations from legacy vendors. This week I took this photo of the JBODs in the above architecture while they were passing through our warehouse:
Yup, that’s potentially over 1 PB of raw storage in 16U of rack space sitting on one shipping pallet. The new owner of that equipment is building a SAS solution that will run on Hyper-V and use SMB 3.0 storage. They’ll scale out bigger and cheaper than they would have done with their incumbent legacy storage vendor – and that’s why they’re planning on buying much more of this kind of storage.
That’s true – Storage Spaces is scallable – you can add more JBODs and more Storage Spaces servers anytime, low-costs and live.
I will test some h/w for performance with 24 disks DELL PV1220.
It’s pity but DataOn is still not available in Poland – we can work only with Dell, Supermicro…
If you are a reseller then there are distributors in the Euro zone, e.g. http://www.mwh.ie
I’m not a reseller, just integrator 🙁
Hi Aidan,
Could you elaborate a bit on “Remember that each SAS cable has 4 SAS ports. So a 6 Gb SAS cable actually offers 24 Gbps of throughput.”
I tried to find out on;
https://en.wikipedia.org/wiki/Serial_attached_SCSI
http://www.scsita.org/
but mostly found out that is was behind reading on the SAS interface..
I did find info on the demartek site (http://www.demartek.com/Demartek_Interface_Comparison.html) about 4 and 8 lane connectors, which speaks of 48Gbps and 96Gbps, is that all usable max bandwith? Or does it vary per vendor or device? Dell just speaks of ports as physical ports (connectors) so there is some confusion there also. I tried to find out how much bandwidth our newly ordered unit a Dell MD3420 has, but all i can find is that the unit has 12Gbps connectors..
A 6 GB SAS cable has 4 simultaneous of communications – 1 cable with 1 plug on each end. That gives you a maximum of 24 Gbps of bandwidth. With 2 cables (MPIO) you have 48 Gbps. As for usability – consult your h/w manufacturer.
Hi Aidan, very nice post!!
I already took note about using 12 Gb SAS even with 6 Gb disk enclosure, which I wonder when I was reading the white paper of Microsoft about “achieving over 1-million IOPS…”.
I am expecting to receive in the following week a Quanta M4600H enclosure with 60x4TB disks. Thinking about the server, I will use 2xLSI 9300-8e 12 Gb SAS (after reading your blog). Which is not totally clear to me is how to get the maximum throughput to one of this JBODs, that most of them have a dual-io SAS interface with 2x4Ports, that doing the maths should be a total of 32 lanes at 6 Gbps if the SAS expander controller at the JBOD do that. DataOn has an sketch showing that their JBOD uses a 36 lanes controller.
My question is that if I use a single JBOD, should I connect all the ports of one LSI to one SAS interface (2 cables miniSAS) and the other LSI to the other interface? That should give 96 Gbps.
Thanks a lot!
Typically each node has 2 connections to each JBOD, with MPIO enabled for LB policy.
Hi Juan Sillero,
How is your experiance with the Quanta m4600h enclosure?
We also have one in production but we are having stability issues.
I hope to hear from you.
Regards,
Jeroen Bosman