In the past few months it’s become clear to me that people are confusing Storage Spaces and Scale-Out File Server (SOFS). They seem to incorrectly think that one requires the other or that the terms are interchangeable. I want to make this clear:
Storage Spaces and Scale-Out File Server are completely different features and do not require each other.
Storage Spaces
The concept of Storage Spaces is simple: you take a JBOD (a bunch of disks with no RAID) and unify them into a single block of management called a Storage Pool. From this pool you create Virtual Disks. Each Virtual Disk can be simple (no fault tolerance), mirrored (2-way or 3-way), or parity (like RAID 5 in concept). The type of Virtual Disk fault tolerance dictates how the slabs (chunks) of each Virtual Disk are spread across the physical disks included in the pool. This is similar to how LUNs are created and protected in a SAN. And yes, a Virtual Disk can be spread across 2, 3+ JBODs.
Note: In WS2012 you only get JBOD tray fault tolerance via 3 JBOD trays.
Storage Spaces can be used as the shared storage of a cluster (note that I did not limit this to a SOFS cluster). For example, 2 or more (check JBOD vendor) servers are connected to a JBOD tray via SAS cables (2 per server with MPIO) instead of connecting the servers to a SAN. Storage Spaces is managed via the Failover Cluster Manager console. Now you have the shared storage requirement of a cluster, such as a Hyper-V cluster or a cluster running the SOFS role.
Yes, the servers in the cluster can be your Hyper-V hosts in a small environment. No, there is no SMB 3.0 or file shares in that configuration. Stop over thinking things – all you need to do is provide shared storage and convert it into CSV that is used as normal by Hyper-V. It is really that simple.
Yes, JBOD + Storage Spaces can be used in a SOFS as the shared storage. In that case, the virtual disks are active on each cluster node, and converted into CSVs. Shares are created on the CSVs, and application servers access the shares via SMB 3.0.
Scale-Out File Server (SOFS)
The SOFS is actually an active/active role that runs on a cluster. The cluster has shared storage between the cluster nodes. Disks are provisioned on the shared storage, made available to each cluster node, added to the cluster, and converted into CSVs. Shares are then created on the CSV and are made active/active on each cluster node via the active/active SOFS cluster role.
SOFS is for application servers only. For example Hyper-V can store the VM files (config, VHD/X, etc) on the SMB 3.0 file shares. SOFS is not for end user shares; instead use virtual file servers that are stored on the SOFS.
Nowhere in this description of a SOFS have I mentioned Storage Spaces. The storage requirement of a SOFS is cluster supported storage. That includes:
- SAS SAN
- iSCSI SAN
- Fibre Channel SAN
- FCoE SAN
- PCI RAID (like the Dell VRTX)
- … and SAS attached shared JBOD + Storage Spaces
Note that I only mentioned Storage Spaces with the JBOD option. Each of the other storage options for a cluster uses hardware RAID and therefore Storage Spaces is unsupported.
Summary
Storage Spaces works with a JBOD to provide a hardware RAID alternative. Storage Spaces on a shared JBOD can be used as cluster storage. This could be a small Hyper-V cluster or it could be a cluster running the active/active SOFS role.
A SOFS is an alternative way of presenting active/active storage to application servers. It requires cluster supported storage, which can be a shared JBOD + Storage Spaces.
Question on Storage Spaces: Can you build Storage spaces on top of existing hardware RAID?
Eg, you have 12 RAID1 pairs and you build a striped partition on top to make a full RAID10?
Not supported. Bad mojo will happen. When you think Storage Spaces, think JBOD (no h/w RAID at all).
So if you’re looking into the “clusterd hardware RAID” path (LSI Syncro CS 9286-8e), how do you acquire storage tiering with SSD/SAS? Since for what I understand, it’s when you’re running Storage Spaces (storage pool -> virtual disk) you will get this functionality and storage spaces on top of hardware RAID = not support. Do you use the RAID-controller’s functionality called CacheCade instead?
You don’t. Storage Spaces and “PCI RAID” are different techs. You do one or the other.
Thanks for the article.. There are so many different levels of abstraction these days that it’s getting tough to keep them all straight!
Can I make a CSV that consists of three drives, one in each of three separate 2012R2 servers (for example 3 servers each with a 2tb SATA drive to build out a single CSV that is a protected, shared 2TB disk)? My goal is to have a lab where each of three Hyper-V hosts houses a copy of ALL VMs on a shared CSV, so that I have fault tolerance, while allowing each Hyper-V host share the VM load.
I am envisioning this similar to how Exchange 2010 DAGs work – Each server has a copy of each DB which is all replicated and can be mounted on any node with a copy of it. In my analogy, the VMs would be like the DBs in that there will always be a copy replicated to each node, but any VM can run on any host.
Thanks!
-Steve
No.
Steve Jones: What you are looking for is this: http://hyperv.starwindsoftware.com/native-san-for-hyper-v-free-edition
The free edition allows a 128GB CSV to span 2 physical servers. You present that CSV to the Hyper-V cluster as a CSV drive and then put your VM”s on it. The StarWind software keeps the CSV in sync across the 2 nodes via the network.
The paid version allows unlimited CSV sizes across up to 3 physical hosts.
And will be totally unsupported by Microsoft.
I’m confused about this paragraph: “SOFS is for application servers only. (That I understand.) For example Hyper-V can store the VM files (config, VHD/X, etc) on the SMB 3.0 file shares. SOFS is not for end user shares; instead use virtual file servers that are stored on the SOFS.
Can I store the virtual file servers on the File Server Cluster as a Role?
?!?!?!?!?!?!?
No. Create a virtual machine to be your file server. Store that virtual machine on the SOFS. Simple.
Planning a storage spaces solution atm for our Hyper-V cluster. 4 Nodes, ~30 vm’s. Would you recommend a 2 node SOFS or a JBOD connected to all 4 servers using storage spaces?
Worried about the extra complexity/networking and potential windows update fail causing the SOFS to be less reliable that our current iSCSI SAN.
Would microsoft support a 4 node cluster running hyper-v with storage spaces in this way?
Yup, they’d support it, assuming the JBOD does. I’d recommend having a SOFS as a separate tier; this won’t limit your compute (Hyper-V) scalability.
I was one of those so confused — thanks for writing this!
Can Storage Spaces combine storage attached to more than one server? (Always looking for ways to reduce hardware count — like avoiding external shared JBODs.) If so, it seem like it could allow RAID1 across servers to help provide fault tolerance in small Hyper-V clusters. I wonder how SSD cache and dedupe would fit in. I’m probably dreaming, right?
While this article is very helpful in distinguishing SS from SOFS, I was left vaguely wondering where SOFS provides best value. Sounds like it’s best suited to providing file services fault tolerance for cluster servers — including file servers — in bigger-than-minimum-sized sites. It’s totally out of scope for minimum-sized SMB-style Hyper-V failover clusters. Is that right?
Answering my own question, it appears SS cannot combine physical storage attached to multiple servers. (Interesting — Hyper-V can, sort of.)
Further, shared JBODs cannot provide fault tolerance without pricey dual-port SAS drives, dual controllers, etc.
So if we drop back a notch to High Availability, accept lower robustness, and go with Hyper-V Replica Server and non-shared storage (does it make sense to use SS on single servers and add SSD cache?), or even a shared non-fault-tolerant JBOD with boot-from-backup Disaster Recovery…
Shop around on the JBODs – check the HCL for the Storage Spaces category to see what’s available. Also look at Cluster-in-a-Box (CiB).
Clustering = HA. Hyper-V Replica = disaster recovery (DR).
I thought I had it, but the more I read, the more I get confused. Thanks for the awesome site. What is the best scenario for 2 nodes with about 6TB each to host their Hyper-V environment (file shares, domain controllers, etc.)? Of course the customer only has 1 node in and the other coming after the fiscal year….
Consultants answer: that depends.
Good answer. It does depend…one customer went with the 2 node no shared, and the other with shared storage.
Would it make any sense to bust up LUNs on a SAN then bring them back together with simple storage space vdisk layout to create a sofs for hyper-v?
No it would not. You do not do Storage Spaces on a SAN to create CSVs. You use the SAN’s RAID technology to create CSVs.
Hello Aidan,
I’ve been having a lot of fun trying everything out but i wonder if you could explain something to me.
When i look in the cluster manager for roles and disks, my csv is linked to only one member of the cluster.
I am useing a smb3 sofs for vm’s at this moment but i never exceed the capacity of only one sofs member.
When i look at the performance monitor, only the active server is providing resources.
When i pull the plug out of a server in my two server cluster, the other takes over with quite a lot of missing pings.
I don’t seem to have an active/active solution but cluster validation reports that everything is fine.
Is this normal behaviour?
Regards, Jort
Yes. The CSV -ownership- is balanced. If you’re using Storage Spaces then SMB clients will be redirected to the CSV owner after the initial connection to get best throughput. The balancing of CSV ownership ensures the workload is balanced.
Let’s suppose we have 2 servers and 3 DAS trays with JBODs. Half of disks – SSD, other – SATA. Also we have 3 windows server 2012 r2 hyper-v hosts
Is it possible to make two storage spaces (high-speed and low-speed) with file shares and create 3-node hyper-v cluster using this storage spaces?
Will this solution be HA in case of tray or server fault?
Yes, assuming you spread the disks across the JBOD trays and implement 2-way or 3-way mirroring for your virtual disks.
Hi, can i use Hyper-V 2012 R2 (free version) for SOFS and use data deduplication and also use storage pools feature?
No. Hyper-V Server 2012 R2 is to be used as a host, and a host only.
So i have JBOD with 48 SAS Drives and 2 Dell R720. I installed windows 2012 R2 on both servers and configure windows failover cluster both nodes. Create Storage Pool with 2 Hot Spare. Now how this storage can be used by other servers as NFS or ISCSI ?
It can, but I’ve not bothered looking into using modern storage with stone age protocols.
Hi Aidan, AMAZING blog. Thanks for your continued efforts in writing it.
I have implemented Hyper-V clusters before using Dell SANs. But this time round, I’m going to use clustered storage spaces.
My question relates to general use file servers for end-user file shares.
Obviously, I can have SOFS and run VM fileservers no problem via my Hyper-V cluster, but can I have a SOFS and a General use file server configured from my Storage Server Cluster at the same time? so Hyper-V and SQL access the SOFS shares and my end users access the general use file server cluster role?
Many thanks,
Rich
You can implement traditional HA file shares for end users on the SOFS cluster in co-existance with the continuously available file shares for Hyper-V/SQL. They won’t be continuously available.
Hi Aidan
Thanks for your article – well explained.
We have an environment with 2 JBOD enclosures (with some SSD and some normal drives in them). They are identical. They are connected to two storage servers with two SAS controllers in each. When all connected, each storage server sees each JBOD drive twice (due to the dual controllers).
We’d like to create a highly available SOFS from this hardware. Am I right in thinking that I can simply create a big storage pool on each server and Storage spaces will work out that each drive listed twice is the same? Will storage spaces also automatically spread the data intelligently across both JBODs? DataOn tiering is also automatic too?
Why is it that to achieve JBOD redundancy you have to have 3 JBODs?
Finally I assume after setting up some virtual disks on your storage pool you can then promote them to CSVs using SOFS/Failover clustering.
Enable MPIO with LB policy. That’ll sort out your double vision issue.
One big pool will do the trick. Storage Spaces will do the spreading and tiering intelligently (that’s where the SES stuff comes in).
3 JBODs (for 2-way mirroring) is a quorum thing, like in clustering.
Yes, the virtual disks become CSVs.
I’ve been reading a lot of your articles lately and have been impressed with your SOFS mojo.
We will be setting up a two node SOFS to serve out VM storage for a 3 node Hyper-V cluster containing 90 VMs. The SOFS will contain one large CSV. I am looking for best practices for the number of shares to create to hold the vhdx files for the Hyper-V hosts. If I have one share, will all traffic be routed through one SOFS node? Or for better performace should I place my vhdx files on multiple shares?
Any recommendations?
Best practice from Microsoft is to have 1 share per CSV. There should be at least 1 CSV SOFS node – WS2012R2 CSV sharing in conjunction with SMB redirection optimizes the use of hardware to get the most performance with this configuration.
Now that you’re back from Ignite and recovering from the food, I have a followup question.
If best practice is 1 share per CSV, would I get better overall throughput with creating two CSVs, each with a single share from a single Storage Spaces pool?
I don’t have enough information on your SOFS. Let’s say it’s 2 nodes with shared JBOD(s). Then I’d have at least 1 CSV per node, giving you a minimum of 2 CSVs, and therefore 2 shares. Each CSV is owned by 1 node, balanced across both nodes, therefore each node owns 1 CSV. When hosts connect to the shares for the first time, they hit either one randomly. If they hit the non-owner, there is a brief moment with redirected IO across the SOFS cluster network(s) to the owner node. SMB redirects the host to the owner, therefore providing direct connection from the Host->Share->Owner node in the SOFS->CSV on the JBOD(s).
Fantastic explanation! I did mention earlier that it was a 2 node SOFS, (SuperMicro CiB with shared JBOD. So your example was right on the mark.
Thanks for sharing your expertise. Now I’ve got to find a way to budget the purchase of 8 enterprise SSDs for tiering…
Thanks, great article!
I’ve done a lot of reading and I’ve never seen it pointed out anywhere else that I can build a Hyper-V cluster with only 2 servers and one (or more) JBODs. The cluster I want to build is small-ish (12-15 VMs and only two that do any intense computing or disk I/O, and that only occasionally) but I want to maximize reliability. I thought I had to have at least 4 servers (2 for SOFS, 2 for Hyper-V) and the network h/w to support SMB Multipath between the Hyper-V hosts and the SOFS.
But, if I understand correctly, I could use 2 servers, 3 dual-controller JBODs connected in MPIO to multiple SAS HBAs in each server, teamed NICs and dual PSUs to eliminate any single point of failure. The only way I would ever even have to failover VMs and run on 1 server would be if a motherboard, CPU or RAM died. If that’s correct, that’s exactly what I want!
BTW, I bought your Hyper-V book and loved it!
Thanks
It’s an SME scneario, and the Windows Server teams don’t have much interest there – they focus on people with 1+ PB of storage in a single unit. JBOD + Storage Spaces is cluster-supported storage. Therefore 2 servers can use it for a cluster. That’s exactly what most of the CiB’s offer: 2 servers + JBOD. And that’s exactly what you’ve described.
Concerning Steve F’s observation of a “mini” cluster there is still a single point of failure, the cluster software itself. While clustering is much, much better nowadays, it is still a level of complexity that isn’t always needed. It sounds like a better solution for a small VM environment would be two identical server with DAS and using replicas. Each server could host half the VMs and house replicas of the corresponding server. Much simpler to setup and manage. Of course, changes would require some downtime but most, especially smaller shops will have windows available for maintenance/patches.
But that does not give you HA. That gives you DR. If someone want’s HA, then a cluster is the solution.
Hi Aidan,
we have a problem performance with NFS protocol.
Max performance to write is 15MB/s and the worse issue is that the “Higest Active Time” goes to 100%.
Why?
We have DataON JBOD.
Thanks
Mario
No idea. I have never deployed NFS. Have the shares got continuous availability turned on?
Yes of course, but NFS is not supported on CSV.
I dunno. As I said, I don’t work with NFS. I suspect you’ll have to go to MSFT to sort that out.
Hey Aidan, great blog.
I’m pretty new to Hyper-V, but have plenty of experience in VMware and I’m being asked to start building out Hyper-V clusters.
My question:
If we are using FC SAN (HP 3PAR) as our storage, is there any reason to be using SOFS instead of straight CSV? I am trying to get all of these options wrapped around my head.
Of course, if you already have an article on the pros/cons, I’d happily take that link in lieu of an answer.
At this point, probably nothing of enough of substance. Keep it simple and mainstream.
Thanks for the continued efforts. Just reconfirming my understanding since I’m doing a deployment.
So if one has a parallel SAS device like DELL PowerVault MD3220 one should use the Hardware RAID features and not bother with a clustered SS or SOFS.
If one has a JBOD with no HW RAID capabilities, it’s recommended to build,lets say, the 2-node Hyper-V cluster on the same 2 nodes as the clustered SS.
If one wants to build Hyper-V cluster over SOFS one needs to build a, lets say, 2-node SOFS and a 2-node Hyper-V cluster on 2 separate servers and in that case the NIC’s better be RDMA-enabled.
Yeah. You don’t need RDMA, but it improves things. If you’re going RDMA then check out Mellanox. Price them before you rule them out – trust me!
Hello,
I was wondering about
If one has a JBOD with no HW RAID capabilities, it’s recommended to build,lets say, the 2-node Hyper-V cluster on the same 2 nodes as the clustered SS.
Is there any performance issues?
None – that’s exactly what a cluster in a box (CiB) is. See DataON. In this scenario, there is no SMB 3.0 – you just store the VMs on the CSVs that Storage Space enables.
Hello,
We have a deployment like this:
2 Dell R630 servers with hyperV connected to Dell MD1400 (JBOD no raid capability, 3SSd, 9HHD) via DAS. The two HyperV are in cluster configured with storage spaces. How can i get the best performance it? Do i need to add SOFS role on existing Hyper V cluster?
Add 1 SSD (make it 4). Do not add the SOFS role. Best performance? MPIO set to least block, 2-way mirroring (mirroring is required for Hyper-V anyway), 64 KB interleave for the virtual disks (use PoSH, not FCM), use 64 KB allocation unit size when formatting the CSVs, and have 2 CSVs (1 per server that is SAS-attached to the JBOD). Spread the VMs evenly across the JBOD. Use a script to live migrate VMs to the owner of the CSV that they are stored on: https://aidanfinn2.wpcomstaging.com/?p=19576
Hi Aidan, thanks for the comprehensive info! Just a quick question if I may..
We have an iscsi SAN with 2 connected sas shelves connected to a storage network serving 16 blades and 10 2u servers.. we are thinking about SOFS, but don’t know if there is any efficiency or performance gains to be had by creating a couple csvs on the iscsi box and moving to SOFS with 2 or more File Servers. can you address that? Should we just stay with the current configuration using csv volumes?
Keep it simple. If you have a SAN, just use it as a SAN.
We have 1 NEC server connected to 2 x Intel JBODs with a single SAS HBA.
24 total SSD’s (12 in each JBOD)
1) Do you recommend any special Storage Spaces settings as this setup will be hosting SQL Databases?
2) PS command Get-Enclosure always shows the enclosure’s as “unhealthy” even after bios, firmware and driver updates for the HBA’s / Enclosures / Server etc.. It’s healthy according to the physical unit, the pools, vd’s and volumes all show healthy so I’m not sure it matters… but maybe you’ve dealt with this before.
Is your HBA and JBOD on the Storage Spaces HCL? Use PowerShell to configure the virtual disks so you can control the interleave size. Also take control of the allocation unit size when you format NTFS. Set interleave and allocation unit to the same size, which should be dictated by the nature of data in the database.
I a little confused still on the load balancing. If I have 6 CSVs with 3 nodes in my pool without SOFS, would that perform better than if I had 1 CSV with 3 nodes AND a SOFS role?
Hi Aidan,
I know this is a slightly older post but hoping you can help, planning on building a failover cluster environment 2xR630 running 2016 (at present only have one available so will be a single node cluster) /w H830 in HBA mode connected via DAC to MD1420 DAS. I believe using SOFS this should be able to work as a failover cluster, i’ve configured MPIO and been through the DELL setup guide an I think i’ve done everything right, but I can’t get Failover Cluster to create/add a storage pool.
Sorry Jack – I haven’t touched Storage Spaces on hardware in years.