An iSCSI storage area network (SAN) offers numerous benefits for organizations both large and small. Since they are based on standard SCSI and TCP/IP protocols, iSCSI storage systems are relatively inexpensive. They require far less hardware (no specialized adapters are typically required) than Fibre Channel (FC) SAN deployments. In addition, the hardware that iSCSI uses tends to be much less expensive and much easier to implement, operate and manage. Moreover, far more IT staff members are familiar with the underlying Ethernet technology than with Fibre Channel. When everything is added up, the total cost of ownership of iSCSI is dramatically lower than Fibre Channel.
But the rapid growth of iSCSI SANs and storage systems have placed an enormous strain on Ethernet network administrators, many of whom have little to no SAN storage knowledge or experience. Many Ethernet network admins treat an iSCSI SAN the same way they treat a standard TCP/IP LAN. At first glance, this seems reasonable because iSCSI is SCSI mapped to TCP/IP. Therefore, iSCSI should behave like any TCP/IP packet, right? The short answer is no. Treating an iSCSI storage system like a standard LAN is a recipe for disaster.
Although the sales pitch about running the iSCSI SAN on your LAN infrastructure is factually true, it is not practical. The iSCSI storage system packet is different. It does not like latency and cannot tolerate any packet loss. The iSCSI protocol is hypersensitive to congestion, oversubscription and packet loss.
Take the example of an overloaded iSCSI path from the server to the storage. The TCP/IP protocol will normally drop some packets, and then requires them to be resent. This adds latency, and latency creates longer response times. In addition, if a network path carrying iSCSI storage traffic is oversubscribed, dropped packets must be resent, and performance degrades. That's not the worst part. The SCSI protocol, which is the key part of iSCSI, is notoriously impatient and can timeout relatively quickly. When that happens, the network admin's phone will flood with user calls. This is because the application using that storage will crash and all of the users will be down. In order to solve the problem, the servers and applications need to be rebooted.
iSCSI SAN network design considerations
How the network is designed makes all the difference as to whether or not the iSCSI SAN works well or is a constant source of headaches. It starts with the Ethernet adapter and switch.
Forget about using 10/100 Ethernet because it lacks the throughput to be effective for most production environments today. This means the network must minimally use Gigabit Ethernet; 10 Gigabit Ethernet is even better, especially in a virtualized server environment.
Next, forget about using iSCSI as a wide-area network (WAN) technology except for replication purposes. Any kind of distance adds both speed of light and TCP latency. Performance will be a problem over any distance outside a metropolitan area. Then there are the problems of SCSI timeouts and security concerns. So for the most part, keep the iSCSI SAN as a campus technology.
One network design best practice for the iSCSI SAN system is to segregate the iSCSI traffic from the general purpose TCP/IP traffic. The key rationale is so that the TCP/IP traffic does not compromise the iSCSI performance and vice versa. Storage traffic tends to be far more intense and demanding than TCP/IP networks are used to. Using the same LAN means one or both will suffer. Separation allows optimization of each without compromising the other. Utilization of layer 2 VLANs is a practical and efficient methodology for implementing this segregation. The best practice though is to have a dedicated LAN for the iSCSI traffic; do not share that network with other network traffic.
Oversubscription and iSCSI storage systems
Finally, be very careful of too much oversubscription and your iSCSI storage system. Oversubscription in general is a good thing because it allows higher utilization of resources and infrastructure. But too much of a good thing is not so good. Oversubscription allows more resources to be assigned than can be fulfilled if all users required them at the same time. All networks and SANs are designed with some amount of oversubscription. The key is to not overdo it. One little known fact to most Ethernet admins is that the rules of thumb used for standard TCP/IP networks are much higher than for iSCSI. In other words they just do not work on an iSCSI SAN. So if the Ethernet network admin oversubscribes the iSCSI SAN based on past TCP/IP experience, delays are certain and outages are more probable. In general, it's best not to oversubscribe or oversubscribe the dedicated LAN or layer 2 VLAN at a much lower level.
Overall, don't assume standard Ethernet best practices will work well with an iSCSI storage system. Understand the requirements of iSCSI SANs before implementation and everyone will be happier.
About the author: Marc Staimer is the founder, senior analyst, and CDS of Dragon Slayer Consulting in Beaverton, OR. The consulting practice of 11 years has focused in the areas of strategic planning, product development, and market development. With more than 28 years of marketing, sales and business experience in infrastructure, storage, server, software and virtualization, he's considered one of the industry's leading experts. Marc can be reached at firstname.lastname@example.org.
This was first published in October 2010