The Cuddletech Veritas Volume Manager Series FC-AL Theory and Looping: Working with Sun Storage by: B. Rockwood benr@cuddletech.com An Overview ----------- FC-AL stands for Fibre Channel-Arbitrated Loop. Fibre Channel has quickly evolved into a standard for high speed storage. It is spelled Fibre rather than Fiber, because Fibre channel doesn't necessarily have to run on Fiber Optics, but can also run on 9pin copper. Fibre Channel, at its simplest, is a high speed serial interconnect transport. It's a method by which to move bits. Arbitrated Loop, however, is the protocol that governs how thoughs bits move. There are two common Fibre Channel protocols, Arbitrated Loop (FC-AL) and Switched Fabric (FC-SW). This tutorial is intended to make you more accustom to how FC-AL works, in relation to Sun's fibre channel storage devices; namely the A5000 family of arrays. We'll discuss how FC-AL works, how to cable for FC-AL, and common misconceptions, as well as some performance topics. We will not, however, discuss FC-SW. FC-SW is what most people consider the basis of a SAN. While we won't talk about building SAN's, understanding FC-AL will put you ahead of the game. FC-SW may be discussed at a later point in time, but at this point I have yet to get the materials I need to make me really comfortable with it, most notably, I need the hardware to test. If you are purely interested in SAN's, please refer to the websites of both Vixel (www.vixel.com), and Brocade (www.brocade.com). Both sites offer several wonderful whitepapers on the subject. The Very Basics --------------- Sun offers several Fibre Channel products, all predominantly utilizing Arbitrated Loop. Generally you'll use 3 components: 1) a FC-AL Host Bus Adapter (HBA) 2) a Fibre Channel Hub 3) a FC-AL Array, such as the A5200 The first component is on the server-side. Get used to calling FC-AL cards, "HBA's", you'll get in trouble with some people is you insist on calling them "f cal cards". (Trust me, I know.) You'll also recall that SCSI controllers are also called HBA's, but most people don't refer to them that way. As I was saying, Sun sells several FC-AL HBA's, in both PCI and SBus varieties. Most commonly used is the SBus Dual-Ported FC-AL, which is "dual ported" because it has two FC-AL links on a single card. Then we have the on-board SOC+ FC-AL HBA, found on Sun Midrange Type 4 I/O Boards. Then you'll also generally see PCI Single Port HBA's in the PCI systems. All of these work the same once in the system, so don't worry. One thing to note about Sun and HBA's is that they recently introduced a new FC HBA that can do both AL, and Gigabit Ethernet (GE) on the same card! It's dual ported, obviously. This is a BIG step forward. Previously you have FC-AL cards, and GE cards an ne'er should the two collide. Before we leave the immediate topic of HBA's, you should know that HBA's aren't just on the host side, they are also in the arrays. It's the same controller, just in a different locale. You probably already figured that out. But if your really new to FC, you should also know that in each HBA port you'll have a slot, in which you slide a GBIC (Gigabit Interface Connector) in. GBIC's actually convert electronic voltage or bits, into light which is blasted across our fiber cable. So on the host and array side, we'll have an HBA, with a GBIC in it, which the fiber cable actually plugs into. GBIC's are always hot swappable (you'll loose a link, but GBIC's are configured) and are considered unreliable due to heat and other factors involved in the light conversion. Therefore you should always have extra GBIC's laying around. It's fairly uncommon for GBIC's to fail, but they are more failure prone than disks, so keep that in mind. The other component mentioned is a fibre channel hub. FC hubs allow us to plug multiple devices (arrays) into a single HBA. We'll get more specific about this in a bit, but for now, that's why we like 'em. I hope we're clear so far about the components we're going to use. To illustrate at this point, let's say I'm going to attach a A5200 to an Ultra2. I'd put a SBus FC HBA in the Ultra2, then slide a GBIC into the HBA port. Next I'd make sure my A5200's HBA had a GBIC in it, then I'd simply plug a fiber cable into both GBIC's. That's all that we'd need to do to get a basic setup going, on the hardware end. This isn't a complete picture, but I want you to have an idea of what we're playing with here. Fibre Channel, and how AL works ------------------------------- Earlier I mentioned that Fibre Channel is a high speed serial interface... the key here is SERIAL. Keep that in mind as we move forward. Fibre Channel could be compared to Ethernet. It's a low level transport, nothing more. Its the higher level protocols that do the real work, and magic of moving data. Currently FC can move data a "1 Gigabit/s". Don't be demystified when you read some places that 1 Gigabit is actually 100MB/s. This really gets some people. Don't get mixed up with FastEthernet. Remember that FastEther moves data at a rate of 100 Megabits/second, while Fibre Channel moves data at a rate of 100 MegaBYTES/second. So if you do the math, and work on the terminology, FastEther actually moves data at 12.5 MegaBytes/second, and FC moves data at 100 MegaBytes/second. (12.5MB/s = 100Mbit / 8) Now, with that out of the way, lets jump into how AL works. Arbitrated Loop can be thought of like a token ring. Now don't get carried away with that analogy, but it'll get you in the ball park. When a loop is formed, you have a bunch of devices (up to 127) that can communicate on the loop. Modern FC-AL disks can talk directly on the loop, such as in the A5x000's. Each device is called a node. A "node" is any devices that can talk on the loop, including HBA's, Disks, and the like. Note that GBIC's themselves, or hubs aren't nodes, they just move the data, they don't talk. If you look through books on Fibre Channel you'll find whole chapters about different types of nodes. The nodes found on Sun systems are "NL_Ports" (meaning: Node,arbitrate Loop_Port). Don't worry about calling ports by that name, but keep it in mind if you pick up any books on FC. So, we've got all these nodes on the the loop. Only two devices can talk to each other at a time. When someone on the loop wants to talk they call for arbitration. Essentially, the node says it wants to talk to someone, and waits for a chance to take over the loop. When a node takes over the loop, its said that the node won arbitration. There is no master node that controls the loop (no single point of failure!), each node fights for itself. Once arbitration is won, the node connects to who it needs to (generally the host) and moves the data. Once the transfer is done, someone else takes over the loop by winning arbitration, and so on. What you should be noticing now is that everything on the loop is _serial_ like mentioned earlier. One after another they move data, not all at the same time. The reason that we can move data in serial, however, is that FC is so damned fast. We'll talk about this more later, but try to keep this in mind when building your loops.