AoE Explained

Here we explain what AoE is and how it is utilised in Coraid Ethernet Storage products. It is a useful guide to how it is different from regular iSCSI, and following this on the next page we look into the differences in more depth.

AoE Protocol • CORAID EtherDrive products use the AoE protocol to connect disks to servers using Ethernet. • ATA-over-Ethernet (AoE) is a thin protocol layer directly on top of Ethernet. • AoE is a block storage protocol. ATA disk commands (i.e. read disk sector x, write disk sector y) are put directly into standard Ethernet frames using the AoE protocol. • AoE is a non-routed protocol, therefore does not require IP or TCP protocol layers. This eliminates unnecessary processing and makes network connection to disks simple. • AoE is open source, and available free without a license.

AoE Driver • An AoE protocol driver in the host operating system provides the initiator function for the host to access storage connected to an Ethernet port. • The AoE driver bridges the operating systems block device driver interface to the hosts Ethernet driver. This allows the host to use a standard Ethernet NIC. • AoE target devices (ie. EtherDrive disk storage appliances) connect via standard Ethernet. • AoE can work with any OS. An AoE driver is included in the Linux 2.6.11+ kernels and available for other operating systems including FreeBSD, Solaris, MacOS X, and Microsoft Windows. • AoE target devices are just like local disks. They can be used like a normal hard disk drive, but since they are network connected, the disk can be shared with any network connected host. • AoE target devices can be partitioned like a normal disk and support any filesystem like a normal disk.

AoE is Block Storage • As with any block storage device, the host operating system can also create RAID sets from AoE devices. • The AoE block devices can be managed with storage virtualization tools such as logical volume managers and other volume management tools. • AoE devices naturally work with disk-to-disk backup software and virtual tape library software systems designed to work with hard disk drives. • AoE devices work with any filesystem or as directly accessed (raw) disks as required by some database applications.

Shared Storage • A single AoE device can be partitioned. • Each partition can be accessed by a separate server or application. • Each partition can have its own unique filesystem (ie. EXT3, JFS, XFS, Reiser, etc.).

Cluster Shared Storage • A single AoE device can be shared if a cluster filesystem is used (ie. GFS, Lustre, etc.). • The AoE protocol can be used for unrestricted access to the disk volume or can be used to apply access rules. • Using port based VLAN, access zoning can be established. • CORAID's EtherDrive SAN storage appliances (SR-series and LD-series) also support optional MAC address filtering to restrict server access for each logical AoE device. • CORAID's AoE storage devices support a "configuration string" which is written to the storage device. Once enabled, the config string can be used to restrict disk access to only hosts that use the config string in their AoE storage requests.

Reliable Storage Protocol   1. AoE initiator broadcasts an AoE configuration string message to discover AoE target devices on the network.  Initiators repeat broadcasts periodically. 2. AoE targets respond with available AoE device addresses.  This identifies what storage volumes are available on the network. 3. AoE initiator sends AoE messages with unique tag identifiers and ATA disk commands.  AoE targets allow multiple AoE messages to be queued enabling high performance. 4. AoE targets execute AoE messages (disk read/write commands) and echo message identity tags in the responses. 5. If no tag response is received, initiator resends AoE message.  Ethernet networks rarely drop frames, so retransmission is rare.

 

Simple Connections • CORAID EtherDrive SAN storage can be directly connected to a server using Ethernet. • Disks can be accessed individually identified by their shelf and slot number. CORAID refers to these AoE devices as storage "physical LUNs". For example a disk in slot #4 of shelf #1 would be accessed on a Linux system as /dev/etherd/e1.4, (also referred to as EtherDrive LUN e1.4) • AoE target devices are just like local disks. They can be used in addition to disks that may already be installed inside the server chassis. • RJ-45 Gigabit connections must use CAT6 Ethernet cables to ensure full operating speed. Cross-over cables are not required, since the Gigabit interface on CORAID's appliance auto senses Ethernet cable polarity.

 

RAID • CORAID EtherDrive SAN storage appliances include RAID controller functionality within the chassis to allow one or more RAID sets to be accessed as "logical" AoE devices called "LUNs". (ie. LUN 1.1 shown here is a RAID5 made up of 5 disks in slots 0, 1, 2, 3, and 4, and accessed as one large block storage device, such as /dev/etherd/e1.1) • A simple console port command line interface is provided on the Storage Appliance to configure the RAID setup. • RAID can also be implemented from the host, using server RAID software.

 

LAN Integration • Using standard Ethernet networking (ie. Ethernet switches) the number of AoE storage devices can be expanded. • Each EtherDrive storage appliance is assigned a unique shelf number. • AoE packets are standard Ethernet frames, and they can share a LAN with other LAN traffic, however CORAID recommends a separate storage network to ensure the highest storage access performance. (ie. separate from the client LAN traffic, as shown in the diagram)

 

Shared Storage Pool • Using standard Ethernet networking, multiple servers can access the same storage network. • Storage is not captive to one server, therefore if a server failure occurs, its storage can be mounted by a backup server. • Using Ethernet you can construct redundant connections between servers and storage, eliminating single points of failure. • Clusters of servers can share common storage using cluster filesystems.