Theory of Operation & Deployment Choices

Cinder Backends and Storage Virtual Machines.  Storage Virtual Machines (SVMs, formerly known as Vservers) contain data volumes and one or more LIFs through which they serve data to clients. SVMs can either contain one or more FlexVol volumes, or a single Infinite Volume.

SVMs securely isolate the shared virtualized data storage and network, and each SVM appears as a single dedicated storage virtual machine to clients. Each SVM has a separate administrator authentication domain and can be managed independently by its SVM administrator.

In a cluster, SVMs facilitate data access. A cluster must have at least one SVM to serve data. SVMs use the storage and network resources of the cluster. However, the volumes and LIFs are exclusive to the SVM. Multiple SVMs can coexist in a single cluster without being bound to any node in a cluster. However, they are bound to the physical cluster on which they exist.

	Important
	When deploying Cinder with clustered Data ONTAP, NetApp recommends that each Cinder backend refer to a single SVM within a cluster through the use of the netapp_vserver configuration option. While the driver can operate without the explicit declaration of a mapping between a backend and an SVM, a variety of advanced functionality (e.g. volume type extra-specs) will be disabled.

Cinder volumes and FlexVol volumes.  Data ONTAP FlexVol volumes (commonly referred to as volumes) and OpenStack Block Storage volumes (commonly referred to as Cinder volumes) are not semantically analogous. A FlexVol volume is a container of logical data elements (for example: files, Snapshot™ copies, clones, LUNs, et cetera) that is abstracted from physical elements (for example: individual disks, and RAID groups). A Cinder volume is a block device. Most commonly, these block devices are made available to OpenStack Compute instances. NetApp’s various driver options for deployment of FAS as a provider of Cinder storage place Cinder volumes, snapshot copies, and clones within FlexVol volumes.

	Important
	The FlexVol volume is an overarching container for one or more Cinder volumes.

	Note
	NetApp's OpenStack Cinder drivers are not supported for use with Infinite Volumes, as Data ONTAP currently only supports FlexClone files and FlexClone LUNs with FlexVol volumes.

Cinder volume representation within a FlexVol volume.  A Cinder volume has a different representation in Data ONTAP when stored in a FlexVol volume, dependent on storage protocol utilized with Cinder:

 Cinder Scheduling and resource pool selection.  When Cinder volumes are created, the Cinder scheduler selects a resource pool from the available storage pools: see the section called “Storage Pools” for an overview. Table 4.8, “Behavioral Differences in Cinder Volume Placement” details the behavioral changes in NetApp's Cinder drivers when scheduling the provisioning of new Cinder volumes.

Cinder snapshots versus NetApp Snapshots.  A NetApp Snapshot copy is a point-in-time file system image. Low-overhead NetApp Snapshot copies are made possible by the unique features of the WAFL storage virtualization technology that is part of Data ONTAP. The high performance of the NetApp Snapshot makes it highly scalable. A NetApp Snapshot takes only a few seconds to create — typically less than one second, regardless of the size of the volume or the level of activity on the NetApp storage system. After a Snapshot copy has been created, changes to data objects are reflected in updates to the current version of the objects, as if NetApp Snapshot copies did not exist. Meanwhile, the NetApp Snapshot version of the data remains completely stable. A NetApp Snapshot incurs no performance overhead; users can comfortably store up to 255 NetApp Snapshot copies per FlexVol volume, all of which are accessible as read-only and online versions of the data.

Since NetApp Snapshots are taken at the FlexVol level, they can not be directly leveraged within an OpenStack context, as a user of Cinder requests a snapshot be taken of a particular Cinder volume (not the containing FlexVol volume). As a Cinder volume is represented as either a file on NFS or as a LUN (in the case of iSCSI or Fibre Channel), the way that Cinder snapshots are created is through use of Data ONTAP's' FlexClone technology. By leveraging the FlexClone technology to facilitate Cinder snapshots, it is possible to create many thousands of Cinder snapshots for a single Cinder volume.

FlexClone files or FlexClone LUNs and their parent files or LUNs that are present in the FlexClone volume continue to share blocks the same way they do in the parent FlexVol volume. In fact, all the FlexClone entities and their parents share the same underlying physical data blocks, minimizing physical disk space usage.

E-Series snapshots.  The cinder driver can create hardware-based snapshots on E-Series. The E-Series uses copy-on-write snapshots, which can be created within seconds. In order to ensure snapshots can be deleted in any order, the cinder driver limits E-Series to 3 snapshots per volume. Snapshots on E-Series do not require an additional license.

E-Series snapshots are typically used for relatively brief operations, such as making a backup. If you require many snapshots or long-lasting snapshots, consider FAS.

	Important
	When Cinder is deployed with Data ONTAP, Cinder snapshots are created leveraging the FlexClone feature of Data ONTAP. As such, a license option for FlexClone must be enabled.

The NetApp Cinder driver can operate in two independent modes: a direct mode where the Cinder processes directly interact with NetApp FAS storage systems, and an intermediated mode where the Cinder processes interact with an additional software entity that issues provisioning and management requests on behalf of Cinder.

OnCommand® Workflow Automator.  OnCommand® Workflow Automator (WFA) is a flexible framework that provides automation for storage-related tasks, customization, scripting capabilities, and integration with higher-order IT systems such as orchestration software through web services.

While WFA can be utilized in conjunction with the NetApp unified Cinder driver, a deployment of Cinder and WFA does introduce additional complexity, management entities, and potential points of failure within a cloud architecture. If you have an existing set of workflows that are written within the WFA framework, and are looking to leverage them in lieu of the default provisioning behavior of the Cinder driver operating directly against a FAS system, then it may be desirable to use the intermediated mode.

SANtricity® Web Services Proxy.  The NetApp SANtricity® Web Services Proxy provides access through standard HTTPS mechanisms to configuring management services for E-Series storage arrays. You can install Web Services Proxy on either Linux or Windows. As Web Services Proxy satisfies the client request by collecting data or executing configuration change requests to a target storage array, the Web Services Proxy module issues SYMbol requests to the target storage arrays. Web Services Proxy provides a Representative State Transfer (REST)-style API for managing E-Series controllers. The API enables you to integrate storage array management into other applications or ecosystems.

Recommendation

Unless you have a significant existing investment with OnCommand Workflow Automator that you wish to leverage in an OpenStack deployment, it is recommended that you start with the direct mode of operation when deploying Cinder with a NetApp FAS system. When Cinder is used with a NetApp E-Series system, use of the SANtricity Web Services Proxy in the intermediated mode is currently required. The SANtricity Web Services Proxy may be deployed in a highly-available topology using an active/passive strategy.

FAS.  If rich data management, deep data protection, and storage efficiency are desired and should be availed directly by the storage, the NetApp FAS product line is a natural fit for use within Cinder deployments. Massive scalability, nondisruptive operations, proven storage efficiencies, and a unified architecture (NAS and SAN) are key features offered by the Data ONTAP storage operating system. These capabilities are frequently leveraged in existing virtualization deployments and thus align naturally to OpenStack use cases.

E-Series.  For cloud environments where higher performance is critical, or where higher-value data management features are not needed or are implemented within an application, the NetApp E-Series product line can provide a cost-effective underpinning for a Cinder deployment. NetApp E-Series storage offers a feature called Dynamic Disk Pools, which simplifies data protection by removing the complexity of configuring RAID groups and allocating hot spares. Utilization is improved by dynamically spreading data, parity, and spare capacity across all drives in a pool, reducing performance bottlenecks due to hot-spots. Additionally, should a drive failure occur, DDP enables the pool to return to an optimal state significantly faster than RAID6, while reducing the performance impact during the reconstruction of a failed drive.

	Note
	As of the Icehouse release, NetApp has integrations with Cinder for both FAS and E-Series, and either storage solution can be included as part of a Cinder deployment to leverage the native benefits that either platform has to offer.

Clustered Data ONTAP represents NetApp’s platform for delivering future innovation in the FAS product line. Its inherent qualities of virtualization of network interfaces, disk subsystem, and administrative storage controller map well to OpenStack constructs. The Storage Virtual Machine storage server (SVM, historically referred to as Vserver) can span across all nodes of a given clustered Data ONTAP deployment, for example. The elasticity provided to expand or contract a Storage Virtual Machine across horizontally scalable resources are capabilities critical to cloud deployment unique to the clustered Data ONTAP mode of operation.

The Data ONTAP 7-Mode drivers are primarily provided to allow rapid use of prior deployed FAS systems for OpenStack block storage requirements. There is no current intention to enhance the 7-Mode driver’s capabilities beyond providing basic bug fixes.

	Recommendation
	NetApp strongly recommends that all OpenStack deployments built upon the NetApp FAS product set leverage clustered Data ONTAP.

A frequent question from customers and partners is whether to utilize NFS or iSCSI as the storage protocol with a Cinder deployment on top of the NetApp FAS product line. Both protocol options are TCP/IP-based, deliver similar throughputs and latencies, support Cinder features, snapshot copies and cloning are supported to similar degrees, as well as advertisement of other storage efficienty, data protection, and high availability features.

iSCSI. 

NFS. 

	Recommendation
	Deploying the NetApp Cinder driver with clustered Data ONTAP utilizing the NFS storage protocol yields a more scalable OpenStack deployment than iSCSI with negligible performance differences. If Cinder is being used to provide block storage services independent of other OpenStack services, the iSCSI protocol must be utilized.

Tip

A related use case for the use of iSCSI with OpenStack deployments involves creating a FlexVol volume to serve as the storage for OpenStack compute nodes. As more hypervisor nodes are added, a master boot LUN can simply be cloned for each node, and compute nodes can become completely stateless. Since the configuration of hypervisor nodes are usually nearly identical (except for node-specific data like configuration files, logs, etc), the boot disk lends well to optimizations like deduplication and compression. Currently this configuration must be done outside of the management scope of Cinder, but it serves as another example of how the differentiated capabilities of NetApp storage can be leveraged to ease the deployment and ongoing operation of an OpenStack cloud deployment.

Cinder includes a Fibre Channel zone manager facility for configuring zoning in Fibre Channel fabrics, specifically supporting Cisco and Brocade Fibre Channel switches. The user is required to configure the zoning parameters in the Cinder configuration file (cinder.conf). An example configuration using Brocade is given below:

zoning_mode=fabric 

[fc-zone-manager]
fc_fabric_names=fabricA,fabricB
zoning_policy=initiator-target
brcd_sb_connector=cinder.zonemanager.drivers.brocade.brcd_fc_zone_client_cli.BrcdFCZoneClientCLI
fc_san_lookup_service=cinder.zonemanager.drivers.brocade.brcd_fc_san_lookup_service.BrcdFCSanLookupService
zone_driver=cinder.zonemanager.drivers.brocade.brcd_fc_zone_driver.BrcdFCZoneDriver

[fabricA] 
fc_fabric_address=hostname
fc_fabric_user=username
fc_fabric_password=password
principal_switch_wwn=00:00:00:00:00:00:00:00

[fabricB] 
fc_fabric_address=hostname
fc_fabric_user=username
fc_fabric_password=password
principal_switch_wwn=00:00:00:00:00:00:00:00
            

	Important
	While OpenStack has support for several Fibre Channel fabric switch vendors, NetApp has validated their drivers with the use of Brocade switches. For more information on other vendors, refer to the upstream documentation.

The Storage Service Catalog (SSC) is a concept that describes a set of capabilities that enables efficient, repeated, and consistent use and management of storage resources by the definition of policy-based services and the mapping of those services to the backend storage technology. It is meant to abstract away the actual technical implementations of the features at a storage backend into a set of simplified configuration options.

The storage features are organized or combined into groups based on the customer needs to achieve a particular scenario or use case. Based on the catalog of the storage features, intelligent provisioning decisions are made by infrastructure or software enabling the storage service catalog. In OpenStack, this is achieved together by the Cinder filter scheduler and the NetApp driver by making use of volume type extra-specs support together with the filter scheduler. There are some prominent features which are exposed in the NetApp driver including mirroring, de-duplication, compression, and thin provisioning.

When the NetApp unified driver is used with clustered Data ONTAP and E-Series storage systems, you can leverage extra specs with Cinder volume types to ensure that Cinder volumes are created on storage backends that have certain properties (e.g. QoS, mirroring, compression) configured.

Extra specs are associated with Cinder volume types, so that when users request volumes of a particular volume type, they are created on storage backends that meet the list of requirements (e.g. available space, extra specs, etc). You can use the specs in Table 4.9, “NetApp supported Extra Specs for use with Cinder Volume Types” later in this section when defining Cinder volume types with the cinder type-key command.

When a thick-provisioned Cinder volume is created, an amount of space is reserved from the backend storage system equal to the size of the requested volume. Because users typically do not actually consume all the space in the Cinder volume, overall storage efficiency is reduced. With thin-provisioned Cinder volumes, on the other hand, space is only carved from the backend storage system as required for actual usage. A thin-provisioned Cinder volume can grow up to its nominal size, but for space-accounting only the actual physically used space counts.

Thin-provisioning allows for over-subscription because you can present more storage space to the hosts connecting to the storage controller than is actually currently available on the storage controller. As an example, in a 1TB storage pool, if four 250GB thick-provisioned volumes are created, it would be necessary to add more storage capacity to the pool in order to create another 250GB volume, even if all volumes are at less than 25% utilization. With thin-provisioning, it is possible to allocate a new volume without exhausting the physical capacity of the storage pool, as only the utilized storage capacity of the volumes impacts the available capacity of the pool.

Thin-provisioning with over-subscription allows flexibility in capacity planning and reduces waste of storage capacity. The storage administrator is able to simply grow storage pools as needed to fill capacity requirements.

As of the Liberty release, all NetApp drivers conform to the standard Cinder scheduler-based over-subscription framework as described here, in which the max_over_subscription_ratio and reserved_percentage configuration options are used to control the degree of over-subscription allowed in the relevant storage pool. Note that the Cinder scheduler only allows over-subscription of a storage pool if the pool reports the thin-provisioning-support capability, as described for each type of NetApp platform below.

The default max_over_subscription_ratio for all drivers is 20.0, and the default reserved_percentage is 0. With these values and thin-provisioning-support capability on (see below), if there is 5TB of actual free space currently available in the backing store for a Cinder pool, then up to 1,000 Cinder volumes of 100GB capacity may be provisioned before getting a failure, assuming actual physical space used averages 5% of nominal capacity.

Data ONTAP Thin Provisioning.  In Data ONTAP multiple forms of thin-provisioning are possible. By default, the nfs_sparsed_volumes configuration option is True, so that files that back Cinder volumes with our NFS drivers are sparsely provisioned, occupying essentially no space when they are created, and growing as data is actually written into the file. With block drivers, on the other hand, the default netapp_lun_space_reservation configuration option is 'enabled' and the corresponding behavior is to reserve space for the entire LUN backing a cinder volume. For thick-provisioned Cinder volumes with NetApp drivers, set nfs_sparsed_volumes to False. For thin-provisioned Cinder volumes with NetApp block drivers, set netapp_lun_space_reservation to 'disabled'.

With Data ONTAP, the flexvols that act as storage pools for Cinder volumes may themselves be thin-provisioned since when these flexvols are carved from storage aggregates this may be done without space guarantees, i.e. the flexvols themselves grow up to their nominal size as actual physical space is consumed.

Data ONTAP drivers report the thin-provisioning-support capability if either the files or LUNs backing cinder volumes in a storage pool are thin-provisioned, or if the flexvol backing the storage pool itself is thin-provisioned. Note that with Data ONTAP drivers, the thin-provisioning-support and thick-provisioning-support capabilities are mutually-exclusive.

E-Series Thin Provisioning.  E-Series thin-provisioned volumes may only be created on Dynamic Disk Pools (DDP). They have 2 different capacities that are relevant: virtual capacity, and physical capacity. Virtual capacity is the capacity that is reported by the volume, while physical (repository), capacity is the actual storage capacity of the pool being utilized by the volume. Physical capacity must be defined/increased in 4GB increments. Thin volumes have two different growth options for physical capacity: automatic and manual. Automatically expanding thin volumes will increase in capacity in 4GB increments, as needed. A thin volume configured as manually expanding must be manually expanded using the appropriate storage management software.

With E-series, thin-provisioned volumes and thick-provisioned volumes may be created in the same storage pool, so the thin-provisioning-support and thick-provisioning-support may both be reported to the scheduler for the same storage pool.