REST

Rest Collector¶

The Rest collectors uses the REST protocol to collect data from ONTAP systems.

The RestPerf collector is an extension of this collector, therefore they share many parameters and configuration settings.

Target System¶

Target system can be cDot ONTAP system. 9.12.1 and after are supported, however the default configuration files may not completely match with all versions. See REST Strategy for more details.

Requirements¶

No SDK or other requirements. It is recommended to create a read-only user for Harvest on the ONTAP system (see prepare monitored clusters for details)

Metrics¶

The collector collects a dynamic set of metrics. ONTAP returns JSON documents and Harvest allows you to define templates to extract values from the JSON document via a dot notation path. You can view ONTAP's full set of REST APIs by visiting https://docs.netapp.com/us-en/ontap-automation/reference/api_reference.html#access-a-copy-of-the-ontap-rest-api-reference-documentation

As an example, the /api/storage/aggregates endpoint, lists all data aggregates in the cluster. Below is an example response from this endpoint:

{
  "records": [
    {
      "uuid": "3e59547d-298a-4967-bd0f-8ae96cead08c",
      "name": "umeng_aff300_aggr2",
      "space": {
        "block_storage": {
          "size": 8117898706944,
          "available": 4889853616128
        }
      },
      "state": "online",
      "volume_count": 36
    }
  ]
}

The Rest collector will take this document, extract the records section and convert the metrics above into: name, space.block_storage.size, space.block_storage.available, state and volume_count. Metric names will be taken, as is, unless you specify a short display name. See counters for more details.

Parameters¶

The parameters of the collector are distributed across three files:

Harvest configuration file (default: harvest.yml)
Rest configuration file (default: conf/rest/default.yaml)
Each object has its own configuration file (located in conf/rest/$version/)

Except for addr and datacenter, all other parameters of the Rest collector can be defined in either of these three files. Parameters defined in the lower-level file, override parameters in the higher-level ones. This allows you to configure each object individually, or use the same parameters for all objects.

The full set of parameters are described below.

Collector configuration file¶

This configuration file contains a list of objects that should be collected and the filenames of their templates ( explained in the next section).

Additionally, this file contains the parameters that are applied as defaults to all objects. As mentioned before, any of these parameters can be defined in the Harvest or object configuration files as well.

parameter	type	description	default
`client_timeout`	duration (Go-syntax)	how long to wait for server responses	30s
`schedule`	list, required	how frequently to retrieve metrics from ONTAP
- `data`	duration (Go-syntax)	how frequently this collector/object should retrieve metrics from ONTAP	3 minutes

The template should define objects in the objects section. Example:

objects:
  Aggregate: aggr.yaml

For each object, we define the filename of the object configuration file. The object configuration files are located in subdirectories matching the ONTAP version that was used to create these files. It is possible to have multiple version-subdirectories for multiple ONTAP versions. At runtime, the collector will select the object configuration file that closest matches the version of the target ONTAP system.

Object configuration file¶

The Object configuration file ("subtemplate") should contain the following parameters:

parameter	type	description
`name`	string, required	display name of the collector that will collect this object
`query`	string, required	REST endpoint used to issue a REST request
`object`	string, required	short name of the object
`counters`	string	list of counters to collect (see notes below)
`plugins`	list	plugins and their parameters to run on the collected data
`export_options`	list	parameters to pass to exporters (see notes below)

Template Example:¶

name:                     Volume
query:                    api/storage/volumes
object:                   volume

counters:
  - ^^name                                        => volume
  - ^^svm.name                                    => svm
  - ^aggregates.#.name                            => aggr
  - ^anti_ransomware.state                        => antiRansomwareState
  - ^state                                        => state
  - ^style                                        => style
  - space.available                               => size_available
  - space.overwrite_reserve                       => overwrite_reserve_total
  - space.overwrite_reserve_used                  => overwrite_reserve_used
  - space.percent_used                            => size_used_percent
  - space.physical_used                           => space_physical_used
  - space.physical_used_percent                   => space_physical_used_percent
  - space.size                                    => size
  - space.used                                    => size_used
  - hidden_fields:
      - anti_ransomware.state
      - space
  - filter:
      - name=*harvest*

plugins:
  - LabelAgent:
      exclude_equals:
        - style `flexgroup_constituent`

export_options:
  instance_keys:
    - aggr
    - style
    - svm
    - volume
  instance_labels:
    - antiRansomwareState
    - state

`counters`¶

This section defines the list of counters that will be collected. These counters can be labels, numeric metrics or histograms. The exact property of each counter is fetched from ONTAP and updated periodically.

The display name of a counter can be changed with => (e.g., space.block_storage.size => space_total).

Counters that are stored as labels will only be exported if they are included in the export_options section.

The counters section allows you to specify hidden_fields and filter parameters. Please find the detailed explanation below.

`hidden_fields`¶

There are some fields that ONTAP will not return unless you explicitly ask for them, even when using the URL parameter fields=**. hidden_fields is how you tell ONTAP which additional fields it should include in the REST response.

`filter`¶

The filter is used to constrain the data returned by the endpoint, allowing for more targeted data retrieval. The filtering uses ONTAP's REST record filtering. The example above asks ONTAP to only return records where a volume's name matches *harvest*.

If you're familiar with ONTAP's REST record filtering, the example above would become name=*harvest* and appended to the final URL like so:

https://CLUSTER_IP/api/storage/volumes?fields=*,anti_ransomware.state,space&name=*harvest*

Refer to the ONTAP API specification, sections: query parameters and record filtering, for more details.

`export_options`¶

Parameters in this section tell the exporters how to handle the collected data. The set of parameters varies by exporter. For Prometheus and InfluxDB exporters, the following parameters can be defined:

instances_keys (list): display names of labels to export with each data-point
instance_labels (list): display names of labels to export as a separate data-point
include_all_labels (bool): export all labels with each data-point (overrides previous two parameters)

RestPerf Collector¶

RestPerf collects performance metrics from ONTAP systems using the REST protocol. The collector is designed to be easily extendable to collect new objects or to collect additional counters from already configured objects.

This collector is an extension of the Rest collector. The major difference between them is that RestPerf collects only the performance (perf) APIs. Additionally, RestPerf always calculates final values from the deltas of two subsequent polls.

Metrics¶

RestPerf metrics are calculated the same as ZapiPerf metrics. More details about how performance metrics are calculated can be found here.

Parameters¶

The parameters of the collector are distributed across three files:

Harvest configuration file (default: harvest.yml)
RestPerf configuration file (default: conf/restperf/default.yaml)
Each object has its own configuration file (located in conf/restperf/$version/)

Except for addr, datacenter and auth_style, all other parameters of the RestPerf collector can be defined in either of these three files. Parameters defined in the lower-level file, override parameters in the higher-level file. This allows the user to configure each objects individually, or use the same parameters for all objects.

The full set of parameters are described below.

RestPerf configuration file¶

This configuration file (the "template") contains a list of objects that should be collected and the filenames of their configuration (explained in the next section).

Additionally, this file contains the parameters that are applied as defaults to all objects. (As mentioned before, any of these parameters can be defined in the Harvest or object configuration files as well).

parameter	type	description	default
`use_insecure_tls`	bool, optional	skip verifying TLS certificate of the target system	false
`client_timeout`	duration (Go-syntax)	how long to wait for server responses	30s
`latency_io_reqd`	int, optional	threshold of IOPs for calculating latency metrics (latencies based on very few IOPs are unreliable)	100
`schedule`	list, required	the poll frequencies of the collector/object, should include exactly these three elements in the exact same other:
- `counter`	duration (Go-syntax)	poll frequency of updating the counter metadata cache	20 minutes
- `instance`	duration (Go-syntax)	poll frequency of updating the instance cache	10 minutes
- `data`	duration (Go-syntax)	poll frequency of updating the data cache Note Harvest allows defining poll intervals on sub-second level (e.g. `1ms`), however keep in mind the following: API response of an ONTAP system can take several seconds, so the collector is likely to enter failed state if the poll interval is less than `client_timeout`. Small poll intervals will create significant workload on the ONTAP system, as many counters are aggregated on-demand. Some metric values become less significant if they are calculated for very short intervals (e.g. latencies)	1 minute

The template should define objects in the objects section. Example:

objects:
  SystemNode: system_node.yaml
  HostAdapter: hostadapter.yaml

Note that for each object we only define the filename of the object configuration file. The object configuration files are located in subdirectories matching to the ONTAP version that was used to create these files. It is possible to have multiple version-subdirectories for multiple ONTAP versions. At runtime, the collector will select the object configuration file that closest matches to the version of the target ONTAP system. (A mismatch is tolerated since RestPerf will fetch and validate counter metadata from the system.)

Object configuration file¶

Refer Object configuration file

`counters`¶

Refer Counters

Some counters require a "base-counter" for post-processing. If the base-counter is missing, RestPerf will still run, but the missing data won't be exported.

`export_options`¶

Refer Export Options

REST

Rest Collector¶

Target System¶

Requirements¶

Metrics¶

Parameters¶

Collector configuration file¶

Object configuration file¶

Template Example:¶

counters¶

hidden_fields¶

filter¶

export_options¶

RestPerf Collector¶

Metrics¶

Parameters¶

RestPerf configuration file¶

Object configuration file¶

counters¶

export_options¶

`counters`¶

`hidden_fields`¶

`filter`¶

`export_options`¶

`counters`¶

`export_options`¶