Configure Harvest (basic)
The main configuration file,
harvest.yml, consists of the following sections, described below:
All pollers are defined in
harvest.yml, the main configuration file of Harvest, under the section
|Poller name (header)||required||Poller name, user-defined value|
||required||Datacenter name, user-defined value|
||required by some collectors||IPv4 or FQDN of the target system|
||required||List of collectors to run for this poller|
||required||List of exporter names from the
||required by Zapi* collectors||Either
||Absolute paths to SSL (client) certificate and key used to authenticate with the target system.
If not provided, the poller will look for
To create certificates for ONTAP systems, see using certificate authentication
||optional, bool||If true, disable TLS verification when connecting to ONTAP cluster||false|
||optional, string||Path to a yaml file that contains cluster credentials. The file should have the same shape as
||optional, section||Section that defines how Harvest should fetch credentials via external script. See here for details.|
||optional, string||Minimum TLS version to use when connecting to ONTAP cluster: One of tls10, tls11, tls12 or tls13||Platform decides|
||optional, list of key-value pairs||Each of the key-value pairs will be added to a poller's metrics. Details below|
||Maximum size of the log file before it will be rotated||
||Number of rotated log files to keep||
||optional, list of collector names||Matching collectors log their ZAPI request/response|
||optional, bool||Use the ZAPI API if the cluster supports it, otherwise allow Harvest to choose REST or ZAPI, whichever is appropriate to the ONTAP version. See rest-strategy for details.|
This section is optional. If there are parameters identical for all your pollers (e.g. datacenter, authentication method, login preferences), they can be grouped under this section. The poller section will be checked first and if the values aren't found there, the defaults will be consulted.
All exporters need two types of parameters:
exporter parameters- defined in
export_options- these options are defined in the
Matrixdata structure that is emitted from collectors and plugins
The following two parameters are required for all exporters:
|Exporter name (header)||required||Name of the exporter instance, this is a user-defined value|
||required||Name of the exporter class (e.g. Prometheus, InfluxDB, Http) - these can be found under the
Note: when we talk about the Prometheus Exporter or InfluxDB Exporter, we mean the Harvest modules that send the data to a database, NOT the names used to refer to the actual databases.
This section is optional. You can uncomment the
grafana_api_token key and add your Grafana API token so
not prompt you for the key when importing dashboards.
Tools: #grafana_api_token: 'aaa-bbb-ccc-ddd'
Collectors are configured by their own configuration files (templates), which are stored in subdirectories
Most collectors run concurrently and collect a subset of related metrics.
For example, node related metrics are grouped together and run independently of the disk related metrics.
Below is a snippet from
In this example, the
default.yaml template contains a list of objects (e.g. Node) that reference sub-templates (e.g.
node.yaml). This decomposition groups related metrics together and at runtime, a
Zapi collector per object will be
created and each of these collectors will run concurrently.
Using the snippet below, we expect there to be four
Zapi collectors running, each with a different subtemplate and
collector: Zapi objects: Node: node.yaml Aggregate: aggr.yaml Volume: volume.yaml SnapMirror: snapmirror.yaml
At start-up, Harvest looks for two files (
custom.yaml) in the
conf directory of the
default.yaml is installed by default, while the
custom.yaml is an optional file
you can create
to add new templates.
When present, the
custom.yaml file will be merged with the
This behavior can be overridden in your
here for an example.
For a list of collector-specific parameters, refer to their individual documentation.
Zapi and ZapiPerf¶
Rest and RestPerf¶
Labels offer a way to add additional key-value pairs to a poller's metrics. These allow you to tag a cluster's metrics in a cross-cutting fashion. Here's an example:
cluster-03: datacenter: DC-01 addr: 10.0.1.1 labels: - org: meg # add an org label with the value "meg" - ns: rtp # add a namespace label with the value "rtp"
These settings add two key-value pairs to each metric collected from
cluster-03 like this:
Keep in mind that each unique combination of key-value pairs increases the amount of stored data. Use them sparingly. See PrometheusNaming for details.
If you would rather not list cluster credentials in your
harvest.yml, you can use the
harvest.yml to point to a file that contains the credentials.
At runtime, the
credentials_file will be read and the included credentials will be used to authenticate with the
This is handy when integrating with 3rd party credential stores. See #884 for examples.
The format of the
credentials_file is similar to
harvest.yml and can contain multiple cluster credentials.
Pollers: cluster1: addr: 10.193.48.11 credentials_file: secrets/cluster1.yml exporters: - prom1
Pollers: cluster1: username: harvest password: foo
You can fetch authentication information via an external script by using the
credentials_script section in
Pollers section of your
harvest.yml as shown in the example below.
At runtime, Harvest will invoke the script referenced in the
Harvest will call the script with two arguments via
standard in, in this order:
1. address of the cluster taken from your
harvest.yaml file, section
2. username of the cluster taken from your
harvest.yaml file, section
The script should use the two arguments to look up and return the password via the script's
If the script doesn't finish within the specified
timeout, Harvest will kill the script and any spawned processes.
Credential scripts are defined in your
harvest.yml under the
Below are the options for the
|path||string||absolute path to script that takes two arguments: addr and username, in that order|
|schedule||go duration or
||schedule used to call the authentication script. If the value is
|timeout||go duration||amount of time Harvest will wait for the script to finish before killing it and descendents||10s|
Pollers: ontap1: datacenter: rtp addr: 10.1.1.1 collectors: - Rest - RestPerf credentials_script: path: ./get_pass schedule: 3h timeout: 10s
- Make sure your script is executable
- Ensure the user/group that executes your poller also has read and execute permissions on the script.
suas the user/group that runs Harvest and make sure you can execute the script too.