Monitor Harvest
Harvest Metadata¶
Harvest publishes metadata metrics about the key components of Harvest.
Many of these metrics are used in the Harvest Metadata dashboard.
If you want to understand more about these metrics, read on!
Metrics are published for:
- collectors
- pollers
- clusters being monitored
- exporters
Here's a high-level summary of the metadata metrics Harvest publishes with details below.
| Metric | Description | Units |
|---|---|---|
| metadata_collector_api_time | amount of time to collect data from monitored cluster object | microseconds |
| metadata_collector_instances | number of objects collected from monitored cluster | scalar |
| metadata_collector_metrics | number of counters collected from monitored cluster | scalar |
| metadata_collector_parse_time | amount of time to parse XML, JSON, etc. for cluster object | microseconds |
| metadata_collector_plugin_time | amount of time for all plugins to post-process metrics | microseconds |
| metadata_collector_poll_time | amount of time it took for the poll to finish | microseconds |
| metadata_collector_task_time | amount of time it took for each collector's subtasks to complete | microseconds |
| metadata_component_count | number of metrics collected for each object | scalar |
| metadata_component_status | status of the collector - 0 means running, 1 means standby, 2 means failed | enum |
| metadata_exporter_count | number of metrics and labels exported | scalar |
| metadata_exporter_time | amount of time it took to render, export, and serve exported data | microseconds |
| metadata_target_goroutines | number of goroutines that exist within the poller | scalar |
| metadata_target_status | status of the system being monitored. 0 means reachable, 1 means unreachable | enum |
| metadata_collector_calc_time | amount of time it took to compute metrics between two successive polls, specifically using properties like raw, delta, rate, average, and percent. This metric is available for ZapiPerf/RestPerf collectors. | microseconds |
| metadata_collector_skips | number of metrics that were not calculated between two successive polls. This metric is available for ZapiPerf/RestPerf collectors. | scalar |
Collector Metadata¶
A poller publishes the metadata metrics for each collector and exporter associated with it.
Let's say we start a poller with the Zapi collector and the out-of-the-box default.yaml exporting metrics to
Prometheus. That means you will be monitoring 22 different objects (uncommented lines in default.yaml as of 23.02).
When we start this poller, we expect it to export 23 metadata_component_status metrics.
One for each of the 22 objects, plus one for the Prometheus exporter.
The following curl confirms there are 23 metadata_component_status metrics reported.
curl -s http://localhost:12990/metrics | grep -v "#" | grep metadata_component_status | wc -l
23
These metrics also tell us which collectors are in an standby or failed state.
For example, filtering on components not in the running state shows the following since this cluster doesn't have any ClusterPeers, SecurityAuditDestinations, or SnapMirrors. The reason is listed as no instances and the metric value is 1 which means standby.
curl -s http://localhost:12990/metrics | grep -v "#" | grep metadata_component_status | grep -Evo "running"
metadata_component_status{name="Zapi", reason="no instances",target="ClusterPeer",type="collector",version="23.04.1417"} 1
metadata_component_status{name="Zapi", reason="no instances",target="SecurityAuditDestination",type="collector",version="23.04.1417"} 1
metadata_component_status{name="Zapi", reason="no instances",target="SnapMirror",type="collector",version="23.04.1417"} 1
The log files for the poller show a similar story. The poller starts with 22 collectors, but drops to 19 after three of the collectors go to standby because there are no instances to collect.
2023-04-17T13:14:18-04:00 INF ./poller.go:539 > updated status, up collectors: 22 (of 22), up exporters: 1 (of 1) Poller=u2
2023-04-17T13:14:18-04:00 INF collector/collector.go:342 > no instances, entering standby Poller=u2 collector=Zapi:SecurityAuditDestination task=data
2023-04-17T13:14:18-04:00 INF collector/collector.go:342 > no instances, entering standby Poller=u2 collector=Zapi:ClusterPeer task=data
2023-04-17T13:14:18-04:00 INF collector/collector.go:342 > no instances, entering standby Poller=u2 collector=Zapi:SnapMirror task=data
2023-04-17T13:15:18-04:00 INF ./poller.go:539 > updated status, up collectors: 19 (of 22), up exporters: 1 (of 1) Poller=u2