K8 Deployment¶

The Helm chart is a reference

The Helm chart below is a reference. Use it as is, change it to fit your environment, or deploy Harvest on Kubernetes any other way you prefer.

This page walks through one way to run Harvest on Kubernetes, using an example Helm chart from the Harvest repository. You can use the chart as it is, adjust it for your environment, or use it as a guide and write your own manifests. The chart's values.yaml describes every value you can set.

Each cluster you list runs in its own pod. The pod reads its password from a Kubernetes Secret, so the password is never in your values file.

Prerequisites¶

Kubernetes 1.21 or later and Helm 3.10 or later
Network access from the pods to the management address of each monitored system
A user on each system with the permissions Harvest needs, see Prepare monitored systems
To use the default Prometheus integration, the Prometheus Operator, for example kube-prometheus-stack. No Prometheus yet? See Install Prometheus.

Get the example chart¶

git clone --depth 1 https://github.com/NetApp/harvest.git
cd harvest

Install Prometheus (skip if you already have one)¶

Skip this if you already run the Prometheus Operator. This installs kube-prometheus-stack into the namespace monitoring. It keeps data for one year or until it reaches 45GiB, whichever comes first. The data is stored on a 50Gi persistent volume from the cluster's default StorageClass, so it survives a pod restart. There are no memory limits. Grow storage and retentionSize together to fit your clusters. The volume can be expanded later if the StorageClass allows it.

cat > prometheus.yaml <<'EOF'
prometheus:
  prometheusSpec:
    retention: 1y                 # keep one year of data
    retentionSize: 45GiB          # stop before the volume is full
    resources:
      requests:
        memory: 400Mi             # no limits
    storageSpec:                  # keep data when the pod restarts
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi       # <====== adjust to your clusters
EOF

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace --values prometheus.yaml

The volume comes from the default StorageClass of your cluster. If kubectl get storageclass shows none marked (default), add storageClassName: <name> under spec.

Wait until the pod prometheus-kube-prometheus-stack-prometheus-0 is Running:

kubectl get pods --namespace monitoring

The chart labels its PodMonitors release: kube-prometheus-stack, which is what this Prometheus selects. If your Prometheus was installed with another release name, see PodMonitor.

Quick start¶

This installs Harvest for one ONTAP cluster into the namespace harvest.

1. Create the namespace and a Secret with the password

Replace <ontap-password> with the password of the user Harvest logs in with.

kubectl create namespace harvest

kubectl create secret generic cluster-a-credentials \
  --namespace harvest \
  --from-literal=password='<ontap-password>'

2. Write a values file

Change the lines marked <======: the cluster address and the username. Then paste the whole block. It creates values.yaml.

cat > values.yaml <<'EOF'
harvestConfig:
  Defaults:
    use_insecure_tls: true              # <====== adjust as needed to enable/disable TLS checks
  Pollers:
    cluster-a:
      datacenter: DC-01
      addr: cluster-a.example.com       # <====== your cluster address
      username: harvest                 # <====== your username
      password: ${CLUSTER_A_PASSWORD}   # placeholder, not the password
      collectors:
        - Rest
        - RestPerf
      extraEnvVars:                     # fills the placeholder from the Secret
        - name: CLUSTER_A_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cluster-a-credentials
              key: password
EOF

No Prometheus Operator? The install in the next step fails because the PodMonitor type does not exist. First create no-operator.yaml from "No Prometheus Operator" under Prometheus integration, then add --values no-operator.yaml to the install command.

3. Install

helm install harvest ./deploy/helm/harvest --namespace harvest --values values.yaml

This runs the Harvest release the chart was built for (the chart's appVersion), so every pod runs the same version. Because the release is named harvest, the pod's name starts with harvest-harvest-poller-cluster-a. Use another release name such as ontap to get shorter names.

To run another Harvest release, add its tag. Release tags end in -1, for example 26.08.0-1:

helm install harvest ./deploy/helm/harvest --namespace harvest --values values.yaml \
  --set image.tag=<release-tag>

For the newest release or a nightly build, use a moving tag and pull it every time a pod starts:

helm install harvest ./deploy/helm/harvest --namespace harvest --values values.yaml \
  --set image.tag=latest --set image.pullPolicy=Always

Use --set image.tag=nightly for a nightly build.

4. Verify

kubectl get pods --namespace harvest

Wait until the pod is Running and 1/1. Then look at the metrics:

kubectl port-forward --namespace harvest deploy/harvest-harvest-poller-cluster-a 12990:12990

In another terminal:

curl -s http://localhost:12990/metrics

Metrics appear after the first collection finishes, usually within a minute.

Check that Prometheus has the data. Start a port-forward to Prometheus:

kubectl port-forward --namespace monitoring svc/kube-prometheus-stack-prometheus 9090:9090

In another terminal, ask Prometheus whether the scrape works (1 means up):

curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=up{namespace="harvest"}'

Then ask for Harvest data from your cluster. A number above zero means the data is there:

curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=count({datacenter="DC-01"})'

The queries need a minute or two after the pod starts: the first collection plus one scrape.

You can also open http://localhost:9090, run the same queries on the Query page, or open Status > Targets and look for podMonitor/harvest/... with state UP.

Optional settings¶

None of these are needed for the quick start.

Credentials from files (optional)

Harvest expands ${VAR} in harvest.yml from the environment when it starts (variable expansion). The quick start uses this to keep the password in a Secret. Create the Secret any way you like: kubectl, External Secrets, Vault, or sealed-secrets. The chart only references it by name.

Certificates, a CA bundle and a credentials file are files. Mount them from a Secret with extraVolumes and extraVolumeMounts, then point the poller setting at the mounted path. This example switches cluster-a to certificate authentication. Replace the file names in the first command with your own:

kubectl create secret generic cluster-a-tls --namespace harvest \
  --from-file=tls.crt=cluster-a.crt --from-file=tls.key=cluster-a.key

cat > certs.yaml <<'EOF'
harvestConfig:
  Pollers:
    cluster-a:
      auth_style: certificate_auth
      ssl_cert: /opt/harvest/certs/tls.crt
      ssl_key: /opt/harvest/certs/tls.key
      extraVolumes:
        - name: certs
          secret:
            secretName: cluster-a-tls
      extraVolumeMounts:
        - name: certs
          mountPath: /opt/harvest/certs
          readOnly: true
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values certs.yaml

Helm merges the files in order, so certs.yaml only adds to cluster-a and values.yaml stays as it is.

Resources (optional)

Each poller requests 100m CPU and 256Mi memory by default and has no limits, so a large cluster is never killed for using more memory. Adjust the requests to your clusters and add limits for production. They apply to every poller:

cat > resources.yaml <<'EOF'
poller:
  resources:
    requests: {cpu: 200m, memory: 512Mi}
    limits: {memory: 1Gi}
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values resources.yaml

Collectors and templates (optional)

Choose collectors per poller with collectors. To change a collector's schedule or enable more objects, add collectorsExtensions to the poller, see extend an existing object:

cat > collectors.yaml <<'EOF'
harvestConfig:
  Pollers:
    cluster-a:
      collectorsExtensions:
        Rest:
          jitter: 1m
          schedule:
            - data: 5m
        RestPerf:
          objects:
            Qtree: qtree.yaml
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values collectors.yaml

Private registry (optional)

To use a mirror or private registry, name a docker-registry Secret that exists in the harvest namespace:

cat > registry.yaml <<'EOF'
image:
  registry: registry.example.com
  repository: team/harvest
  pullSecrets: [my-regcred]
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values registry.yaml

Prometheus integration¶

The default needs nothing more than the quick start. The sections below are alternatives, use at most one of them.

The chart creates one PodMonitor for each cluster, so ten clusters give ten PodMonitors. A PodMonitor is a small Prometheus Operator object that tells Prometheus which pod to scrape. It does not run anything itself. If you do not run the Prometheus Operator, turn the PodMonitors off and scrape the pods yourself.

The commands in each section below start from the quick start values.yaml.

PodMonitor (default)¶

Nothing to add, this is what the quick start installed. Each PodMonitor scrapes its poller's container port http on /metrics every minute. Change the interval, labels and more under monitoring.scrape.podMonitor, or per poller with the podMonitor key. Set podMonitor.enabled: false on a poller to skip its PodMonitor.

The PodMonitor sets the Prometheus instance label to the poller name, for example cluster-a. This keeps instance the same when a pod restarts and gets a new IP address, so your graphs and queries keep working. To change it, add your own entry under relabelings, which runs after the chart's.

Your Prometheus must select the PodMonitors. The chart labels them release: kube-prometheus-stack. Find the label your Prometheus expects:

kubectl get prometheus --all-namespaces -o yaml | grep -A3 podMonitorSelector

If it shows another value, set the label to match:

cat > prometheus-label.yaml <<'EOF'
monitoring:
  scrape:
    podMonitor:
      labels:
        release: <your-prometheus-release>
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values prometheus-label.yaml

Or tell kube-prometheus-stack to select every PodMonitor:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --reuse-values \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Admin service discovery (optional)

Instead of PodMonitors, you can let pollers register with a small admin pod that Prometheus reads using HTTP service discovery. New and removed pollers are picked up without touching Prometheus. Enable one or the other, not both.

cat > admin-sd.yaml <<'EOF'
monitoring:
  scrape:
    adminSD:
      enabled: true
    podMonitor:
      enabled: false
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values admin-sd.yaml
kubectl get pods,svc --namespace harvest

The chart creates the admin pod, a headless Service and a ScrapeConfig. It also adds the settings pollers need to find the admin pod to the generated harvest.yml, see heart_beat_url. The ScrapeConfig is labeled release: kube-prometheus-stack, change it with monitoring.scrape.adminSD.labels in the same way as above. It also sets the instance label to the poller name, the same as the PodMonitor does.

Check which pollers registered. Prometheus refreshes the list about once a minute, so allow a few minutes after an install or restart:

kubectl port-forward --namespace harvest svc/harvest-harvest-admin-sd 8887:8887

In another terminal:

curl -s http://localhost:8887/api/v1/sd

To go back to PodMonitors, upgrade without admin-sd.yaml.

Basic authentication for admin service discovery (optional)

To protect the list with basic authentication, set Admin.httpsd.auth_basic and give the password to the admin pod, every poller, and Prometheus. The Secret must be in the harvest namespace:

kubectl create secret generic sd-credentials --namespace harvest \
  --from-literal=username=sd --from-literal=password='<sd-password>'

cat > admin-auth.yaml <<'EOF'
harvestConfig:
  Admin:
    httpsd:
      listen: :8887
      auth_basic:
        username: sd
        password: ${SD_PASSWORD}
poller:
  extraEnvVars:               # every poller pod
    - name: SD_PASSWORD
      valueFrom:
        secretKeyRef: {name: sd-credentials, key: password}
admin:
  extraEnvVars:               # the admin pod
    - name: SD_PASSWORD
      valueFrom:
        secretKeyRef: {name: sd-credentials, key: password}
monitoring:
  scrape:
    adminSD:
      httpSDConfig:           # Prometheus
        basicAuth:
          username: {name: sd-credentials, key: username}
          password: {name: sd-credentials, key: password}
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest \
  --values values.yaml --values admin-sd.yaml --values admin-auth.yaml

Check the list with the same username and password. Start a port-forward to the admin Service:

kubectl port-forward --namespace harvest svc/harvest-harvest-admin-sd 8887:8887

In another terminal:

curl -s -u sd:<sd-password> http://localhost:8887/api/v1/sd

Without -u the admin pod returns 401.

A poller without the password sends its heartbeat with empty credentials, so the admin pod rejects it and the poller is missing from the list.

TLS for admin service discovery (optional)

To serve the list over https, the admin pod needs a certificate and key. Every poller needs the certificate too: it trusts the admin pod with it, and a poller exits at startup if the file is missing. Prometheus needs the certificate as its CA. The certificate must name the admin Service, <release>-harvest-admin-sd.<namespace>.svc, in its subject alternative names. With the release harvest in the namespace harvest that is harvest-harvest-admin-sd.harvest.svc. Harvest needs a TLS 1.3 certificate. This creates a self-signed one and stores it in a Secret:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 365 \
  -keyout tls.key -out tls.crt -subj "/CN=harvest-harvest-admin-sd" \
  -addext "subjectAltName=DNS:harvest-harvest-admin-sd.harvest.svc"

kubectl create secret generic sd-tls --namespace harvest \
  --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key

cat > admin-tls.yaml <<'EOF'
harvestConfig:
  Admin:
    httpsd:
      tls:
        cert_file: /opt/harvest/sd-tls/tls.crt
        key_file: /opt/harvest/sd-tls/tls.key
admin:
  extraVolumes:               # the admin pod gets the certificate and the key
    - name: sd-tls
      secret:
        secretName: sd-tls
  extraVolumeMounts:
    - name: sd-tls
      mountPath: /opt/harvest/sd-tls
      readOnly: true
poller:
  extraVolumes:               # every poller gets the certificate only
    - name: sd-tls
      secret:
        secretName: sd-tls
        items:
          - key: tls.crt
            path: tls.crt
  extraVolumeMounts:
    - name: sd-tls
      mountPath: /opt/harvest/sd-tls
      readOnly: true
monitoring:
  scrape:
    adminSD:
      httpSDConfig:           # Prometheus
        tlsConfig:
          ca:
            secret: {name: sd-tls, key: tls.crt}
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest \
  --values values.yaml --values admin-sd.yaml --values admin-tls.yaml

The chart switches the heartbeat URL and the ScrapeConfig URL to https when cert_file is set. Add admin-auth.yaml to the command if you also use basic authentication. Check the list. Start a port-forward to the admin Service:

kubectl port-forward --namespace harvest svc/harvest-harvest-admin-sd 8887:8887

In another terminal:

curl -s --cacert tls.crt --resolve harvest-harvest-admin-sd.harvest.svc:8887:127.0.0.1 \
  https://harvest-harvest-admin-sd.harvest.svc:8887/api/v1/sd

No Prometheus Operator (optional)

Turn both off. The chart then creates only the pollers, and you point Prometheus at the pods yourself.

cat > no-operator.yaml <<'EOF'
monitoring:
  scrape:
    podMonitor:
      enabled: false
    adminSD:
      enabled: false
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values no-operator.yaml

Every pod serves metrics on its container port named http (default 12990) at /metrics. In plain Prometheus, discover them with kubernetes_sd_configs of role pod and keep the ones labeled app.kubernetes.io/name: harvest.

Dashboards¶

The chart does not install Grafana dashboards. Import the dashboards from the grafana/dashboards directory of the Harvest release, see Import Grafana dashboards.

Managing clusters¶

Everything is declarative: describe the clusters you want, then run helm upgrade. Helm creates, updates or deletes the pollers to match.

Note

Helm does not remember which --values files you used. Pass the same ones on every helm upgrade. Without them the release goes back to the chart defaults and your pollers are removed. See what is applied with helm get values harvest --namespace harvest.

Add a cluster¶

Create its Secret, describe the cluster in a new file, and upgrade. Replace the password and the two marked values:

kubectl create secret generic cluster-b-credentials --namespace harvest \
  --from-literal=password='<ontap-password>'

cat > cluster-b.yaml <<'EOF'
harvestConfig:
  Pollers:
    cluster-b:
      datacenter: DC-02
      addr: cluster-b.example.com       # <====== your cluster address
      username: harvest                 # <====== your username
      password: ${CLUSTER_B_PASSWORD}
      collectors:
        - Rest
        - RestPerf
      extraEnvVars:
        - name: CLUSTER_B_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cluster-b-credentials
              key: password
EOF

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml --values cluster-b.yaml
kubectl get pods --namespace harvest

The pod of cluster-b is created, and the pod of cluster-a is not restarted. E-Series, StorageGRID and other systems are added the same way. Set collectors as described in the Harvest configuration, for example [Eseries, EseriesPerf] for E-Series.

Change a cluster¶

Edit the cluster's entry, for example change datacenter in values.yaml, then upgrade:

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml

Add a --values flag for every other file you use, such as --values cluster-b.yaml.

Only the pod of the cluster you changed restarts. Changing a shared section (Exporters, Defaults, Admin) restarts all pollers.

Remove a cluster¶

Leave the cluster's file out of the next upgrade. The pod and PodMonitor are deleted. The Secret is not managed by the chart, so delete it yourself:

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml

kubectl delete secret cluster-b-credentials --namespace harvest

If the cluster is in values.yaml, delete its entry there instead.

Poller names and ports¶

A poller name may contain upper case letters and underscores (AFF250, Dc_1). Kubernetes object names use a lower case form (dc-1), --poller and harvest.yml keep the name you wrote.
Two names that become the same Kubernetes name (Dc_1 and dc-1), or a name that makes the object name longer than 63 characters, fail helm install with a message that names the poller.
prom_port is optional. Each poller is in its own pod, so they can all use poller.defaultPromPort (12990).

Upgrades¶

Run helm upgrade with the same --values files whenever you change your values or want a newer chart:

git pull
helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml
helm history harvest --namespace harvest

Each poller uses the Recreate strategy, so the old pod stops before the new one starts and a poller is never running twice, which would duplicate metrics. Expect a short gap in metrics when a poller restarts.

Update the Harvest image¶

By default the chart runs the Harvest release in its appVersion, and the pods stay on that version until you upgrade. Running git pull and then helm upgrade moves them to the version of the newer chart. To run another release, upgrade with its tag. Release tags end in -1, for example 26.08.0-1:

helm upgrade harvest ./deploy/helm/harvest --namespace harvest --values values.yaml \
  --set image.tag=<release-tag>

To see the image the pods run:

kubectl get pods --namespace harvest -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,IMAGEID:.status.containerStatuses[0].imageID'

If you use a moving tag (latest or nightly), add --set image.pullPolicy=Always as well. The pods then pull the tag every time they start, so they can run different versions. To move all of them to the newest image, restart the pods:

kubectl rollout restart deployment --namespace harvest -l app.kubernetes.io/instance=harvest

To go back to the chart's version, run helm upgrade without the --set image.* flags.

Roll back¶

To roll back to the previous revision, or to one from helm history:

helm rollback harvest --namespace harvest
helm rollback harvest 1 --namespace harvest

A rollback restores your values, including the image tag. The exception is a moving tag such as latest or nightly: a rollback keeps the tag but not the image behind it. To run an older version, run helm upgrade with --set image.tag=<version> as shown above.

Troubleshooting¶

Start with the pods and their logs. Harvest logs to the container's standard error.

kubectl get pods --namespace harvest
kubectl describe pod --namespace harvest -l app.kubernetes.io/name=harvest
kubectl logs --namespace harvest deploy/harvest-harvest-poller-cluster-a

The image is a distroless debug image, so there is a shell if you need one:

kubectl exec -it --namespace harvest deploy/harvest-harvest-poller-cluster-a -- sh

Symptom	Cause and fix
`helm install` fails with `invalid Kubernetes name`	A poller name makes the object name longer than 63 characters. Shorten the name or set `fullnameOverride`
`helm install` fails with `both map to the Kubernetes name`	Two poller names only differ by case or punctuation, rename one
`helm install` fails with `invalid prom_port`	`prom_port` is not a number from 1 to 65535
Pod `CrashLoopBackOff`, log shows an authentication error	Wrong user or password, or the Secret or key does not exist. Check `kubectl describe pod` for `secret ... not found`
Pod running, log shows connection timeouts to the system	The pod cannot reach the management address. Test from the namespace with `kubectl run -it --rm curl --image=curlimages/curl -- curl -vk https://<addr>`; check NetworkPolicies and egress proxies
Pod `ImagePullBackOff` or `ErrImagePull`	The tag does not exist (release tags end in `-1`, for example `26.08.0-1`), or the registry needs `image.pullSecrets`. Check `kubectl describe pod`
Pod is `Running` but not `Ready`	The metrics port is not open yet. The startup probe allows about a minute; look at the poller log for errors
TLS errors such as `x509: certificate signed by unknown authority`	Mount your CA bundle and set `ca_cert`, or set `use_insecure_tls: true` under `harvestConfig.Defaults` as in the quick start
Poller is `UP` in Kubernetes but there is no Prometheus target	Prometheus does not select the PodMonitor. Compare `kubectl get podmonitor -n harvest --show-labels` with the Prometheus `podMonitorSelector`
`up` is `1` but `count({datacenter="DC-01"})` returns nothing	The poller is scraped but has not collected data yet, or cannot reach the system. Wait a minute, then look at the poller log for errors
Admin service discovery: `/api/v1/sd` returns `[]`	`heart_beat_url` is missing or wrong, or the poller cannot reach the admin Service. Look for `Failed connecting to admin node` in the poller log

Uninstall¶

helm uninstall harvest --namespace harvest
kubectl delete namespace harvest

This deletes the pods, ConfigMaps, PodMonitors and the admin resources. Deleting the namespace also deletes the Secrets you created in it.

If you installed Prometheus above and no longer need it:

helm uninstall kube-prometheus-stack --namespace monitoring
kubectl delete namespace monitoring