Containerized Harvest on Linux using Rootless Podman¶

RHEL 8 ships with Podman instead of Docker. There are two ways to run containers with Podman: rootless or with root. Both setups are outlined below. The Podman ecosystem is changing rapidly so the shelf life of these instructions may be short. Make sure you have at least the same versions of the tools listed below.

If you don't want to bother with Podman, you can also install Docker on RHEL 8 and use it to run Harvest per normal.

Setup¶

Make sure your OS is up-to-date with yum update. Podman's dependencies are updated frequently.

sudo yum remove docker-ce
sudo yum module enable -y container-tools:rhel8
sudo yum module install -y container-tools:rhel8
sudo yum install podman podman-docker podman-plugins

We also need to install Docker Compose since Podman uses it for compose workflows. Install docker-compose like this:

VERSION=1.29.2
sudo curl -L "https://github.com/docker/compose/releases/download/$VERSION/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

After all the packages are installed, start the Podman systemd socket-activated service:

sudo systemctl start podman.socket

Containerized Harvest on Linux using Rootful Podman¶

Make sure you're able to curl the endpoint.

sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

If the sudo curl does not print OK⏎ troubleshoot before continuing.

Proceed to Running Harvest

Containerized Harvest on Linux using Rootless Podman¶

To run Podman rootless, we'll create a non-root user named: harvest to run Harvest.

# as root or sudo
usermod --append --groups wheel harvest

Login with the harvest user, set up the podman.socket, and make sure the curl below works. su or sudo aren't sufficient, you need to ssh into the machine as the harvest user or use machinectl login. See sudo-rootless-podman for details.

# these must be run as the harvest user
systemctl --user enable podman.socket
systemctl --user start podman.socket
systemctl --user status podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

If the sudo curl does not print OK⏎ troubleshoot before continuing.

Run podman info and make sure runRoot points to /run/user/$UID/containers (see below). If it doesn't, you'll probably run into problems when restarting the machine. See errors after rebooting.

podman info | grep runRoot
  runRoot: /run/user/1001/containers

Running Harvest¶

By default, Cockpit runs on port 9090, same as Prometheus. We'll change Prometheus's host port to 9091, so we can run both Cockpit and Prometheus. Line 2 below does that.

With these changes, the standard Harvest compose instructions can be followed as normal now. In summary,

Add the clusters, exporters, etc. to your harvest.yml file

Generate a compose file from your harvest.yml by running

docker run --rm \
  --entrypoint "bin/harvest" \
  --volume "$(pwd):/opt/temp" \
  --volume "$(pwd)/harvest.yml:/opt/harvest/harvest.yml" \
  ghcr.io/netapp/harvest \
  generate docker full \
  --output harvest-compose.yml \
  --promPort 9091

Bring everything up

docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans

After starting the containers, you can view them with podman ps -a or using Cockpit https://host-ip:9090/podman.

podman ps -a
CONTAINER ID  IMAGE                                   COMMAND               CREATED        STATUS            PORTS                     NAMES
45fd00307d0a  ghcr.io/netapp/harvest:latest           --poller unix --p...  5 seconds ago  Up 5 seconds ago  0.0.0.0:12990->12990/tcp  poller_unix_v21.11.0
d40585bb903c  localhost/prom/prometheus:latest        --config.file=/et...  5 seconds ago  Up 5 seconds ago  0.0.0.0:9091->9090/tcp    prometheus
17a2784bc282  localhost/grafana/grafana:latest                              4 seconds ago  Up 5 seconds ago  0.0.0.0:3000->3000/tcp    grafana

Troubleshooting¶

Check Podman's troubleshooting docs

Nothing works¶

Make sure the DOCKER_HOST env variable is set and that this curl works.

sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

Make sure your containers can talk to each other.

ping prometheus
PING prometheus (10.88.2.3): 56 data bytes
64 bytes from 10.88.2.3: seq=0 ttl=42 time=0.059 ms
64 bytes from 10.88.2.3: seq=1 ttl=42 time=0.065 ms

Errors after rebooting¶

After restarting the machine, I see errors like these when running podman ps.

podman ps -a
ERRO[0000] error joining network namespace for container 424df6c: error retrieving network namespace at /run/user/1001/netns/cni-5fb97adc-b6ef-17e8-565b-0481b311ba09: failed to Statfs "/run/user/1001/netns/cni-5fb97adc-b6ef-17e8-565b-0481b311ba09": no such file or directory

Run podman info and make sure runRoot points to /run/user/$UID/containers (see below). If it instead points to /tmp/podman-run-$UID you will likely have problems when restarting the machine. Typically, this happens because you used su to become the harvest user or ran podman as root. You can fix this by logging in as the harvest user and running podman system reset.

podman info | grep runRoot
  runRoot: /run/user/1001/containers

Linger errors¶

When you logout, systemd may remove some temp files and tear down Podman's rootless network. Workaround is to run the following as the harvest user. Details here

loginctl enable-linger

Versions¶

The following versions were used to validate this workflow.

podman version

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Thu Jul 29 11:02:43 2021
OS/Arch:      linux/amd64

docker-compose -v
docker-compose version 1.29.2, build 5becea4c

cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

References¶

https://github.com/containers/podman
https://www.redhat.com/sysadmin/sudo-rootless-podman
https://www.redhat.com/sysadmin/podman-docker-compose
https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/
https://podman.io/getting-started/network.html mentions the need for podman-plugins, otherwise rootless containers running in separate containers cannot see each other
Troubleshoot Podman