Cirrus: Support special-case modes of testing

Previously libpod CI was fairly straight-forward, run unit and
integration tests in a standard set of 3 VMs.  Off on the side was a
single special case of running tests as an ordinary user.  There is a
desire to stop using the PAPR system to support testing inside of a
container.

Since having two special cases potentially invites more
down the road, make provisions to handle them more gracefully.  This
commit introduces an environment variable:  ``$SPECIALMODE``.  It's
value has the following meanings within the CI scripts:

Mode 'none': Nothing special, business as usual (default)
Mode 'rootless': Rootless testing
Mode 'in_podman': Build container, run integration tests in it.

This will make adding additional special-cases later easier, as well as
extending the special cases in a Matrix across multiple OS's.

Signed-off-by: Chris Evich <cevich@redhat.com>

8 files changed, 253 insertions, 81 deletions

diff --git a/contrib/cirrus/README.md b/contrib/cirrus/README.md
index 0dabf5df6..ea358d2d7 100644
--- a/contrib/cirrus/README.md
+++ b/contrib/cirrus/README.md
@@ -63,26 +63,26 @@ task (pass or fail) is set based on the exit status of the last script to execut
    Total execution time is capped at 2-hours (includes all the above)
    but this script normally completes in less than an hour.
 
-### ``rootless_testing`` Task
+### ``special_testing`` Task
+
+This task exercises podman under specialized environments or conditions.
+The specific differences from the ``testing`` task depend upon the
+contents of the ``$SPECIALMODE`` environment variable.
+
+| Value     | Meaning                                                               |
+| rootless  | Setup a regular user to build/run integration tests.                  |
+| in_podman | Setup a container image, build/run integration tests inside container |
 
 ***N/B: Steps below are performed by automation***
 
 1. After `gating` passes, spin up one VM per
-   `matrix: image_name` item. Once accessible, ``ssh``
-   into each VM as the `root` user.
+   `matrix: image_name` item.
+
+2. ``setup_environment.sh``: Mostly the same as
+   in ``testing`` task, then specialized depending on ``$SPECIALMODE``.
+
+3. Which tests and how they execute depends on ``$SPECIALMODE``.
 
-2. ``setup_environment.sh``: Configure root's `.bash_profile`
-   the same as for other tasks.  However, also add a regular
-   user account, chown all the source code to them.  Set up
-   fresh ssh pub/priv. keys for the root user, adding the
-   public part to the user's `authorized_keys` file.
-
-3. As root, call ssh to connect to localhost as the user,
-   and run the ``rootless_test.sh`` script from the source
-   tree.  This is needed so the user has a clean process tree
-   and environment - i.e. without `sudo`, `su`, `runuser`,
-   etc. in the mix.  From here, all testing as the user may
-   be performed.
 
 ### ``optional_testing`` Task
 
diff --git a/contrib/cirrus/container_test.sh b/contrib/cirrus/container_test.sh
new file mode 100644
index 000000000..e6c1a3a47
--- /dev/null
+++ b/contrib/cirrus/container_test.sh
@@ -0,0 +1,131 @@
+#!/bin/bash
+set -xeuo pipefail
+
+export GOPATH=/var/tmp/go
+export PATH=$HOME/gopath/bin:$PATH:$GOPATH/bin
+export GOSRC=$GOPATH/src/github.com/containers/libpod
+
+DIST=${DIST:=""}
+CONTAINER_RUNTIME=${DIST:=""}
+
+source /etc/os-release
+
+INTEGRATION_TEST_ENVS=""
+
+# For all distributions not Fedora, we need to skip USERNS tests
+# for now.
+if [ "${ID}" != "fedora" ] || [ "${CONTAINER_RUNTIME}" != "" ]; then
+    INTEGRATION_TEST_ENVS="SKIP_USERNS=1"
+fi
+
+pwd
+
+# -i install
+# -b build
+# -t integration test
+# -u unit test
+# -v validate
+
+build=0
+install=0
+integrationtest=0
+unittest=0
+validate=0
+options=0
+install_tools_made=0
+
+while getopts "biptuv" opt; do
+    case "$opt" in
+    b) build=1
+       options=1
+       ;;
+    i) install=1
+       options=1
+       ;;
+    t) integrationtest=1
+       options=1
+       ;;
+    u) unittest=1
+       options=1
+       ;;
+    v) validate=1
+       options=1
+       ;;
+    esac
+done
+
+# If no options are passed, do everything
+if [ $options -eq 0 ]; then
+    build=1
+    install=1
+    integrationtest=1
+    unittest=1
+    validate=1
+fi
+
+# Make Install tools function used by multiple sections below
+make_install_tools () {
+    # Only make the install tools once
+    if [ $install_tools_made -eq 0 ]; then
+        make install.tools TAGS="${TAGS}"
+    fi
+    install_tools_made=1
+}
+
+CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-none}
+
+if [ "${CONTAINER_RUNTIME}" == "none" ]; then
+    mkdir -p /$GOPATH/src/github.com/containers/
+    mv /var/tmp/checkout $GOSRC
+    cd $GOSRC
+    pwd
+fi
+
+
+export TAGS="seccomp $($GOSRC/hack/btrfs_tag.sh) $($GOSRC/hack/libdm_tag.sh) $($GOSRC/hack/btrfs_installed_tag.sh) $($GOSRC/hack/ostree_tag.sh) $($GOSRC/hack/selinux_tag.sh)"
+
+# Validate
+if [ $validate -eq 1 ]; then
+    make_install_tools
+    # PAPR adds a merge commit, for testing, which fails the
+    # short-commit-subject validation test, so tell git-validate.sh to only check
+    # up to, but not including, the merge commit.
+    export GITVALIDATE_TIP=$(cd $GOSRC; git log -2 --pretty='%H' | tail -n 1)
+    make gofmt TAGS="${TAGS}"
+
+    # Only check lint and gitvalidation on more recent
+    # distros with updated git and tooling
+    if [[ ${DIST} == "Fedora" ]]; then
+        HEAD=$GITVALIDATE_TIP make -C $GOSRC .gitvalidation TAGS="${TAGS}"
+        make lint
+    fi
+fi
+
+# Unit tests
+if [ $unittest -eq 1 ]; then
+    make localunit TAGS="${TAGS}"
+fi
+
+# Make Podman
+if [ $build -eq 1 ]; then
+    make_install_tools
+    make TAGS="${TAGS}" GOPATH=$GOPATH
+    make podman-remote TAGS="${TAGS}" GOPATH=$GOPATH
+fi
+
+# Install Podman
+if [ $install -eq 1 ]; then
+    make_install_tools
+    make TAGS="${TAGS}" install.bin PREFIX=/usr ETCDIR=/etc
+    make TAGS="${TAGS}" install.man PREFIX=/usr ETCDIR=/etc
+    make TAGS="${TAGS}" install.cni PREFIX=/usr ETCDIR=/etc
+    make TAGS="${TAGS}" install.systemd PREFIX=/usr ETCDIR=/etc
+fi
+
+# Run integration tests
+if [ $integrationtest -eq 1 ]; then
+    make TAGS="${TAGS}" test-binaries
+    make varlink_generate
+    make ginkgo $INTEGRATION_TEST_ENVS
+    make ginkgo-remote $INTEGRATION_TEST_ENVS
+fi
diff --git a/contrib/cirrus/integration_test.sh b/contrib/cirrus/integration_test.sh
index 58c8af289..8a2507f38 100755
--- a/contrib/cirrus/integration_test.sh
+++ b/contrib/cirrus/integration_test.sh
@@ -5,33 +5,64 @@ source $(dirname $0)/lib.sh
 
 req_env_var "
 GOSRC $GOSRC
+SCRIPT_BASE $SCRIPT_BASE
 OS_RELEASE_ID $OS_RELEASE_ID
 OS_RELEASE_VER $OS_RELEASE_VER
+CONTAINER_RUNTIME $CONTAINER_RUNTIME
 "
 
-record_timestamp "integration test start"
+exit_handler() {
+    set +ex
+    record_timestamp "integration test end"
+}
+trap exit_handler EXIT
 
-clean_env
+record_timestamp "integration test start"
 
-set -x
 cd "$GOSRC"
-case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in
-    ubuntu-18)
-        make install PREFIX=/usr ETCDIR=/etc
-        make test-binaries
-        SKIP_USERNS=1 make localintegration
-        ;;
-    fedora-29) ;&  # Continue to the next item
-    fedora-28) ;&
-    centos-7) ;&
-    rhel-7)
-        make install PREFIX=/usr ETCDIR=/etc
-        make podman-remote
-        install bin/podman-remote /usr/bin
-        make test-binaries
-        make localintegration
-        ;;
-    *) bad_os_id_ver ;;
-esac
-
-record_timestamp "integration test end"
+
+if [[ "$SPECIALMODE" == "in_podman" ]]
+then
+    set -x
+    ${CONTAINER_RUNTIME} run --rm --privileged --net=host \
+        -v $GOSRC:$GOSRC:Z \
+        --workdir $GOSRC \
+        -e "CGROUP_MANAGER=cgroupfs" \
+        -e "STORAGE_OPTIONS=--storage-driver=vfs" \
+        -e "CRIO_ROOT=$GOSRC" \
+        -e "PODMAN_BINARY=/usr/bin/podman" \
+        -e "CONMON_BINARY=/usr/libexec/podman/conmon" \
+        -e "DIST=$OS_RELEASE_ID" \
+        -e "CONTAINER_RUNTIME=$CONTAINER_RUNTIME" \
+        ${OS_RELEASE_ID}podmanbuild bash $GOSRC/$SCRIPT_BASE/container_test.sh -b -i -t
+
+    exit $?
+elif [[ "$SPECIALMODE" == "rootless" ]]
+then
+    req_env_var "ROOTLESS_USER $ROOTLESS_USER"
+    set -x
+    ssh $ROOTLESS_USER@localhost \
+                -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no \
+                $GOSRC/$SCRIPT_BASE/rootless_test.sh
+    exit $?
+else
+    set -x
+    make
+    make install PREFIX=/usr ETCDIR=/etc
+    make test-binaries
+    clean_env
+
+    case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in
+        ubuntu-18) ;;
+        fedora-29) ;&  # Continue to the next item
+        fedora-28) ;&
+        centos-7) ;&
+        rhel-7)
+            make podman-remote
+            install bin/podman-remote /usr/bin
+            ;;
+        *) bad_os_id_ver ;;
+    esac
+    make localintegration
+    exit $?
+fi
diff --git a/contrib/cirrus/lib.sh b/contrib/cirrus/lib.sh
index e941610e2..6c45b2c5d 100644
--- a/contrib/cirrus/lib.sh
+++ b/contrib/cirrus/lib.sh
@@ -18,6 +18,8 @@ CIRRUS_BUILD_ID=${CIRRUS_BUILD_ID:-DEADBEEF}  # a human
 CIRRUS_BASE_SHA=${CIRRUS_BASE_SHA:-HEAD}
 CIRRUS_CHANGE_IN_REPO=${CIRRUS_CHANGE_IN_REPO:-FETCH_HEAD}
 TIMESTAMPS_FILEPATH="${TIMESTAMPS_FILEPATH:-/var/tmp/timestamps}"
+SPECIALMODE="${SPECIALMODE:-none}"
+export CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-podman}
 
 if ! [[ "$PATH" =~ "/usr/local/bin" ]]
 then
@@ -81,6 +83,7 @@ CIRRUS_USER_COLLABORATOR $CIRRUS_USER_COLLABORATOR
 CIRRUS_USER_PERMISSION $CIRRUS_USER_PERMISSION
 CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR
 CIRRUS_HTTP_CACHE_HOST $CIRRUS_HTTP_CACHE_HOST
+SPECIALMODE $SPECIALMODE
 $(go env)
 PACKER_BUILDS $PACKER_BUILDS
     " | while read NAME VALUE
@@ -127,15 +130,6 @@ bad_os_id_ver() {
     exit 42
 }
 
-run_rootless() {
-    if [[ -z "$ROOTLESS_USER" ]]
-    then
-        return 1
-    else
-        return 0
-    fi
-}
-
 stub() {
     echo "STUB: Pretending to do $1"
 }
@@ -179,6 +173,13 @@ setup_rootless() {
         return 0
     fi
 
+    # Only do this once
+    cd $GOSRC
+    make install.catatonit
+    go get github.com/onsi/ginkgo/ginkgo
+    go get github.com/onsi/gomega/...
+    dnf -y update runc
+
     # Guarantee independence from specific values
     ROOTLESS_UID=$[RANDOM+1000]
     ROOTLESS_GID=$[RANDOM+1000]
diff --git a/contrib/cirrus/rootless_test.sh b/contrib/cirrus/rootless_test.sh
index d0e2ceb95..88b38f45b 100755
--- a/contrib/cirrus/rootless_test.sh
+++ b/contrib/cirrus/rootless_test.sh
@@ -12,9 +12,9 @@ OS_RELEASE_ID $OS_RELEASE_ID
 OS_RELEASE_VER $OS_RELEASE_VER
 "
 
-if ! run_rootless
+if [[ "$UID" == "0" ]]
 then
-    echo "Error: Expected rootless env. vars not set or empty"
+    echo "Error: Expected to be running as a regular user"
     exit 1
 fi
 
@@ -24,16 +24,9 @@ echo "Hello, my name is $USER and I live in $PWD can I be your friend?"
 record_timestamp "rootless test start"
 
 cd "$GOSRC"
-case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in
-    ubuntu-18) ;&  # Continue to the next item
-    fedora-29) ;&
-    fedora-28)
-        make
-        make varlink_generate
-        make test-binaries
-        make ginkgo
-        ;;
-    *) bad_os_id_ver ;;
-esac
+make
+make varlink_generate
+make test-binaries
+make ginkgo
 
 record_timestamp "rootless test end"
diff --git a/contrib/cirrus/setup_container_environment.sh b/contrib/cirrus/setup_container_environment.sh
new file mode 100755
index 000000000..23df4fe8b
--- /dev/null
+++ b/contrib/cirrus/setup_container_environment.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+set -e
+
+source $(dirname $0)/lib.sh
+
+req_env_var "
+GOSRC $GOSRC
+OS_RELEASE_ID $OS_RELEASE_ID
+CONTAINER_RUNTIME $CONTAINER_RUNTIME
+"
+
+DIST=$OS_RELEASE_ID
+IMAGE=${DIST}podmanbuild
+
+# Since CRIU 3.11 has been pushed to Fedora 28 the checkpoint/restore
+# test cases are actually run. As CRIU uses iptables to lock and unlock
+# the network during checkpoint and restore it needs the following two
+# modules loaded.
+modprobe ip6table_nat || :
+modprobe iptable_nat || :
+
+# Build the test image
+${CONTAINER_RUNTIME} build -t ${IMAGE} -f Dockerfile.${DIST} .
diff --git a/contrib/cirrus/setup_environment.sh b/contrib/cirrus/setup_environment.sh
index 96d0e1b55..55706954e 100755
--- a/contrib/cirrus/setup_environment.sh
+++ b/contrib/cirrus/setup_environment.sh
@@ -43,7 +43,6 @@ then
         "export OS_RELEASE_ID=\"$(os_release_id)\"" \
         "export OS_RELEASE_VER=\"$(os_release_ver)\"" \
         "export OS_REL_VER=\"$(os_release_id)-$(os_release_ver)\"" \
-        "export ROOTLESS_USER=$ROOTLESS_USER" \
         "export BUILT_IMAGE_SUFFIX=\"-$CIRRUS_REPO_NAME-${CIRRUS_CHANGE_IN_REPO:0:8}\"" \
         "export GOPATH=\"/var/tmp/go\"" \
         'export PATH="$HOME/bin:$GOPATH/bin:/usr/local/bin:$PATH"' \
@@ -75,14 +74,17 @@ then
     # Reload to incorporate any changes from above
     source "$SCRIPT_BASE/lib.sh"
 
-    if run_rootless
-    then
-        setup_rootless
-        make install.catatonit
-        go get github.com/onsi/ginkgo/ginkgo
-        go get github.com/onsi/gomega/...
-        dnf -y update runc
-    fi
+    case "$SPECIALMODE" in
+        rootless)
+            X=$(echo "export ROOTLESS_USER='some${RANDOM}dude'" | \
+                tee -a "$HOME/$ENVLIB") && eval "$X" && echo "$X"
+            setup_rootless
+            ;;
+        in_podman)  # Assumed to be Fedora
+            dnf install -y podman buildah
+            $SCRIPT_BASE/setup_container_environment.sh
+            ;;
+    esac
 fi
 
 show_env_vars
diff --git a/contrib/cirrus/unit_test.sh b/contrib/cirrus/unit_test.sh
index fd9e82509..4ace19d10 100755
--- a/contrib/cirrus/unit_test.sh
+++ b/contrib/cirrus/unit_test.sh
@@ -15,17 +15,8 @@ clean_env
 
 set -x
 cd "$GOSRC"
-case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in
-    ubuntu-18) ;&  # Continue to the next item
-    fedora-29) ;&
-    fedora-28) ;&
-    centos-7) ;&
-    rhel-7)
-        make install.tools
-        make localunit
-        make
-        ;;
-    *) bad_os_id_ver ;;
-esac
+make install.tools
+make localunit
+make
 
 record_timestamp "unit test end"