diff options
Diffstat (limited to 'contrib')
73 files changed, 1496 insertions, 655 deletions
diff --git a/contrib/cirrus/README.md b/contrib/cirrus/README.md index 0d315c4f5..e175479f1 100644 --- a/contrib/cirrus/README.md +++ b/contrib/cirrus/README.md @@ -5,6 +5,7 @@ Similar to other integrated github CI/CD services, Cirrus utilizes a simple YAML-based configuration/description file: ``.cirrus.yml``. Ref: https://cirrus-ci.org/ + ## Workflow All tasks execute in parallel, unless there are conditions or dependencies @@ -12,24 +13,34 @@ which alter this behavior. Within each task, each script executes in sequence, so long as any previous script exited successfully. The overall state of each task (pass or fail) is set based on the exit status of the last script to execute. -### ``full_vm_testing`` Task -1. Unconditionally, spin up one VM per ``matrix: image_name`` item defined - in ``.cirrus.yml``. Once accessible, ``ssh`` into each VM and run the following - scripts. +### ``gating`` Task + +***N/B: Steps below are performed by automation*** + +1. Launch a purpose-built container in Cirrus's community cluster. + For container image details, please see + [the contributors guide](https://github.com/containers/libpod/blob/master/CONTRIBUTING.md#go-format-and-lint). + +3. ``validate``: Perform standard `make validate` source verification, + Should run for less than a minute or two. + +4. ``lint``: Execute regular `make lint` to check for any code cruft. + Should also run for less than a few minutes. -2. ``setup_environment.sh``: Configure root's ``.bash_profile`` - for all subsequent scripts (each run in a new shell). Any - distribution-specific environment variables are also defined - here. For example, setting tags/flags to use compiling. -3. ``verify_source.sh``: Perform per-distribution source - verification, lint-checking, etc. This acts as a minimal - gate, blocking extended use of VMs when a PR's code or commits - would otherwise not be accepted. Should run for less than a minute. +### ``testing`` Task -4. ``unit_test.sh``: Execute unit-testing, as defined by the ``Makefile``. - This should execute within 10-minutes, but often much faster. +***N/B: Steps below are performed by automation*** + +1. After `gating` passes, spin up one VM per + `matrix: image_name` item. Once accessible, ``ssh`` + into each VM as the `root` user. + +2. ``setup_environment.sh``: Configure root's `.bash_profile` + for all subsequent scripts (each run in a new shell). Any + distribution-specific environment variables are also defined + here. For example, setting tags/flags to use compiling. 5. ``integration_test.sh``: Execute integration-testing. This is much more involved, and relies on access to external @@ -37,39 +48,163 @@ task (pass or fail) is set based on the exit status of the last script to execut Total execution time is capped at 2-hours (includes all the above) but this script normally completes in less than an hour. -### ``build_vm_images`` Task -1. When a PR is merged (``$CIRRUS_BRANCH`` == ``master``), run another - round of the ``full_vm_testing`` task (above). +### ``optional_testing`` Task + +***N/B: Steps below are performed by automation*** + +1. Optionally executes in parallel with ``testing``. Requires + **prior** to job-start, the magic string ``***CIRRUS: SYSTEM TEST***`` + is found in the pull-request *description*. The *description* is the first + text-box under the main *summary* line in the github WebUI. -2. After confirming the tests all pass post-merge, spin up a special VM - capable of communicating with the GCE API. Once accessible, ``ssh`` into - the special VM and run the following scripts. +2. ``setup_environment.sh``: Same as for other tasks. -3. ``setup_environment.sh``: Configure root's ``.bash_profile`` - for all subsequent scripts (each run in a new shell). Any - distribution-specific environment variables are also defined - here. For example, setting tags/flags to use compiling. +3. ``system_test.sh``: Build both dependencies and libpod, install them, + then execute `make localsystem` from the repository root. -4. ``build_vm_images.sh``: Examine the merged PR's description on github. - If it contains the magic string ``***CIRRUS: REBUILD IMAGES***``, then - continue. Otherwise display a message, take no further action, and - exit successfully. This prevents production of new VM images unless - they are called for, thereby saving the cost of needlessly storing them. -5. If the magic string was found, utilize [the packer tool](http://packer.io/docs/) +### ``cache_images`` Task + +Modifying the contents of cache-images is done by making changes to +one or more of the ``./contrib/cirrus/packer/*_setup.sh`` files. Testing +those changes currently requires adding a temporary commit to a PR that +updates ``.cirrus.yml``: + +* Remove all task sections except ``cache_images_task``. +* Remove the ``only_if`` condition and ``depends_on`` dependencies + +The new image names will be displayed at the end of output, assuming the build +is successful, at that point the temporary commit may be removed. Finally, +the new names may be used as ``image_name`` values in ``.cirrus.yml``. + +***N/B: Steps below are performed by automation*** + +1. When a PR is merged (``$CIRRUS_BRANCH`` == ``master``), run another + round of the ``gating`` and ``testing`` tasks (above). + +2. Assuming tests pass, if the commit message contains the magic string + ``***CIRRUS: REBUILD IMAGES***``, then this task continues. Otherwise + simply mark the master branch as 'passed'. + +3. ``setup_environment.sh``: Same as for other tasks. + +4. ``build_vm_images.sh``: Utilize [the packer tool](http://packer.io/docs/) to produce new VM images. Create a new VM from each base-image, connect - to them with ``ssh``, and perform these steps as defined by the - ``libpod_images.json`` file. + to them with ``ssh``, and perform the steps as defined by the + ``$PACKER_BASE/libpod_images.json`` file: - 1. Copy the current state of the repository into ``/tmp/libpod``. + 1. On a base-image VM, as root, copy the current state of the repository + into ``/tmp/libpod``. 2. Execute distribution-specific scripts to prepare the image for - use by the ``full_vm_testing`` task (above). + use by the ``integration_testing`` task (above). For example, + ``fedora_setup.sh``. 3. If successful, shut down each VM and create a new GCE Image - named after the base image and the commit sha of the merge. - -***Note:*** The ``.cirrus.yml`` file must be manually updated with the new -images names, then the change sent in via a secondary pull-request. This -ensures that all the ``full_vm_testing`` tasks can pass with the new images, -before subjecting all future PRs to them. A workflow to automate this -process is described in comments at the end of the ``.cirrus.yml`` file. + named with the base image, and the commit sha of the merge. + +### Base-images + +Base-images are VM disk-images specially prepared for executing as GCE VMs. +In particular, they run services on startup similar in purpose/function +as the standard 'cloud-init' services. + +* The google services are required for full support of ssh-key management + and GCE OAuth capabilities. Google provides native images in GCE + with services pre-installed, for many platforms. For example, + RHEL, CentOS, and Ubuntu. + +* Google does ***not*** provide any images for Fedora or Fedora Atomic + Host (as of 11/2018), nor do they provide a base-image prepared to + run packer for creating other images in the ``build_vm_images`` Task + (above). + +* Base images do not need to be produced often, but doing so completely + manually would be time-consuming and error-prone. Therefor a special + semi-automatic *Makefile* target is provided to assist with producing + all the base-images: ``libpod_base_images`` + +To produce new base-images, including an `image-builder-image` (used by +the ``cache_images`` Task) some input parameters are required: + +* ``GCP_PROJECT_ID``: The complete GCP project ID string e.g. foobar-12345 + identifying where the images will be stored. + +* ``GOOGLE_APPLICATION_CREDENTIALS``: A *JSON* file containing + credentials for a GCE service account. This can be [a service + account](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually) + or [end-user + credentials](https://cloud.google.com/docs/authentication/end-user#creating_your_client_credentials) + +* ``RHEL_IMAGE_FILE`` and ``RHEL_CSUM_FILE`` complete paths + to a `rhel-server-ec2-*.raw.xz` and it's cooresponding + checksum file. These must be supplied manually because + they're not available directly via URL like other images. + +* ``RHSM_COMMAND`` contains the complete string needed to register + the VM for installing package dependencies. The VM will be de-registered + upon completion. + +* Optionally, CSV's may be specified to ``PACKER_BUILDS`` + to limit the base-images produced. For example, + ``PACKER_BUILDS=fedora,image-builder-image``. + +If there is an existing 'image-builder-image' within GCE, it may be utilized +to produce base-images (in addition to cache-images). However it must be +created with support for nested-virtualization, and with elevated cloud +privileges (to access GCE, from within the GCE VM). For example: + +``` +$ alias pgcloud='sudo podman run -it --rm -e AS_ID=$UID + -e AS_USER=$USER -v $HOME:$HOME:z quay.io/cevich/gcloud_centos:latest' + +$ URL=https://www.googleapis.com/auth +$ SCOPES=$URL/userinfo.email,$URL/compute,$URL/devstorage.full_control + +# The --min-cpu-platform is critical for nested-virt. +$ pgcloud compute instances create $USER-making-images \ + --image-family image-builder-image \ + --boot-disk-size "200GB" \ + --min-cpu-platform "Intel Haswell" \ + --machine-type n1-standard-2 \ + --scopes $SCOPES +``` + +Alternatively, if there is no image-builder-image available yet, a bare-metal +CentOS 7 machine with network access to GCE is required. Software dependencies +can be obtained from the ``packer/image-builder-image_base_setup.sh`` script. + +In both cases, the following can be used to setup and build base-images. + +``` +$ IP_ADDRESS=1.2.3.4 # EXTERNAL_IP from command output above +$ rsync -av $PWD centos@$IP_ADDRESS:. +$ scp $GOOGLE_APPLICATION_CREDENTIALS centos@$IP_ADDRESS:. +$ ssh centos@$IP_ADDRESS +... +``` + +When ready, change to the ``packer`` sub-directory, and build the images: + +``` +$ cd libpod/contrib/cirrus/packer +$ make libpod_base_images GCP_PROJECT_ID=<VALUE> \ + GOOGLE_APPLICATION_CREDENTIALS=<VALUE> \ + RHEL_IMAGE_FILE=<VALUE> \ + RHEL_CSUM_FILE=<VALUE> \ + RHSM_COMMAND=<VALUE> \ + PACKER_BUILDS=<OPTIONAL> +``` + +Assuming this is successful (hence the semi-automatic part), packer will +produce a ``packer-manifest.json`` output file. This contains the base-image +names suitable for updating in ``.cirrus.yml``, `env` keys ``*_BASE_IMAGE``. + +On failure, it should be possible to determine the problem from the packer +output. Sometimes that means setting `PACKER_LOG=1` and troubleshooting +the nested virt calls. It's also possible to observe the (nested) qemu-kvm +console output. Simply set the ``TTYDEV`` parameter, for example: + +``` +$ make libpod_base_images ... TTYDEV=$(tty) + ... +``` diff --git a/contrib/cirrus/build_vm_images.sh b/contrib/cirrus/build_vm_images.sh index ffbb2d5d5..ecdf1d877 100755 --- a/contrib/cirrus/build_vm_images.sh +++ b/contrib/cirrus/build_vm_images.sh @@ -8,12 +8,13 @@ CNI_COMMIT $CNI_COMMIT CRIO_COMMIT $CRIO_COMMIT RUNC_COMMIT $RUNC_COMMIT PACKER_BUILDS $PACKER_BUILDS +BUILT_IMAGE_SUFFIX $BUILT_IMAGE_SUFFIX CENTOS_BASE_IMAGE $CENTOS_BASE_IMAGE UBUNTU_BASE_IMAGE $UBUNTU_BASE_IMAGE FEDORA_BASE_IMAGE $FEDORA_BASE_IMAGE +FAH_BASE_IMAGE $FAH_BASE_IMAGE RHEL_BASE_IMAGE $RHEL_BASE_IMAGE RHSM_COMMAND $RHSM_COMMAND -BUILT_IMAGE_SUFFIX $BUILT_IMAGE_SUFFIX SERVICE_ACCOUNT $SERVICE_ACCOUNT GCE_SSH_USERNAME $GCE_SSH_USERNAME GCP_PROJECT_ID $GCP_PROJECT_ID @@ -22,36 +23,30 @@ SCRIPT_BASE $SCRIPT_BASE PACKER_BASE $PACKER_BASE " -require_regex '\*\*\*\s*CIRRUS:\s*REBUILD\s*IMAGES\s*\*\*\*' 'Not re-building VM images' - show_env_vars # Everything here is running on the 'image-builder-image' GCE image # Assume basic dependencies are all met, but there could be a newer version # of the packer binary PACKER_FILENAME="packer_${PACKER_VER}_linux_amd64.zip" -mkdir -p "$HOME/packer" -cd "$HOME/packer" -# image_builder_image has packer pre-installed, check if same version requested -if ! [[ -r "$PACKER_FILENAME" ]] +if [[ -d "$HOME/packer" ]] then - curl -L -O https://releases.hashicorp.com/packer/$PACKER_VER/$PACKER_FILENAME - curl -L https://releases.hashicorp.com/packer/${PACKER_VER}/packer_${PACKER_VER}_SHA256SUMS | \ - grep 'linux_amd64' > ./sha256sums - sha256sum --check ./sha256sums - unzip -o $PACKER_FILENAME - ./packer --help &> /dev/null # verify exit(0) + cd "$HOME/packer" + # image_builder_image has packer pre-installed, check if same version requested + if [[ -r "$PACKER_FILENAME" ]] + then + cp $PACKER_FILENAME "$GOSRC/$PACKER_BASE/" + cp packer "$GOSRC/$PACKER_BASE/" + fi fi set -x -cd "$GOSRC" -# N/B: /usr/sbin/packer is a DIFFERENT tool, and will exit 0 given the args below :( -TEMPLATE="./$PACKER_BASE/libpod_images.json" - -$HOME/packer/packer inspect "$TEMPLATE" - -#$HOME/packer/packer build -machine-readable "-only=$PACKER_BUILDS" "$TEMPLATE" | tee /tmp/packer_log.csv -$HOME/packer/packer build "-only=$PACKER_BUILDS" "$TEMPLATE" - -# TODO: Report back to PR names of built images +cd "$GOSRC/$PACKER_BASE" +make libpod_images \ + PACKER_BUILDS=$PACKER_BUILDS \ + PACKER_VER=$PACKER_VER \ + GOSRC=$GOSRC \ + SCRIPT_BASE=$SCRIPT_BASE \ + PACKER_BASE=$PACKER_BASE \ + BUILT_IMAGE_SUFFIX=$BUILT_IMAGE_SUFFIX diff --git a/contrib/cirrus/integration_test.sh b/contrib/cirrus/integration_test.sh index 226053724..a50bd448f 100755 --- a/contrib/cirrus/integration_test.sh +++ b/contrib/cirrus/integration_test.sh @@ -9,7 +9,7 @@ OS_RELEASE_ID $OS_RELEASE_ID OS_RELEASE_VER $OS_RELEASE_VER " -show_env_vars +clean_env set -x cd "$GOSRC" @@ -19,10 +19,13 @@ case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in make test-binaries "BUILDTAGS=$BUILDTAGS" SKIP_USERNS=1 make localintegration "BUILDTAGS=$BUILDTAGS" ;; - fedora-28) ;& # Continue to the next item + fedora-29) ;& # Continue to the next item + fedora-28) ;& centos-7) ;& rhel-7) - stub 'integration testing not working on $OS_RELEASE_ID' + make install PREFIX=/usr ETCDIR=/etc + make test-binaries + make localintegration ;; *) bad_os_id_ver ;; esac diff --git a/contrib/cirrus/lib.sh b/contrib/cirrus/lib.sh index 2fa91258b..3b567b7a7 100644 --- a/contrib/cirrus/lib.sh +++ b/contrib/cirrus/lib.sh @@ -4,8 +4,21 @@ # to be sourced by other scripts, not called directly. # Under some contexts these values are not set, make sure they are. -USER="$(whoami)" -HOME="$(getent passwd $USER | cut -d : -f 6)" +export USER="$(whoami)" +export HOME="$(getent passwd $USER | cut -d : -f 6)" + +# These are normally set by cirrus, if not use some reasonable defaults +ENVLIB=${ENVLIB:-.bash_profile} +CIRRUS_WORKING_DIR=${CIRRUS_WORKING_DIR:-/var/tmp/go/src/github.com/containers/libpod} +SCRIPT_BASE=${SCRIPT_BASE:-./contrib/cirrus} +PACKER_BASE=${PACKER_BASE:-./contrib/cirrus/packer} +CIRRUS_BUILD_ID=${CIRRUS_BUILD_ID:-DEADBEEF} # a human +cd "$CIRRUS_WORKING_DIR" +CIRRUS_BASE_SHA=${CIRRUS_BASE_SHA:-$(git rev-parse upstream/master || git rev-parse origin/master)} +CIRRUS_CHANGE_IN_REPO=${CIRRUS_CHANGE_IN_REPO:-$(git rev-parse HEAD)} +CIRRUS_REPO_NAME=${CIRRUS_REPO_NAME:-libpod} +cd - + if ! [[ "$PATH" =~ "/usr/local/bin" ]] then export PATH="$PATH:/usr/local/bin" @@ -73,6 +86,18 @@ PACKER_BUILDS $PACKER_BUILDS do [[ -z "$NAME" ]] || echo "export $NAME=\"$VALUE\"" done + echo "" + echo "##### $(go version) #####" + echo "" +} + +# Unset environment variables not needed for testing purposes +clean_env() { + req_env_var " + UNSET_ENV_VARS $UNSET_ENV_VARS + " + echo "Unsetting $(echo $UNSET_ENV_VARS | wc -w) environment variables" + unset -v UNSET_ENV_VARS $UNSET_ENV_VARS || true # don't fail on read-only } # Return a GCE image-name compatible string representation of distribution name @@ -98,15 +123,17 @@ stub() { ircmsg() { req_env_var " - SCRIPT_BASE $SCRIPT_BASE - GOSRC $GOSRC CIRRUS_TASK_ID $CIRRUS_TASK_ID 1 $1 " - SCRIPT="$GOSRC/$SCRIPT_BASE/podbot.py" + # Sometimes setup_environment.sh didn't run + SCRIPT="$(dirname $0)/podbot.py" NICK="podbot_$CIRRUS_TASK_ID" NICK="${NICK:0:15}" # Any longer will break things + set +e $SCRIPT $NICK $1 + echo "Ignoring exit($?)" + set -e } # Run sudo in directory with GOPATH set @@ -117,23 +144,6 @@ cdsudo() { sudo --preserve-env=GOPATH --non-interactive bash -c "$CMD" } -# Skip a build if $1 does not match in the PR Title/Description with message $2 -require_regex() { - req_env_var " - CIRRUS_CHANGE_MESSAGE $CIRRUS_CHANGE_MESSAGE - 1 $1 - 2 $2 - " - regex="$1" - msg="$2" - if ! echo "$CIRRUS_CHANGE_MESSAGE" | egrep -q "$regex" - then - echo "***** The PR Title/Description did not match the regular expression: $MAGIC_RE" - echo "***** $msg" - exit 0 - fi -} - # Helper/wrapper script to only show stderr/stdout on non-zero exit install_ooe() { req_env_var "SCRIPT_BASE $SCRIPT_BASE" @@ -171,6 +181,19 @@ install_cni_plugins() { sudo cp bin/* /usr/libexec/cni } +install_runc_from_git(){ + wd=$(pwd) + DEST="$GOPATH/src/github.com/opencontainers/runc" + rm -rf "$DEST" + ooe.sh git clone https://github.com/opencontainers/runc.git "$DEST" + cd "$DEST" + ooe.sh git fetch origin --tags + ooe.sh git checkout -q "$RUNC_COMMIT" + ooe.sh make static BUILDTAGS="seccomp selinux" + sudo install -m 755 runc /usr/bin/runc + cd $wd +} + install_runc(){ OS_RELEASE_ID=$(os_release_id) echo "Installing RunC from commit $RUNC_COMMIT" @@ -193,14 +216,7 @@ install_runc(){ cd "$GOPATH/src/github.com/containers/libpod" ooe.sh sudo make install.libseccomp.sudo fi - DEST="$GOPATH/src/github.com/opencontainers/runc" - rm -rf "$DEST" - ooe.sh git clone https://github.com/opencontainers/runc.git "$DEST" - cd "$DEST" - ooe.sh git fetch origin --tags - ooe.sh git checkout -q "$RUNC_COMMIT" - ooe.sh make static BUILDTAGS="seccomp selinux" - sudo install -m 755 runc /usr/bin/runc + install_runc_from_git } install_buildah() { @@ -277,21 +293,29 @@ install_varlink(){ } _finalize(){ + set +e # Don't fail at the very end + set +e # make errors non-fatal echo "Removing leftover giblets from cloud-init" cd / sudo rm -rf /var/lib/cloud/instance? sudo rm -rf /root/.ssh/* sudo rm -rf /home/* + sudo rm -rf /tmp/* + sudo rm -rf /tmp/.??* + sync + sudo fstrim -av } rh_finalize(){ + set +e # Don't fail at the very end # Allow root ssh-logins if [[ -r /etc/cloud/cloud.cfg ]] then sudo sed -re 's/^disable_root:.*/disable_root: 0/g' -i /etc/cloud/cloud.cfg fi echo "Resetting to fresh-state for usage as cloud-image." - sudo $(type -P dnf || type -P yum) clean all + PKG=$(type -P dnf || type -P yum || echo "") + [[ -z "$PKG" ]] || sudo $PKG clean all # not on atomic sudo rm -rf /var/cache/{yum,dnf} sudo rm -f /etc/udev/rules.d/*-persistent-*.rules sudo touch /.unconfigured # force firstboot to run @@ -299,7 +323,35 @@ rh_finalize(){ } ubuntu_finalize(){ + set +e # Don't fail at the very end echo "Resetting to fresh-state for usage as cloud-image." sudo rm -rf /var/cache/apt _finalize } + +rhel_exit_handler() { + set +ex + req_env_var " + GOPATH $GOPATH + RHSMCMD $RHSMCMD + " + cd / + sudo rm -rf "$RHSMCMD" + sudo rm -rf "$GOPATH" + sudo subscription-manager remove --all + sudo subscription-manager unregister + sudo subscription-manager clean +} + +rhsm_enable() { + req_env_var " + RHSM_COMMAND $RHSM_COMMAND + " + export GOPATH="$(mktemp -d)" + export RHSMCMD="$(mktemp)" + trap "rhel_exit_handler" EXIT + # Avoid logging sensitive details + echo "$RHSM_COMMAND" > "$RHSMCMD" + ooe.sh sudo bash "$RHSMCMD" + sudo rm -rf "$RHSMCMD" +} diff --git a/contrib/cirrus/optional_system_test.sh b/contrib/cirrus/optional_system_test.sh deleted file mode 100755 index 705dda5ad..000000000 --- a/contrib/cirrus/optional_system_test.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash - -set -e -source $(dirname $0)/lib.sh - -MAGIC_RE='\*\*\*\s*CIRRUS:\s*SYSTEM\s*TEST\s*\*\*\*' -if ! echo "$CIRRUS_CHANGE_MESSAGE" | egrep -q "$MAGIC_RE" -then - echo "Skipping system-testing because PR title or description" - echo "does not match regular expression: $MAGIC_RE" - exit 0 -fi - -req_env_var " -GOSRC $GOSRC -OS_RELEASE_ID $OS_RELEASE_ID -OS_RELEASE_VER $OS_RELEASE_VER -" - -show_env_vars - -set -x -cd "$GOSRC" -make localsystem diff --git a/contrib/cirrus/packer/.gitignore b/contrib/cirrus/packer/.gitignore new file mode 100644 index 000000000..8f7bdeaf7 --- /dev/null +++ b/contrib/cirrus/packer/.gitignore @@ -0,0 +1,7 @@ +*json +packer +packer*zip +packer_cache +cidata* +meta-data +user-data diff --git a/contrib/cirrus/packer/Makefile b/contrib/cirrus/packer/Makefile new file mode 100644 index 000000000..9bf27373e --- /dev/null +++ b/contrib/cirrus/packer/Makefile @@ -0,0 +1,108 @@ + +# N/B: PACKER_BUILDS variable is required. Should contain CSV of +# builder name(s) from applicable YAML file, +# e.g for names see libpod_images.yml + +PACKER_VER ?= 1.3.1 +PACKER_DIST_FILENAME := packer_${PACKER_VER}_linux_amd64.zip + +# Only needed for libpod_base_images target +TIMESTAMP := $(shell date +%s) +GOSRC ?= $(shell realpath "./../../../") +PACKER_BASE ?= contrib/cirrus/packer +SCRIPT_BASE ?= contrib/cirrus + +# For debugging nested-virt, use +#TTYDEV := $(shell tty) +TTYDEV := /dev/null + +.PHONY: all +all: libpod_images + +%.json: %.yml + @python3 -c 'import json,yaml; json.dump( yaml.load(open("$<").read()), open("$@","w"), indent=2);' + +${PACKER_DIST_FILENAME}: + @curl -L --silent --show-error \ + -O https://releases.hashicorp.com/packer/${PACKER_VER}/${PACKER_DIST_FILENAME} + +packer: ${PACKER_DIST_FILENAME} + @curl -L --silent --show-error \ + https://releases.hashicorp.com/packer/${PACKER_VER}/packer_${PACKER_VER}_SHA256SUMS \ + | grep 'linux_amd64' > /tmp/packer_sha256sums + @sha256sum --check /tmp/packer_sha256sums + @unzip -o ${PACKER_DIST_FILENAME} + @touch --reference=Makefile ${PACKER_DIST_FILENAME} + +.PHONY: test +test: libpod_base_images.json libpod_images.json packer + ./packer inspect libpod_base_images.json > /dev/null + ./packer inspect libpod_images.json > /dev/null + @echo "All good" + +.PHONY: libpod_images +libpod_images: libpod_images.json packer +ifndef PACKER_BUILDS + $(error PACKER_BUILDS is undefined, expected builder-names CSV) +endif + ./packer build -only=${PACKER_BUILDS} \ + -var GOSRC=$(GOSRC) \ + -var PACKER_BASE=$(PACKER_BASE) \ + -var SCRIPT_BASE=$(SCRIPT_BASE) \ + libpod_images.json + @echo "" + @echo "Finished. The images mentioned above, and in packer-manifest.json" + @echo "can be used in .cirrus.yml as values for the 'image_name' keys" + @echo "" + +cidata.ssh: + ssh-keygen -f $@ -P "" -q + +cidata.ssh.pub: cidata.ssh + touch $@ + +meta-data: + echo "local-hostname: localhost.localdomain" > $@ + +user-data: cidata.ssh.pub + bash make-user-data.sh + +cidata.iso: user-data meta-data + genisoimage -output cidata.iso -volid cidata -input-charset utf-8 -joliet -rock user-data meta-data + +# This is intended to be run by a human, with admin access to the libpod GCE project. +.PHONY: libpod_base_images +libpod_base_images: libpod_base_images.json cidata.iso cidata.ssh packer +ifndef GCP_PROJECT_ID + $(error GCP_PROJECT_ID is undefined, expected complete GCP project ID string e.g. foobar-12345) +endif +ifndef GOOGLE_APPLICATION_CREDENTIALS + $(error GOOGLE_APPLICATION_CREDENTIALS is undefined, expected absolute path to JSON file, like $HOME/.config/gcloud/legacy_credentials/*/adc.json) +endif +ifndef RHEL_IMAGE_FILE + $(error RHEL_IMAGE_FILE is undefined, expected full path to a rhel-server-ec2-*.raw.xz file) +endif +ifndef RHEL_CSUM_FILE + $(error RHEL_CSUM_FILE is undefined, expected full path to a rhel-server-ec2-*.raw.xz.SHA256SUM file) +endif +ifndef RHSM_COMMAND + $(error RHSM_COMMAND is undefined, expected string required for temporarily registering VM) +endif + PACKER_CACHE_DIR=/tmp ./packer build \ + -var TIMESTAMP=$(TIMESTAMP) \ + -var TTYDEV=$(TTYDEV) \ + -var GCP_PROJECT_ID=$(GCP_PROJECT_ID) \ + -var GOOGLE_APPLICATION_CREDENTIALS=$(GOOGLE_APPLICATION_CREDENTIALS) \ + -var GOSRC=$(GOSRC) \ + -var PACKER_BASE=$(PACKER_BASE) \ + -var SCRIPT_BASE=$(SCRIPT_BASE) \ + -var RHEL_BASE_IMAGE_NAME=$(shell basename $(RHEL_IMAGE_FILE) | tr -d '[[:space:]]' | sed -r -e 's/\.x86_64\.raw\.xz//' | tr '[[:upper:]]' '[[:lower:]]' | tr '[[:punct:]]' '-') \ + -var RHEL_IMAGE_FILE=$(RHEL_IMAGE_FILE) \ + -var RHEL_CSUM_FILE=$(RHEL_CSUM_FILE) \ + -var 'RHSM_COMMAND=$(RHSM_COMMAND)' \ + -only $(PACKER_BUILDS) \ + libpod_base_images.json + @echo "" + @echo "Finished. The images mentioned above, and in packer-manifest.json" + @echo "can be used in .cirrus.yml as values for the *_BASE_IMAGE keys." + @echo "" diff --git a/contrib/cirrus/packer/README.md b/contrib/cirrus/packer/README.md index 8ff6947e9..9a07ed960 100644 --- a/contrib/cirrus/packer/README.md +++ b/contrib/cirrus/packer/README.md @@ -1,2 +1,3 @@ These are definitions and scripts consumed by packer to produce the -various distribution images used for CI testing. +various distribution images used for CI testing. For more details +see the [Cirrus CI documentation](../README.md) diff --git a/contrib/cirrus/packer/centos_setup.sh b/contrib/cirrus/packer/centos_setup.sh index 7b2308739..a13050569 100644 --- a/contrib/cirrus/packer/centos_setup.sh +++ b/contrib/cirrus/packer/centos_setup.sh @@ -29,6 +29,7 @@ ooe.sh sudo yum -y install \ btrfs-progs-devel \ bzip2 \ device-mapper-devel \ + emacs-nox \ findutils \ glib2-devel \ glibc-static \ @@ -63,6 +64,7 @@ ooe.sh sudo yum -y install \ runc \ skopeo-containers \ unzip \ + vim \ which \ xz diff --git a/contrib/cirrus/packer/fah_base-setup.sh b/contrib/cirrus/packer/fah_base-setup.sh new file mode 100644 index 000000000..606c4f336 --- /dev/null +++ b/contrib/cirrus/packer/fah_base-setup.sh @@ -0,0 +1,45 @@ + +# N/B: This script is not intended to be run by humans. It is used to configure the +# FAH base image for importing, so that it will boot in GCE. + +set -e + +# Load in library (copied by packer, before this script was run) +source $GOSRC/$SCRIPT_BASE/lib.sh + +install_ooe + +if [[ "$1" == "pre" ]] +then + echo "Upgrading Atomic Host" + setenforce 0 + ooe.sh atomic host upgrade + + echo "Configuring Repositories" + ooe.sh sudo tee /etc/yum.repos.d/ngompa-gce-oslogin.repo <<EOF +[ngompa-gce-oslogin] +name=Copr repo for gce-oslogin owned by ngompa +baseurl=https://copr-be.cloud.fedoraproject.org/results/ngompa/gce-oslogin/fedora-\$releasever-\$basearch/ +type=rpm-md +skip_if_unavailable=True +gpgcheck=1 +gpgkey=https://copr-be.cloud.fedoraproject.org/results/ngompa/gce-oslogin/pubkey.gpg +repo_gpgcheck=0 +enabled=1 +enabled_metadata=1 +EOF + echo "Installing necessary packages and google services" + # Google services are enabled by default, upon install. + ooe.sh rpm-ostree install rng-tools google-compute-engine google-compute-engine-oslogin + echo "Rebooting..." + systemctl reboot # Required for upgrade + package installs to be active +elif [[ "$1" == "post" ]] +then + echo "Enabling necessary services" + systemctl enable rngd # Must reboot before enabling + rh_finalize + echo "SUCCESS!" +else + echo "Expected to be called with 'pre' or 'post'" + exit 6 +fi diff --git a/contrib/cirrus/packer/fah_setup.sh b/contrib/cirrus/packer/fah_setup.sh new file mode 100644 index 000000000..2e053b396 --- /dev/null +++ b/contrib/cirrus/packer/fah_setup.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# This script is called by packer on the subject fah VM, to setup the podman +# build/test environment. It's not intended to be used outside of this context. + +set -e + +# Load in library (copied by packer, before this script was run) +source /tmp/libpod/$SCRIPT_BASE/lib.sh + +req_env_var " +SCRIPT_BASE $SCRIPT_BASE +" + +install_ooe + +ooe.sh sudo atomic host upgrade + +ooe.sh sudo rpm-ostree uninstall cloud-init + +rh_finalize + +echo "SUCCESS!" diff --git a/contrib/cirrus/packer/fedora_base-setup.sh b/contrib/cirrus/packer/fedora_base-setup.sh new file mode 100644 index 000000000..c0a1e422c --- /dev/null +++ b/contrib/cirrus/packer/fedora_base-setup.sh @@ -0,0 +1,27 @@ +#!/bin/bash + +# N/B: This script is not intended to be run by humans. It is used to configure the +# fedora base image for importing, so that it will boot in GCE + +set -e + +# Load in library (copied by packer, before this script was run) +source $GOSRC/$SCRIPT_BASE/lib.sh + +[[ "$1" == "post" ]] || exit 0 # nothing to do + +install_ooe + +echo "Updating packages" +ooe.sh dnf -y update + +echo "Installing necessary packages and google services" +ooe.sh dnf -y copr enable ngompa/gce-oslogin +ooe.sh dnf -y install rng-tools google-compute-engine google-compute-engine-oslogin + +echo "Enabling services" +ooe.sh systemctl enable rngd + +rh_finalize + +echo "SUCCESS!" diff --git a/contrib/cirrus/packer/fedora_setup.sh b/contrib/cirrus/packer/fedora_setup.sh index f9fea04a7..01c468901 100644 --- a/contrib/cirrus/packer/fedora_setup.sh +++ b/contrib/cirrus/packer/fedora_setup.sh @@ -10,6 +10,7 @@ source /tmp/libpod/$SCRIPT_BASE/lib.sh req_env_var " SCRIPT_BASE $SCRIPT_BASE +FEDORA_CNI_COMMIT $FEDORA_CNI_COMMIT CNI_COMMIT $CNI_COMMIT CRIO_COMMIT $CRIO_COMMIT CRIU_COMMIT $CRIU_COMMIT @@ -27,8 +28,8 @@ ooe.sh sudo dnf install -y \ atomic-registries \ btrfs-progs-devel \ bzip2 \ - conmon \ device-mapper-devel \ + emacs-nox \ findutils \ git \ glib2-devel \ @@ -65,11 +66,14 @@ ooe.sh sudo dnf install -y \ runc \ skopeo-containers \ slirp4netns \ + unzip \ + vim \ which \ xz install_varlink +CNI_COMMIT=$FEDORA_CNI_COMMIT install_cni_plugins install_buildah diff --git a/contrib/cirrus/packer/image-builder-image_base-setup.sh b/contrib/cirrus/packer/image-builder-image_base-setup.sh new file mode 100644 index 000000000..b8e2824a7 --- /dev/null +++ b/contrib/cirrus/packer/image-builder-image_base-setup.sh @@ -0,0 +1,75 @@ +#!/bin/bash + +# This script is called by packer on a vanilla CentOS VM, to setup the image +# used for building images FROM base images. It's not intended to be used +# outside of this context. + +set -e + +[[ "$1" == "post" ]] || exit 0 # pre stage not needed + +# Load in library (copied by packer, before this script was run) +source $GOSRC/$SCRIPT_BASE/lib.sh + +req_env_var " + TIMESTAMP $TIMESTAMP + GOSRC $GOSRC + SCRIPT_BASE $SCRIPT_BASE + PACKER_BASE $PACKER_BASE +" + +install_ooe + +echo "Updating packages" +ooe.sh sudo yum -y update + +echo "Configuring repositories" +ooe.sh sudo yum -y install centos-release-scl epel-release + +echo "Installing packages" +ooe.sh sudo yum -y install \ + genisoimage \ + golang \ + google-cloud-sdk \ + libvirt \ + libvirt-admin \ + libvirt-client \ + libvirt-daemon \ + make \ + python34 \ + python34 \ + python34-PyYAML \ + python34-PyYAML \ + qemu-img \ + qemu-kvm \ + qemu-kvm-tools \ + qemu-user \ + rsync \ + unzip \ + util-linux \ + vim + +sudo ln -s /usr/libexec/qemu-kvm /usr/bin/ + +sudo tee /etc/modprobe.d/kvm-nested.conf <<EOF +options kvm-intel nested=1 +options kvm-intel enable_shadow_vmcs=1 +options kvm-intel enable_apicv=1 +options kvm-intel ept=1 +EOF + +echo "Installing packer" +sudo mkdir -p /root/$(basename $PACKER_BASE) +sudo cp $GOSRC/$PACKER_BASE/*packer* /root/$(basename $PACKER_BASE) +sudo mkdir -p /root/$(basename $SCRIPT_BASE) +sudo cp $GOSRC/$SCRIPT_BASE/*.sh /root/$(basename $SCRIPT_BASE) + +install_scl_git + +echo "Cleaning up" +cd / +rm -rf $GOSRC + +rh_finalize + +echo "SUCCESS!" diff --git a/contrib/cirrus/packer/libpod_base_images.yml b/contrib/cirrus/packer/libpod_base_images.yml new file mode 100644 index 000000000..109b9b8d5 --- /dev/null +++ b/contrib/cirrus/packer/libpod_base_images.yml @@ -0,0 +1,193 @@ +--- + +variables: + # Complete local path to this repository (Required) + GOSRC: + # Relative path to this (packer) subdirectory (Required) + PACKER_BASE: + # Relative path to cirrus scripts subdirectory (Required) + SCRIPT_BASE: + # Unique ID for naming new base-images (required) + TIMESTAMP: + # Required for output from qemu builders + TTYDEV: + # RHEL images require click-through agreements to obtain (required) + RHEL_BASE_IMAGE_NAME: + RHEL_IMAGE_FILE: + RHEL_CSUM_FILE: + # RHEL requires a subscription to install/update packages + RHSM_COMMAND: + + # Latest Fedora release + FEDORA_IMAGE_URL: "https://dl.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2" + FEDORA_CSUM_URL: "https://dl.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-29-1.2-x86_64-CHECKSUM" + FEDORA_BASE_IMAGE_NAME: 'fedora-cloud-base-29-1-2' # Name to use in GCE + # Prior Fedora release + PRIOR_FEDORA_IMAGE_URL: "https://dl.fedoraproject.org/pub/fedora/linux/releases/28/Cloud/x86_64/images/Fedora-Cloud-Base-28-1.1.x86_64.qcow2" + PRIOR_FEDORA_CSUM_URL: "https://dl.fedoraproject.org/pub/fedora/linux/releases/28/Cloud/x86_64/images/Fedora-Cloud-28-1.1-x86_64-CHECKSUM" + PRIOR_FEDORA_BASE_IMAGE_NAME: 'fedora-cloud-base-28-1-1' # Name to use in GCE + FAH_IMAGE_URL: "https://dl.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-29-20181025.1/AtomicHost/x86_64/images/Fedora-AtomicHost-29-20181025.1.x86_64.qcow2" + FAH_CSUM_URL: "https://dl.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-29-20181025.1/AtomicHost/x86_64/images/Fedora-AtomicHost-29-20181025.1-x86_64-CHECKSUM" + FAH_BASE_IMAGE_NAME: 'fedora-atomichost-29-20181025-1' # Name to use in GCE + + # The name of the image in GCE used for packer build libpod_images.yml + IBI_BASE_NAME: 'image-builder-image' + CIDATA_ISO: 'cidata.iso' # produced by Makefile + + # Path to json file (required, likely ~/.config/gcloud/legacy_credentials/*/adc.json) + GOOGLE_APPLICATION_CREDENTIALS: + # The complete project ID (required, not the short name) + GCP_PROJECT_ID: + # Pre-existing storage bucket w/ lifecycle-enabled + XFERBUCKET: "packer-import" # pre-created, globally unique, lifecycle-enabled + +# Don't leak sensitive values in error messages / output +sensitive-variables: + - 'GOOGLE_APPLICATION_CREDENTIALS' + - 'GCP_PROJECT_ID' + - 'RHSM_COMMAND' + +# What images to produce in which cloud +builders: + - name: '{{user `IBI_BASE_NAME`}}' + type: 'googlecompute' + image_name: '{{user `IBI_BASE_NAME`}}-{{user `TIMESTAMP`}}' + image_family: '{{user `IBI_BASE_NAME`}}' + source_image_project_id: 'centos-cloud' + source_image_family: 'centos-7' + project_id: '{{user `GCP_PROJECT_ID`}}' + account_file: '{{user `GOOGLE_APPLICATION_CREDENTIALS`}}' + communicator: 'ssh' + ssh_username: 'centos' + ssh_pty: 'true' + # The only supported zone in Cirrus-CI, as of addition of this comment + zone: 'us-central1-a' + # Enable nested virtualization in case it's ever needed + image_licenses: + - 'https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx' + min_cpu_platform: "Intel Broadwell" # nested-virt requirement + + - &nested_virt + name: 'fedora' + type: 'qemu' + accelerator: "kvm" + iso_url: '{{user `FEDORA_IMAGE_URL`}}' + disk_image: true + format: "raw" + disk_size: 5120 + iso_checksum_url: '{{user `FEDORA_CSUM_URL`}}' + iso_checksum_type: "sha256" + output_directory: '/tmp/{{build_name}}' + vm_name: "disk.raw" # actually qcow2, name required for post-processing + boot_wait: '5s' + shutdown_command: 'shutdown -h now' + headless: true + qemu_binary: "/usr/libexec/qemu-kvm" + qemuargs: # List-of-list format required to override packer-generated args + - - "-m" + - "1024" + - - "-cpu" + - "host" + - - "-device" + - "virtio-rng-pci" + - - "-chardev" + - "tty,id=pts,path={{user `TTYDEV`}}" + - - "-device" + - "isa-serial,chardev=pts" + - - "-cdrom" + - "{{user `CIDATA_ISO`}}" + - - "-netdev" + - "user,id=net0,hostfwd=tcp::{{ .SSHHostPort }}-:22" + - - "-device" + - "virtio-net,netdev=net0" + communicator: 'ssh' + ssh_private_key_file: 'cidata.ssh' + ssh_username: 'root' + + - <<: *nested_virt + name: 'prior_fedora' + iso_url: '{{user `PRIOR_FEDORA_IMAGE_URL`}}' + iso_checksum_url: '{{user `PRIOR_FEDORA_CSUM_URL`}}' + + - <<: *nested_virt + name: 'fah' + iso_url: '{{user `FAH_IMAGE_URL`}}' + iso_checksum_url: '{{user `FAH_CSUM_URL`}}' + disk_size: 10240 + + - <<: *nested_virt + name: 'rhel' + iso_url: 'file://{{user `RHEL_IMAGE_FILE`}}' + iso_checksum_url: 'file://{{user `RHEL_CSUM_FILE`}}' + disk_size: 10240 + +provisioners: + - type: 'shell' + inline: + - 'mkdir -p /tmp/libpod/{{user `SCRIPT_BASE`}}' + - 'mkdir -p /tmp/libpod/{{user `PACKER_BASE`}}' + + - type: 'file' + source: '{{user `GOSRC`}}/.cirrus.yml' + destination: '/tmp/libpod/.cirrus.yml' + + - type: 'file' + source: '{{user `GOSRC`}}/{{user `SCRIPT_BASE`}}/' + destination: '/tmp/libpod/{{user `SCRIPT_BASE`}}/' + + - type: 'file' + source: '{{user `GOSRC`}}/{{user `PACKER_BASE`}}/' + destination: '/tmp/libpod/{{user `PACKER_BASE`}}/' + + - &shell_script + type: 'shell' + inline: + - 'chmod +x /tmp/libpod/{{user `PACKER_BASE`}}/{{build_name}}_base-setup.sh' + - '/tmp/libpod/{{user `PACKER_BASE`}}/{{build_name}}_base-setup.sh pre' + expect_disconnect: true # Allow this to reboot the VM + environment_vars: + - 'TIMESTAMP={{user `TIMESTAMP`}}' + - 'GOSRC=/tmp/libpod' + - 'SCRIPT_BASE={{user `SCRIPT_BASE`}}' + - 'PACKER_BASE={{user `PACKER_BASE`}}' + - 'RHSM_COMMAND={{user `RHSM_COMMAND`}}' + + - <<: *shell_script + inline: ['{{user `GOSRC`}}/{{user `PACKER_BASE`}}/{{build_name}}_base-setup.sh'] + expect_disconnect: false + pause_before: '10s' + inline: + - '/tmp/libpod/{{user `PACKER_BASE`}}/{{build_name}}_base-setup.sh post' + +post-processors: + - - type: "compress" + only: ['fedora', 'prior_fedora', 'fah', 'rhel'] + output: '/tmp/{{build_name}}/disk.raw.tar.gz' + format: '.tar.gz' + compression_level: 9 + - &gcp_import + only: ['fedora'] + type: "googlecompute-import" + project_id: '{{user `GCP_PROJECT_ID`}}' + account_file: '{{user `GOOGLE_APPLICATION_CREDENTIALS`}}' + bucket: '{{user `XFERBUCKET`}}' + gcs_object_name: '{{build_name}}-{{user `TIMESTAMP`}}-{{uuid}}.tar.gz' + image_name: "{{user `FEDORA_BASE_IMAGE_NAME`}}-{{user `TIMESTAMP`}}" + image_description: 'Based on {{user `FEDORA_IMAGE_URL`}}' + image_family: '{{user `FEDORA_BASE_IMAGE_NAME`}}' + - <<: *gcp_import + only: ['prior_fedora'] + image_name: "{{user `PRIOR_FEDORA_BASE_IMAGE_NAME`}}-{{user `TIMESTAMP`}}" + image_description: 'Based on {{user `PRIOR_FEDORA_IMAGE_URL`}}' + image_family: '{{user `PRIOR_FEDORA_BASE_IMAGE_NAME`}}' + - <<: *gcp_import + only: ['fah'] + image_name: "{{user `FAH_BASE_IMAGE_NAME`}}-{{user `TIMESTAMP`}}" + image_description: 'Based on {{user `FAH_IMAGE_URL`}}' + image_family: '{{user `FAH_BASE_IMAGE_NAME`}}' + - <<: *gcp_import + only: ['rhel'] + image_name: "{{user `RHEL_BASE_IMAGE_NAME`}}-{{user `TIMESTAMP`}}" + image_description: 'Based on {{user `RHEL_IMAGE_FILE`}}' + image_family: '{{user `RHEL_BASE_IMAGE_NAME`}}' + - type: 'manifest' diff --git a/contrib/cirrus/packer/libpod_images.json b/contrib/cirrus/packer/libpod_images.json deleted file mode 100644 index 9dac3e8ea..000000000 --- a/contrib/cirrus/packer/libpod_images.json +++ /dev/null @@ -1,130 +0,0 @@ -{ - "variables": { - "FEDORA_CNI_COMMIT": "{{env `FEDORA_CNI_COMMIT`}}", - "CNI_COMMIT": "{{env `CNI_COMMIT`}}", - "CRIO_COMMIT": "{{env `CRIO_COMMIT`}}", - "CRIU_COMMIT": "{{env `CRIU_COMMIT`}}", - "RUNC_COMMIT": "{{env `RUNC_COMMIT`}}", - - "CENTOS_BASE_IMAGE": "{{env `CENTOS_BASE_IMAGE`}}" , - "UBUNTU_BASE_IMAGE": "{{env `UBUNTU_BASE_IMAGE`}}", - "FEDORA_BASE_IMAGE": "{{env `FEDORA_BASE_IMAGE`}}", - "RHEL_BASE_IMAGE": "{{env `RHEL_BASE_IMAGE`}}", - - "GOSRC": "{{env `GOSRC`}}", - "PACKER_BASE": "{{env `PACKER_BASE`}}", - "SCRIPT_BASE": "{{env `SCRIPT_BASE`}}", - - "SERVICE_ACCOUNT": "{{env `SERVICE_ACCOUNT`}}", - "GCP_PROJECT_ID": "{{env `GCP_PROJECT_ID`}}", - "BUILT_IMAGE_SUFFIX": "{{env `BUILT_IMAGE_SUFFIX`}}", - "GCE_SSH_USERNAME": "{{env `GCE_SSH_USERNAME`}}", - "RHSM_COMMAND": "{{env `RHSM_COMMAND`}}" - }, - "sensitive-variables": [ - "GCP_PROJECT_ID", "SERVICE_ACCOUNT", "GCE_SSH_USERNAME", "RHSM_COMMAND" - ], - "builders": [ - { - "name": "rhel-7", - "type": "googlecompute", - "project_id": "{{user `GCP_PROJECT_ID`}}", - "zone": "us-central1-a", - "source_image": "{{user `RHEL_BASE_IMAGE`}}", - "image_name": "{{user `RHEL_BASE_IMAGE`}}{{user `BUILT_IMAGE_SUFFIX`}}", - "image_family": "{{user `RHEL_BASE_IMAGE`}}-libpod", - "service_account_email": "{{user `SERVICE_ACCOUNT`}}", - "communicator": "ssh", - "ssh_username": "ec2-user", - "ssh_pty": "true" - },{ - "name": "centos-7", - "type": "googlecompute", - "project_id": "{{user `GCP_PROJECT_ID`}}", - "zone": "us-central1-a", - "source_image": "{{user `CENTOS_BASE_IMAGE`}}", - "image_name": "{{user `CENTOS_BASE_IMAGE`}}{{user `BUILT_IMAGE_SUFFIX`}}", - "image_family": "{{user `CENTOS_BASE_IMAGE`}}-libpod", - "service_account_email": "{{user `SERVICE_ACCOUNT`}}", - "communicator": "ssh", - "ssh_username": "{{user `GCE_SSH_USERNAME`}}", - "ssh_pty": "true" - },{ - "name": "fedora-28", - "type": "googlecompute", - "project_id": "{{user `GCP_PROJECT_ID`}}", - "zone": "us-central1-a", - "source_image": "{{user `FEDORA_BASE_IMAGE`}}", - "image_name": "{{user `FEDORA_BASE_IMAGE`}}{{user `BUILT_IMAGE_SUFFIX`}}", - "image_family": "{{user `FEDORA_BASE_IMAGE`}}-libpod", - "service_account_email": "{{user `SERVICE_ACCOUNT`}}", - "communicator": "ssh", - "ssh_username": "fedora", - "ssh_pty": "true" - },{ - "name": "ubuntu-18", - "type": "googlecompute", - "project_id": "{{user `GCP_PROJECT_ID`}}", - "zone": "us-central1-a", - "source_image": "{{user `UBUNTU_BASE_IMAGE`}}", - "image_name": "{{user `UBUNTU_BASE_IMAGE`}}{{user `BUILT_IMAGE_SUFFIX`}}", - "image_family": "{{user `UBUNTU_BASE_IMAGE`}}-libpod", - "service_account_email": "{{user `SERVICE_ACCOUNT`}}", - "communicator": "ssh", - "ssh_username": "{{user `GCE_SSH_USERNAME`}}", - "ssh_pty": "true" - } - ], - "provisioners": [ - { - "type": "file", - "source": "{{user `GOSRC`}}", - "destination": "/tmp/libpod" - },{ - "type": "shell", - "only": ["rhel-7"], - "script": "{{user `GOSRC`}}/{{user `PACKER_BASE`}}/rhel_setup.sh", - "environment_vars": [ - "SCRIPT_BASE={{user `SCRIPT_BASE`}}", - "CNI_COMMIT={{user `CNI_COMMIT`}}", - "CRIO_COMMIT={{user `CRIO_COMMIT`}}", - "CRIU_COMMIT={{user `CRIU_COMMIT`}}", - "RUNC_COMMIT={{user `RUNC_COMMIT`}}", - "RHSM_COMMAND={{user `RHSM_COMMAND`}}" - ] - },{ - "type": "shell", - "only": ["centos-7"], - "script": "{{user `GOSRC`}}/{{user `PACKER_BASE`}}/centos_setup.sh", - "environment_vars": [ - "SCRIPT_BASE={{user `SCRIPT_BASE`}}", - "CNI_COMMIT={{user `CNI_COMMIT`}}", - "CRIO_COMMIT={{user `CRIO_COMMIT`}}", - "CRIU_COMMIT={{user `CRIU_COMMIT`}}", - "RUNC_COMMIT={{user `RUNC_COMMIT`}}" - ] - },{ - "type": "shell", - "only": ["fedora-28"], - "script": "{{user `GOSRC`}}/{{user `PACKER_BASE`}}/fedora_setup.sh", - "environment_vars": [ - "SCRIPT_BASE={{user `SCRIPT_BASE`}}", - "CNI_COMMIT={{user `FEDORA_CNI_COMMIT`}}", - "CRIO_COMMIT={{user `CRIO_COMMIT`}}", - "CRIU_COMMIT={{user `CRIU_COMMIT`}}", - "RUNC_COMMIT={{user `RUNC_COMMIT`}}" - ] - },{ - "type": "shell", - "only": ["ubuntu-18"], - "script": "{{user `GOSRC`}}/{{user `PACKER_BASE`}}/ubuntu_setup.sh", - "environment_vars": [ - "SCRIPT_BASE={{user `SCRIPT_BASE`}}", - "CNI_COMMIT={{user `CNI_COMMIT`}}", - "CRIO_COMMIT={{user `CRIO_COMMIT`}}", - "CRIU_COMMIT={{user `CRIU_COMMIT`}}", - "RUNC_COMMIT={{user `RUNC_COMMIT`}}" - ] - } - ] -} diff --git a/contrib/cirrus/packer/libpod_images.yml b/contrib/cirrus/packer/libpod_images.yml new file mode 100644 index 000000000..d31c11a8d --- /dev/null +++ b/contrib/cirrus/packer/libpod_images.yml @@ -0,0 +1,96 @@ +--- + +# All of these are required +variables: + # Names of GCE Base images to start from, in .cirrus.yml + RHEL_BASE_IMAGE: '{{env `RHEL_BASE_IMAGE`}}' + CENTOS_BASE_IMAGE: '{{env `CENTOS_BASE_IMAGE`}}' + UBUNTU_BASE_IMAGE: '{{env `UBUNTU_BASE_IMAGE`}}' + FEDORA_BASE_IMAGE: '{{env `FEDORA_BASE_IMAGE`}}' + PRIOR_FEDORA_BASE_IMAGE: '{{env `PRIOR_FEDORA_BASE_IMAGE`}}' + FAH_BASE_IMAGE: '{{env `FAH_BASE_IMAGE`}}' + + # libpod dependencies to build and install into images + FEDORA_CNI_COMMIT: "{{env `FEDORA_CNI_COMMIT`}}" + CNI_COMMIT: "{{env `CNI_COMMIT`}}" + CRIO_COMMIT: "{{env `CRIO_COMMIT`}}" + CRIU_COMMIT: "{{env `CRIU_COMMIT`}}" + RUNC_COMMIT: "{{env `RUNC_COMMIT`}}" + + BUILT_IMAGE_SUFFIX: '{{env `BUILT_IMAGE_SUFFIX`}}' + GOSRC: '{{env `GOSRC`}}' + PACKER_BASE: '{{env `PACKER_BASE`}}' + SCRIPT_BASE: '{{env `SCRIPT_BASE`}}' + + # Protected credentials, decrypted by Cirrus at runtime + GCE_SSH_USERNAME: '{{env `GCE_SSH_USERNAME`}}' + GCP_PROJECT_ID: '{{env `GCP_PROJECT_ID`}}' + RHSM_COMMAND: '{{env `RHSM_COMMAND`}}' + SERVICE_ACCOUNT: '{{env `SERVICE_ACCOUNT`}}' + GOOGLE_APPLICATION_CREDENTIALS: '{{env `GOOGLE_APPLICATION_CREDENTIALS`}}' + +# Don't leak sensitive values in error messages / output +sensitive-variables: + - 'GCE_SSH_USERNAME' + - 'GCP_PROJECT_ID' + - 'RHSM_COMMAND' + - 'SERVICE_ACCOUNT' + +# What images to produce in which cloud +builders: + # v----- is a YAML anchor, allows referencing this object by name (below) + - &gce_hosted_image + name: 'ubuntu-18' + type: 'googlecompute' + image_name: '{{build_name}}{{user `BUILT_IMAGE_SUFFIX`}}' + image_family: '{{build_name}}-libpod' + source_image: '{{user `UBUNTU_BASE_IMAGE`}}' + disk_size: 20 + project_id: '{{user `GCP_PROJECT_ID`}}' + service_account_email: '{{user `SERVICE_ACCOUNT`}}' + communicator: 'ssh' + ssh_username: '{{user `GCE_SSH_USERNAME`}}' + ssh_pty: 'true' + # The only supported zone in Cirrus-CI, as of addition of this comment + zone: 'us-central1-a' + + # v----- is a YAML alias, allows partial re-use of the anchor object + - <<: *gce_hosted_image + name: 'rhel-7' + source_image: '{{user `RHEL_BASE_IMAGE`}}' + + - <<: *gce_hosted_image + name: 'centos-7' + source_image: '{{user `CENTOS_BASE_IMAGE`}}' + + - <<: *gce_hosted_image + name: 'fedora-29' + source_image: '{{user `FEDORA_BASE_IMAGE`}}' + + - <<: *gce_hosted_image + name: 'fedora-28' + source_image: '{{user `PRIOR_FEDORA_BASE_IMAGE`}}' + + - <<: *gce_hosted_image + name: 'fah-29' + source_image: '{{user `FAH_BASE_IMAGE`}}' + +# The brains of the operation, making actual modifications to the base-image. +provisioners: + - type: 'file' + source: '{{user `GOSRC`}}' + destination: '/tmp/libpod' + + - type: 'shell' + script: '{{user `GOSRC`}}/{{user `PACKER_BASE`}}/{{split build_name "-" 0}}_setup.sh' + environment_vars: + - 'SCRIPT_BASE={{user `SCRIPT_BASE`}}' + - 'CNI_COMMIT={{user `CNI_COMMIT`}}' + - 'FEDORA_CNI_COMMIT={{user `FEDORA_CNI_COMMIT`}}' + - 'CRIO_COMMIT={{user `CRIO_COMMIT`}}' + - 'CRIU_COMMIT={{user `CRIU_COMMIT`}}' + - 'RUNC_COMMIT={{user `RUNC_COMMIT`}}' + - 'RHSM_COMMAND={{user `RHSM_COMMAND`}}' + +post-processors: + - - type: 'manifest' diff --git a/contrib/cirrus/packer/make-user-data.sh b/contrib/cirrus/packer/make-user-data.sh new file mode 100644 index 000000000..7f7fa1c1a --- /dev/null +++ b/contrib/cirrus/packer/make-user-data.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# This script is utilized by Makefile, it's not intended to be run by humans + +cat <<EOF > user-data +#cloud-config +timezone: US/Eastern +growpart: + mode: auto +disable_root: false +ssh_pwauth: True +ssh_import_id: [root] +ssh_authorized_keys: + - $(cat cidata.ssh.pub) +users: + - name: root + primary-group: root + homedir: /root + system: true +EOF diff --git a/contrib/cirrus/packer/prior_fedora_base-setup.sh b/contrib/cirrus/packer/prior_fedora_base-setup.sh new file mode 120000 index 000000000..998a5d9fd --- /dev/null +++ b/contrib/cirrus/packer/prior_fedora_base-setup.sh @@ -0,0 +1 @@ +fedora_base-setup.sh
\ No newline at end of file diff --git a/contrib/cirrus/packer/rhel_base-setup.sh b/contrib/cirrus/packer/rhel_base-setup.sh new file mode 100644 index 000000000..8b2073d4f --- /dev/null +++ b/contrib/cirrus/packer/rhel_base-setup.sh @@ -0,0 +1,52 @@ +#!/bin/bash + +# N/B: This script is not intended to be run by humans. It is used to configure the +# rhel base image for importing, so that it will boot in GCE + +set -e + +[[ "$1" == "post" ]] || exit 0 # pre stage is not needed + +# Load in library (copied by packer, before this script was run) +source $GOSRC/$SCRIPT_BASE/lib.sh + +req_env_var " + RHSM_COMMAND $RHSM_COMMAND +" + +install_ooe + +echo "Setting up repos" +# Frequently needed +ooe.sh sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + +# Required for google to manage ssh keys +ooe.sh sudo tee /etc/yum.repos.d/google-cloud-sdk.repo << EOM +[google-cloud-compute] +name=google-cloud-compute +baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-compute-el7-x86_64 +enabled=1 +gpgcheck=1 +repo_gpgcheck=1 +gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg + https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg +EOM + +rhsm_enable + +echo "Installing/removing packages" +ooe.sh sudo yum -y install google-compute-engine google-compute-engine-oslogin +ooe.sh sudo yum -y erase "cloud-init" "rh-amazon-rhui-client*" || true +ooe.sh sudo systemctl enable \ + google-accounts-daemon \ + google-clock-skew-daemon \ + google-instance-setup \ + google-network-daemon \ + google-shutdown-scripts \ + google-startup-scripts + +rhel_exit_handler # release subscription! + +rh_finalize + +echo "SUCCESS!" diff --git a/contrib/cirrus/packer/rhel_setup.sh b/contrib/cirrus/packer/rhel_setup.sh index d296713fc..99376fd65 100644 --- a/contrib/cirrus/packer/rhel_setup.sh +++ b/contrib/cirrus/packer/rhel_setup.sh @@ -18,24 +18,7 @@ RHSM_COMMAND $RHSM_COMMAND install_ooe -export GOPATH="$(mktemp -d)" -export RHSMCMD="$(mktemp)" - -exit_handler() { - set +ex - cd / - sudo rm -rf "$RHSMCMD" - sudo rm -rf "$GOPATH" - sudo subscription-manager remove --all - sudo subscription-manager unregister - sudo subscription-manager clean -} -trap "exit_handler" EXIT - -# Avoid logging sensitive details -echo "$RHSM_COMMAND" > "$RHSMCMD" -ooe.sh sudo bash "$RHSMCMD" -sudo rm -rf "$RHSMCMD" +rhsm_enable ooe.sh sudo yum -y erase "rh-amazon-rhui-client*" ooe.sh sudo subscription-manager repos "--disable=*" @@ -47,26 +30,12 @@ ooe.sh sudo subscription-manager repos \ ooe.sh sudo yum -y update -# Frequently needed -ooe.sh sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm - -# Required for google to manage ssh keys -sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM -[google-cloud-compute] -name=google-cloud-compute -baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-compute-el7-x86_64 -enabled=1 -gpgcheck=1 -repo_gpgcheck=1 -gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg - https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg -EOM - ooe.sh sudo yum -y install \ atomic-registries \ btrfs-progs-devel \ bzip2 \ device-mapper-devel \ + emacs-nox \ findutils \ glib2-devel \ glibc-static \ @@ -74,8 +43,6 @@ ooe.sh sudo yum -y install \ golang \ golang-github-cpuguy83-go-md2man \ golang-github-cpuguy83-go-md2man \ - google-compute-engine \ - google-compute-engine-oslogin \ gpgme-devel \ iptables \ libassuan-devel \ @@ -103,6 +70,7 @@ ooe.sh sudo yum -y install \ runc \ skopeo-containers \ unzip \ + vim \ which \ xz @@ -118,7 +86,7 @@ install_criu install_packer_copied_files -exit_handler # release subscription! +rhel_exit_handler # release subscription! rh_finalize diff --git a/contrib/cirrus/packer/ubuntu_setup.sh b/contrib/cirrus/packer/ubuntu_setup.sh index 4cf1f335b..af5671c90 100644 --- a/contrib/cirrus/packer/ubuntu_setup.sh +++ b/contrib/cirrus/packer/ubuntu_setup.sh @@ -21,11 +21,14 @@ install_ooe export GOPATH="$(mktemp -d)" trap "sudo rm -rf $GOPATH" EXIT +# Avoid getting stuck waiting for user input +export DEBIAN_FRONTEND=noninteractive + # Try twice as workaround for minor networking problems echo "Updating system and installing package dependencies" -ooe.sh sudo apt-get -qq update || sudo apt-get -qq update -ooe.sh sudo apt-get -qq upgrade || sudo apt-get -qq upgrade -ooe.sh sudo apt-get -qq install --no-install-recommends \ +ooe.sh sudo -E apt-get -qq update || sudo -E apt-get -qq update +ooe.sh sudo -E apt-get -qq upgrade || sudo -E apt-get -qq upgrade +ooe.sh sudo -E apt-get -qq install --no-install-recommends \ apparmor \ autoconf \ automake \ @@ -34,6 +37,7 @@ ooe.sh sudo apt-get -qq install --no-install-recommends \ build-essential \ curl \ e2fslibs-dev \ + emacs-nox \ gawk \ gettext \ go-md2man \ @@ -54,6 +58,8 @@ ooe.sh sudo apt-get -qq install --no-install-recommends \ libostree-dev \ libprotobuf-c0-dev \ libprotobuf-dev \ + libseccomp-dev \ + libseccomp2 \ libtool \ libudev-dev \ lsof \ @@ -71,6 +77,7 @@ ooe.sh sudo apt-get -qq install --no-install-recommends \ python3-setuptools \ socat \ unzip \ + vim \ xz-utils echo "Fixing Ubuntu kernel not enabling swap accounting by default" diff --git a/contrib/cirrus/setup_environment.sh b/contrib/cirrus/setup_environment.sh index 167db127f..174bd3daf 100755 --- a/contrib/cirrus/setup_environment.sh +++ b/contrib/cirrus/setup_environment.sh @@ -4,7 +4,6 @@ set -e source $(dirname $0)/lib.sh req_env_var " -CI $CI USER $USER HOME $HOME ENVLIB $ENVLIB @@ -16,12 +15,11 @@ CIRRUS_BUILD_ID $CIRRUS_BUILD_ID" cd "$CIRRUS_WORKING_DIR" # for clarity of initial conditions # Verify basic dependencies -for depbin in go rsync unzip sha256sum curl make +for depbin in go rsync unzip sha256sum curl make python3 git do if ! type -P "$depbin" &> /dev/null then - echo "ERROR: $depbin binary not found in $PATH" - exit 2 + echo "***** WARNING: $depbin binary not found in $PATH *****" fi done @@ -35,14 +33,15 @@ then # N/B: Single-quote items evaluated every time, double-quotes only once (right now). for envstr in \ "$MARK" \ + "export EPOCH_TEST_COMMIT=\"$CIRRUS_BASE_SHA\"" \ "export HEAD=\"$CIRRUS_CHANGE_IN_REPO\"" \ "export TRAVIS=\"1\"" \ "export GOSRC=\"$CIRRUS_WORKING_DIR\"" \ "export OS_RELEASE_ID=\"$(os_release_id)\"" \ "export OS_RELEASE_VER=\"$(os_release_ver)\"" \ - "export OS_REL_VER=\"${OS_RELEASE_ID}-${OS_RELEASE_VER}\"" \ + "export OS_REL_VER=\"$(os_release_id)-$(os_release_ver)\"" \ "export BUILT_IMAGE_SUFFIX=\"-$CIRRUS_REPO_NAME-${CIRRUS_CHANGE_IN_REPO:0:8}\"" \ - "export GOPATH=\"/go\"" \ + "export GOPATH=\"/var/tmp/go\"" \ 'export PATH="$HOME/bin:$GOPATH/bin:/usr/local/bin:$PATH"' \ 'export LD_LIBRARY_PATH="/usr/local/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"' do @@ -53,9 +52,16 @@ then # Some setup needs to vary between distros case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in ubuntu-18) + # Always install runc on Ubuntu + install_runc_from_git envstr='export BUILDTAGS="seccomp $($GOSRC/hack/btrfs_tag.sh) $($GOSRC/hack/btrfs_installed_tag.sh) $($GOSRC/hack/ostree_tag.sh) varlink exclude_graphdriver_devicemapper"' ;; - fedora-28) ;& # Continue to the next item + fedora-29) ;& # Continue to the next item + fedora-28) + RUNC="https://kojipkgs.fedoraproject.org/packages/runc/1.0.0/55.dev.git578fe65.fc${OS_RELEASE_VER}/x86_64/runc-1.0.0-55.dev.git578fe65.fc${OS_RELEASE_VER}.x86_64.rpm" + echo ">>>>> OVERRIDING RUNC WITH $RUNC <<<<<" + dnf -y install "$RUNC" + ;& # Continue to the next item centos-7) ;& rhel-7) envstr='unset BUILDTAGS' # Use default from Makefile diff --git a/contrib/cirrus/success.sh b/contrib/cirrus/success.sh index d1daf9043..2b0cf4655 100755 --- a/contrib/cirrus/success.sh +++ b/contrib/cirrus/success.sh @@ -1,22 +1,20 @@ #!/bin/bash set -e + source $(dirname $0)/lib.sh req_env_var " - CIRRUS_TASK_NAME $CIRRUS_TASK_NAME CIRRUS_BRANCH $CIRRUS_BRANCH - OS_RELEASE_ID $OS_RELEASE_ID - OS_RELEASE_VER $OS_RELEASE_VER - CIRRUS_REPO_CLONE_URL $CIRRUS_REPO_CLONE_URL + CIRRUS_BUILD_ID $CIRRUS_BUILD_ID " -REF_URL="$(echo $CIRRUS_REPO_CLONE_URL | sed 's/.git$//g')" +REF=$(basename $CIRRUS_BRANCH) # PR number or branch named +URL="https://cirrus-ci.com/build/$CIRRUS_BUILD_ID" + if [[ "$CIRRUS_BRANCH" =~ "pull" ]] then - REF_URL="$REF_URL/$CIRRUS_BRANCH" # pull request URL + ircmsg "Cirrus-CI testing successful for PR #$REF: $URL" else - REF_URL="$REF_URL/commits/$CIRRUS_BRANCH" # branch merge + ircmsg "Cirrus-CI testing branch $REF successful: $URL" fi - -ircmsg "Cirrus-CI $CIRRUS_TASK_NAME on $OS_RELEASE_ID-$OS_RELEASE_VER successful for $REF_URL" diff --git a/contrib/cirrus/verify_source.sh b/contrib/cirrus/system_test.sh index 860bafc00..66974f8c6 100755 --- a/contrib/cirrus/verify_source.sh +++ b/contrib/cirrus/system_test.sh @@ -4,11 +4,12 @@ set -e source $(dirname $0)/lib.sh req_env_var " +GOSRC $GOSRC OS_RELEASE_ID $OS_RELEASE_ID OS_RELEASE_VER $OS_RELEASE_VER " -show_env_vars +clean_env set -x cd "$GOSRC" @@ -16,15 +17,17 @@ cd "$GOSRC" case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in ubuntu-18) make install.tools "BUILDTAGS=$BUILDTAGS" - make validate "BUILDTAGS=$BUILDTAGS" - # make lint "BUILDTAGS=$BUILDTAGS" + make "BUILDTAGS=$BUILDTAGS" + make test-binaries "BUILDTAGS=$BUILDTAGS" ;; fedora-28) ;& centos-7) ;& rhel-7) make install.tools - make validate - # make lint + make + make test-binaries ;; *) bad_os_id_ver ;; esac + +make localsystem diff --git a/contrib/cirrus/unit_test.sh b/contrib/cirrus/unit_test.sh index cacc23045..61d9dc73d 100755 --- a/contrib/cirrus/unit_test.sh +++ b/contrib/cirrus/unit_test.sh @@ -9,22 +9,23 @@ OS_RELEASE_ID $OS_RELEASE_ID OS_RELEASE_VER $OS_RELEASE_VER " -show_env_vars +clean_env set -x cd "$GOSRC" case "${OS_RELEASE_ID}-${OS_RELEASE_VER}" in ubuntu-18) + make install.tools "BUILDTAGS=$BUILDTAGS" make localunit "BUILDTAGS=$BUILDTAGS" make "BUILDTAGS=$BUILDTAGS" ;; - fedora-28) + fedora-29) ;& # Continue to the next item + fedora-28) ;& + centos-7) ;& + rhel-7) + make install.tools make localunit make ;; - centos-7) ;& # Continue to the next item - rhel-7) - stub 'unit testing not working on $OS_RELEASE_ID' - ;; *) bad_os_id_ver ;; esac diff --git a/contrib/gate/Dockerfile b/contrib/gate/Dockerfile new file mode 100644 index 000000000..f9b57a6da --- /dev/null +++ b/contrib/gate/Dockerfile @@ -0,0 +1,69 @@ +FROM fedora:29 +RUN dnf -y install \ + atomic-registries \ + btrfs-progs-devel \ + buildah \ + bzip2 \ + conmon \ + container-selinux \ + containernetworking-cni \ + containernetworking-cni-devel \ + device-mapper-devel \ + findutils \ + git \ + glib2-devel \ + glibc-static \ + gnupg \ + golang \ + gpgme-devel \ + iptables \ + libassuan-devel \ + libseccomp-devel \ + libselinux-devel \ + lsof \ + make \ + nmap-ncat \ + ostree-devel \ + procps-ng \ + python \ + python3-dateutil \ + python3-psutil \ + python3-pytoml \ + python3-varlink \ + skopeo-containers \ + slirp4netns \ + rsync \ + which \ + xz \ + && dnf clean all + +ENV GOPATH="/go" \ + PATH="/go/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin" \ + SRCPATH="/usr/src/libpod" \ + GOSRC="/go/src/github.com/containers/libpod" + +# Only needed for installing build-time dependencies +COPY / $GOSRC + +WORKDIR $GOSRC + +# Install dependencies +RUN set -x && \ + go get -u github.com/mailru/easyjson/... && \ + install -D -m 755 "$GOPATH"/bin/easyjson /usr/bin/ && \ + make install.tools && \ + install -D -m 755 $GOSRC/contrib/gate/entrypoint.sh /usr/local/bin/ && \ + rm -rf "$GOSRC" + +# Install cni config +#RUN make install.cni +RUN mkdir -p /etc/cni/net.d/ +COPY cni/87-podman-bridge.conflist /etc/cni/net.d/87-podman-bridge.conflist + +# Make sure we have some policy for pulling images +RUN mkdir -p /etc/containers +COPY test/policy.json /etc/containers/policy.json +COPY test/redhat_sigstore.yaml /etc/containers/registries.d/registry.access.redhat.com.yaml + +VOLUME ["/usr/src/libpod"] +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] diff --git a/contrib/gate/README.md b/contrib/gate/README.md new file mode 100644 index 000000000..709e6035f --- /dev/null +++ b/contrib/gate/README.md @@ -0,0 +1,4 @@ + + +A standard container image for `gofmt` and lint-checking the libpod +repository. The [contributors guide contains the documentation for usage.](https://github.com/containers/libpod/blob/master/CONTRIBUTING.md#go-format-and-lint) diff --git a/contrib/gate/entrypoint.sh b/contrib/gate/entrypoint.sh new file mode 100755 index 000000000..e16094cc0 --- /dev/null +++ b/contrib/gate/entrypoint.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +[[ -n "$SRCPATH" ]] || \ + ( echo "ERROR: \$SRCPATH must be non-empty" && exit 1 ) +[[ -n "$GOSRC" ]] || \ + ( echo "ERROR: \$GOSRC must be non-empty" && exit 2 ) +[[ -r "${SRCPATH}/contrib/gate/Dockerfile" ]] || \ + ( echo "ERROR: Expecting libpod repository root at $SRCPATH" && exit 3 ) + +# Working from a copy avoids needing to perturb the actual source files +mkdir -p "$GOSRC" +/usr/bin/rsync --recursive --links --quiet --safe-links \ + --perms --times "${SRCPATH}/" "${GOSRC}/" +cd "$GOSRC" +make "$@" diff --git a/contrib/python/podman/Makefile b/contrib/python/podman/Makefile index 6ec4159f2..0cbfe2fb3 100644 --- a/contrib/python/podman/Makefile +++ b/contrib/python/podman/Makefile @@ -1,9 +1,10 @@ PYTHON ?= $(shell command -v python3 2>/dev/null || command -v python) DESTDIR ?= / -PODMAN_VERSION ?= '0.0.4' +PODMAN_VERSION ?= '0.11.1.1' .PHONY: python-podman python-podman: + PODMAN_VERSION=$(PODMAN_VERSION) \ $(PYTHON) setup.py sdist bdist .PHONY: lint @@ -16,12 +17,13 @@ integration: .PHONY: install install: + PODMAN_VERSION=$(PODMAN_VERSION) \ $(PYTHON) setup.py install --root ${DESTDIR} .PHONY: upload upload: - $(PODMAN_VERSION) $(PYTHON) setup.py sdist bdist_wheel - twine upload --repository-url https://test.pypi.org/legacy/ dist/* + PODMAN_VERSION=$(PODMAN_VERSION) $(PYTHON) setup.py sdist bdist_wheel + twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/* .PHONY: clobber clobber: uninstall clean diff --git a/contrib/python/podman/podman/libs/_containers_attach.py b/contrib/python/podman/podman/libs/_containers_attach.py index f2dad573b..94247d349 100644 --- a/contrib/python/podman/podman/libs/_containers_attach.py +++ b/contrib/python/podman/podman/libs/_containers_attach.py @@ -19,9 +19,13 @@ class Mixin: """ if stdin is None: stdin = sys.stdin.fileno() + elif hasattr(stdin, 'fileno'): + stdin = stdin.fileno() if stdout is None: stdout = sys.stdout.fileno() + elif hasattr(stdout, 'fileno'): + stdout = stdout.fileno() with self._client() as podman: attach = podman.GetAttachSockets(self._id) @@ -49,7 +53,7 @@ class Mixin: def resize_handler(self): """Send the new window size to conmon.""" - def wrapped(signum, frame): + def wrapped(signum, frame): # pylint: disable=unused-argument packed = fcntl.ioctl(self.pseudo_tty.stdout, termios.TIOCGWINSZ, struct.pack('HHHH', 0, 0, 0, 0)) rows, cols, _, _ = struct.unpack('HHHH', packed) @@ -67,7 +71,7 @@ class Mixin: def log_handler(self): """Send command to reopen log to conmon.""" - def wrapped(signum, frame): + def wrapped(signum, frame): # pylint: disable=unused-argument with open(self.pseudo_tty.control_socket, 'w') as skt: # send conmon reopen log message skt.write('2\n') diff --git a/contrib/python/podman/podman/libs/containers.py b/contrib/python/podman/podman/libs/containers.py index e211a284e..7adecea8f 100644 --- a/contrib/python/podman/podman/libs/containers.py +++ b/contrib/python/podman/podman/libs/containers.py @@ -1,12 +1,12 @@ """Models for manipulating containers and storage.""" import collections -import functools import getpass import json import logging import signal import time +from . import fold_keys from ._containers_attach import Mixin as AttachMixin from ._containers_start import Mixin as StartMixin @@ -14,25 +14,27 @@ from ._containers_start import Mixin as StartMixin class Container(AttachMixin, StartMixin, collections.UserDict): """Model for a container.""" - def __init__(self, client, id, data): + def __init__(self, client, ident, data, refresh=True): """Construct Container Model.""" super(Container, self).__init__(data) - self._client = client - self._id = id + self._id = ident - with client() as podman: - self._refresh(podman) + if refresh: + with client() as podman: + self._refresh(podman) + else: + for k, v in self.data.items(): + setattr(self, k, v) + if 'containerrunning' in self.data: + setattr(self, 'running', self.data['containerrunning']) + self.data['running'] = self.data['containerrunning'] assert self._id == data['id'],\ 'Requested container id({}) does not match store id({})'.format( self._id, data['id'] ) - def __getitem__(self, key): - """Get items from parent dict.""" - return super().__getitem__(key) - def _refresh(self, podman, tries=1): try: ctnr = podman.GetContainer(self._id) @@ -71,18 +73,18 @@ class Container(AttachMixin, StartMixin, collections.UserDict): results = podman.ListContainerChanges(self._id) return results['container'] - def kill(self, signal=signal.SIGTERM, wait=25): + def kill(self, sig=signal.SIGTERM, wait=25): """Send signal to container. default signal is signal.SIGTERM. wait n of seconds, 0 waits forever. """ with self._client() as podman: - podman.KillContainer(self._id, signal) + podman.KillContainer(self._id, sig) timeout = time.time() + wait while True: self._refresh(podman) - if self.status != 'running': + if self.status != 'running': # pylint: disable=no-member return self if wait and timeout < time.time(): @@ -90,20 +92,11 @@ class Container(AttachMixin, StartMixin, collections.UserDict): time.sleep(0.5) - def _lower_hook(self): - """Convert all keys to lowercase.""" - - @functools.wraps(self._lower_hook) - def wrapped(input_): - return {k.lower(): v for (k, v) in input_.items()} - - return wrapped - def inspect(self): """Retrieve details about containers.""" with self._client() as podman: results = podman.InspectContainer(self._id) - obj = json.loads(results['container'], object_hook=self._lower_hook()) + obj = json.loads(results['container'], object_hook=fold_keys()) return collections.namedtuple('ContainerInspect', obj.keys())(**obj) def export(self, target): @@ -115,19 +108,16 @@ class Container(AttachMixin, StartMixin, collections.UserDict): results = podman.ExportContainer(self._id, target) return results['tarfile'] - def commit(self, - image_name, - *args, - changes=[], - message='', - pause=True, - **kwargs): + def commit(self, image_name, **kwargs): """Create image from container. - All changes overwrite existing values. - See inspect() to obtain current settings. + Keyword arguments: + author -- change image's author + message -- change image's message, docker format only. + pause -- pause container during commit + change -- Additional properties to change - Changes: + Change examples: CMD=/usr/bin/zsh ENTRYPOINT=/bin/sh date ENV=TEST=test_containers.TestContainers.test_commit @@ -136,21 +126,23 @@ class Container(AttachMixin, StartMixin, collections.UserDict): USER=bozo:circus VOLUME=/data WORKDIR=/data/application + + All changes overwrite existing values. + See inspect() to obtain current settings. """ - # TODO: Clean up *args, **kwargs after Commit() is complete - try: - author = kwargs.get('author', getpass.getuser()) - except Exception: # pylint: disable=broad-except - author = '' + author = kwargs.get('author', None) or getpass.getuser() + change = kwargs.get('change', None) or [] + message = kwargs.get('message', None) or '' + pause = kwargs.get('pause', None) or True - for c in changes: + for c in change: if c.startswith('LABEL=') and c.count('=') < 2: raise ValueError( 'LABEL should have the format: LABEL=label=value, not {}'. format(c)) with self._client() as podman: - results = podman.Commit(self._id, image_name, changes, author, + results = podman.Commit(self._id, image_name, change, author, message, pause) return results['image'] @@ -175,7 +167,7 @@ class Container(AttachMixin, StartMixin, collections.UserDict): podman.RestartContainer(self._id, timeout) return self._refresh(podman) - def rename(self, target): + def rename(self, target): # pylint: disable=unused-argument """Rename container, return id on success.""" with self._client() as podman: # TODO: Need arguments @@ -183,7 +175,7 @@ class Container(AttachMixin, StartMixin, collections.UserDict): # TODO: fixup objects cached information return results['container'] - def resize_tty(self, width, height): + def resize_tty(self, width, height): # pylint: disable=unused-argument """Resize container tty.""" with self._client() as podman: # TODO: magic re: attach(), arguments @@ -201,7 +193,8 @@ class Container(AttachMixin, StartMixin, collections.UserDict): podman.UnpauseContainer(self._id) return self._refresh(podman) - def update_container(self, *args, **kwargs): + def update_container(self, *args, **kwargs): \ + # pylint: disable=unused-argument """TODO: Update container..., return id on success.""" with self._client() as podman: podman.UpdateContainer() @@ -220,7 +213,7 @@ class Container(AttachMixin, StartMixin, collections.UserDict): obj = results['container'] return collections.namedtuple('StatDetail', obj.keys())(**obj) - def logs(self, *args, **kwargs): + def logs(self, *args, **kwargs): # pylint: disable=unused-argument """Retrieve container logs.""" with self._client() as podman: results = podman.GetContainerLogs(self._id) @@ -239,7 +232,7 @@ class Containers(): with self._client() as podman: results = podman.ListContainers() for cntr in results['containers']: - yield Container(self._client, cntr['id'], cntr) + yield Container(self._client, cntr['id'], cntr, refresh=False) def delete_stopped(self): """Delete all stopped containers.""" diff --git a/contrib/python/podman/podman/libs/images.py b/contrib/python/podman/podman/libs/images.py index 9453fb416..ae1b86390 100644 --- a/contrib/python/podman/podman/libs/images.py +++ b/contrib/python/podman/podman/libs/images.py @@ -27,9 +27,10 @@ class Image(collections.UserDict): @staticmethod def _split_token(values=None, sep='='): + if not values: + return {} return { - k: v1 - for k, v1 in (v0.split(sep, 1) for v0 in values if values) + k: v1 for k, v1 in (v0.split(sep, 1) for v0 in values) } def create(self, *args, **kwargs): @@ -74,7 +75,7 @@ class Image(collections.UserDict): obj = json.loads(results['image'], object_hook=fold_keys()) return collections.namedtuple('ImageInspect', obj.keys())(**obj) - def push(self, target, tlsverify=False): + def push(self, target, tlsverify=True): """Copy image to target, return id on success.""" with self._client() as podman: results = podman.PushImage(self._id, target, tlsverify) diff --git a/contrib/python/podman/test/test_containers.py b/contrib/python/podman/test/test_containers.py index 3de1e54bc..a7a6ac304 100644 --- a/contrib/python/podman/test/test_containers.py +++ b/contrib/python/podman/test/test_containers.py @@ -152,7 +152,7 @@ class TestContainers(PodmanTestCase): changes.append('WORKDIR=/data/application') id = self.alpine_ctnr.commit( - 'alpine3', author='Bozo the clown', changes=changes, pause=True) + 'alpine3', author='Bozo the clown', change=changes, pause=True) img = self.pclient.images.get(id) self.assertIsNotNone(img) diff --git a/contrib/python/podman/test/test_images.py b/contrib/python/podman/test/test_images.py index f97e13b4c..45f0a2964 100644 --- a/contrib/python/podman/test/test_images.py +++ b/contrib/python/podman/test/test_images.py @@ -102,7 +102,7 @@ class TestImages(PodmanTestCase): def test_push(self): path = '{}/alpine_push'.format(self.tmpdir) target = 'dir:{}'.format(path) - self.alpine_image.push(target) + self.alpine_image.push(target, tlsverify=False) self.assertTrue(os.path.isfile(os.path.join(path, 'manifest.json'))) self.assertTrue(os.path.isfile(os.path.join(path, 'version'))) diff --git a/contrib/python/podman/test/test_runner.sh b/contrib/python/podman/test/test_runner.sh index bf097e2b2..651b2e74f 100755 --- a/contrib/python/podman/test/test_runner.sh +++ b/contrib/python/podman/test/test_runner.sh @@ -41,6 +41,7 @@ export TMPDIR=`mktemp -d /tmp/podman.XXXXXXXXXX` trap "cleanup $TMPDIR" EXIT function umount { + set +xeuo pipefail # xargs -r always ran once, so write any mount points to file first mount |awk "/$1/"' { print $3 }' >${TMPDIR}/mounts if [[ -s ${TMPDIR}/mounts ]]; then diff --git a/contrib/python/pypodman/Makefile b/contrib/python/pypodman/Makefile index cd0fcf1de..230eee44d 100644 --- a/contrib/python/pypodman/Makefile +++ b/contrib/python/pypodman/Makefile @@ -1,9 +1,10 @@ PYTHON ?= $(shell command -v python3 2>/dev/null || command -v python) DESTDIR := / -PODMAN_VERSION ?= '0.0.4' +PODMAN_VERSION ?= '0.11.1.1' .PHONY: python-pypodman python-pypodman: + PODMAN_VERSION=$(PODMAN_VERSION) \ $(PYTHON) setup.py sdist bdist .PHONY: lint @@ -16,11 +17,12 @@ integration: .PHONY: install install: + PODMAN_VERSION=$(PODMAN_VERSION) \ $(PYTHON) setup.py install --root ${DESTDIR} .PHONY: upload upload: - $(PODMAN_VERSION) $(PYTHON) setup.py sdist bdist_wheel + PODMAN_VERSION=$(PODMAN_VERSION) $(PYTHON) setup.py sdist bdist_wheel twine upload --repository-url https://test.pypi.org/legacy/ dist/* .PHONY: clobber diff --git a/contrib/python/pypodman/docs/man1/pypodman.1 b/contrib/python/pypodman/docs/man1/pypodman.1 index 09acb205b..45472dab0 100644 --- a/contrib/python/pypodman/docs/man1/pypodman.1 +++ b/contrib/python/pypodman/docs/man1/pypodman.1 @@ -85,7 +85,7 @@ overwriting earlier. Any missing items are ignored. .IP \[bu] 2 From \f[C]\-\-config\-home\f[] command line option + \f[C]pypodman/pypodman.conf\f[] .IP \[bu] 2 -From environment variable, for example: RUN_DIR +From environment variable prefixed with PODMAN_, for example: PODMAN_RUN_DIR .IP \[bu] 2 From command line option, for example: \[en]run\-dir .PP diff --git a/contrib/python/pypodman/pypodman/lib/__init__.py b/contrib/python/pypodman/pypodman/lib/__init__.py index be1b5f467..d9a434254 100644 --- a/contrib/python/pypodman/pypodman/lib/__init__.py +++ b/contrib/python/pypodman/pypodman/lib/__init__.py @@ -3,18 +3,17 @@ import sys import podman from pypodman.lib.action_base import AbstractActionBase -from pypodman.lib.parser_actions import (BooleanAction, BooleanValidate, - ChangeAction, PathAction, - PositiveIntAction, UnitAction) +from pypodman.lib.parser_actions import (ChangeAction, PathAction, + PositiveIntAction, SignalAction, + UnitAction) from pypodman.lib.podman_parser import PodmanArgumentParser from pypodman.lib.report import Report, ReportColumn # Silence pylint overlording... -assert BooleanAction -assert BooleanValidate assert ChangeAction assert PathAction assert PositiveIntAction +assert SignalAction assert UnitAction __all__ = [ diff --git a/contrib/python/pypodman/pypodman/lib/action_base.py b/contrib/python/pypodman/pypodman/lib/action_base.py index a950c362b..5cba7ac5c 100644 --- a/contrib/python/pypodman/pypodman/lib/action_base.py +++ b/contrib/python/pypodman/pypodman/lib/action_base.py @@ -17,29 +17,21 @@ class AbstractActionBase(abc.ABC): Use set_defaults() to set attributes "class_" and "method". These will be invoked as class_(parsed_args).method() """ - parent.add_argument( + parent.add_flag( '--all', - action='store_true', - help=('list all items.' - ' (default: no-op, included for compatibility.)')) - parent.add_argument( - '--no-trunc', - '--notruncate', - action='store_false', - dest='truncate', + help='list all items.') + parent.add_flag( + '--truncate', + '--trunc', default=True, - help='Display extended information. (default: False)') - parent.add_argument( - '--noheading', - action='store_false', - dest='heading', + help="Truncate id's and other long fields.") + parent.add_flag( + '--heading', default=True, - help=('Omit the table headings from the output.' - ' (default: False)')) - parent.add_argument( + help='Include table headings in the output.') + parent.add_flag( '--quiet', - action='store_true', - help='List only the IDs. (default: %(default)s)') + help='List only the IDs.') def __init__(self, args): """Construct class.""" diff --git a/contrib/python/pypodman/pypodman/lib/actions/__init__.py b/contrib/python/pypodman/pypodman/lib/actions/__init__.py index 2668cd8ff..c0d77ddb1 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/__init__.py +++ b/contrib/python/pypodman/pypodman/lib/actions/__init__.py @@ -22,6 +22,8 @@ from pypodman.lib.actions.rm_action import Rm from pypodman.lib.actions.rmi_action import Rmi from pypodman.lib.actions.run_action import Run from pypodman.lib.actions.search_action import Search +from pypodman.lib.actions.start_action import Start +from pypodman.lib.actions.version_action import Version __all__ = [ 'Attach', @@ -47,4 +49,6 @@ __all__ = [ 'Rmi', 'Run', 'Search', + 'Start', + 'Version', ] diff --git a/contrib/python/pypodman/pypodman/lib/actions/_create_args.py b/contrib/python/pypodman/pypodman/lib/actions/_create_args.py index 207f52796..8ab4292e8 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/_create_args.py +++ b/contrib/python/pypodman/pypodman/lib/actions/_create_args.py @@ -1,6 +1,6 @@ """Implement common create container arguments together.""" -from pypodman.lib import BooleanAction, UnitAction +from pypodman.lib import SignalAction, UnitAction class CreateArguments(): @@ -108,11 +108,9 @@ class CreateArguments(): metavar='NODES', help=('Memory nodes (MEMs) in which to allow execution (0-3, 0,1).' ' Only effective on NUMA systems')) - parser.add_argument( + parser.add_flag( '--detach', '-d', - action=BooleanAction, - default=False, help='Detached mode: run the container in the background and' ' print the new container ID. (default: False)') parser.add_argument( @@ -218,7 +216,7 @@ class CreateArguments(): # only way for argparse to handle these options. vol_args = { - 'choices': ['bind', 'tmpfs', 'ignore'], + 'choices': ('bind', 'tmpfs', 'ignore'), 'metavar': 'MODE', 'type': str.lower, 'help': 'Tells podman how to handle the builtin image volumes', @@ -228,12 +226,10 @@ class CreateArguments(): volume_group.add_argument('--image-volume', **vol_args) volume_group.add_argument('--builtin-volume', **vol_args) - parser.add_argument( + parser.add_flag( '--interactive', '-i', - action=BooleanAction, - default=False, - help='Keep STDIN open even if not attached. (default: False)') + help='Keep STDIN open even if not attached.') parser.add_argument('--ipc', help='Create namespace') parser.add_argument( '--kernel-memory', action=UnitAction, help='Kernel memory limit') @@ -278,10 +274,9 @@ class CreateArguments(): metavar='BRIDGE', help='Set the Network mode for the container.' ' (format: bridge, host, container:UUID, ns:PATH, none)') - parser.add_argument( + parser.add_flag( '--oom-kill-disable', - action=BooleanAction, - help='Whether to disable OOM Killer for the container or not') + help='Whether to disable OOM Killer for the container or not.') parser.add_argument( '--oom-score-adj', choices=range(-1000, 1000), @@ -298,41 +293,33 @@ class CreateArguments(): help=("Tune the container's pids limit." " Set -1 to have unlimited pids for the container.")) parser.add_argument('--pod', help='Run container in an existing pod') - parser.add_argument( + parser.add_flag( '--privileged', - action=BooleanAction, help='Give extended privileges to this container.') parser.add_argument( '--publish', '-p', metavar='RANGE', help="Publish a container's port, or range of ports, to the host") - parser.add_argument( + parser.add_flag( '--publish-all', '-P', - action=BooleanAction, help='Publish all exposed ports to random' - ' ports on the host interfaces' - '(default: False)') - parser.add_argument( + ' ports on the host interfaces.') + parser.add_flag( '--quiet', '-q', - action='store_true', help='Suppress output information when pulling images') - parser.add_argument( + parser.add_flag( '--read-only', - action=BooleanAction, help="Mount the container's root filesystem as read only.") - parser.add_argument( + parser.add_flag( '--rm', - action=BooleanAction, - default=False, help='Automatically remove the container when it exits.') parser.add_argument( '--rootfs', - action='store_true', - help=('If specified, the first argument refers to an' - ' exploded container on the file system of remote host.')) + help='If specified, the first argument refers to an' + ' exploded container on the file system of remote host.') parser.add_argument( '--security-opt', action='append', @@ -340,15 +327,14 @@ class CreateArguments(): help='Set security options.') parser.add_argument( '--shm-size', action=UnitAction, help='Size of /dev/shm') - parser.add_argument( + parser.add_flag( '--sig-proxy', - action=BooleanAction, - default=True, help='Proxy signals sent to the podman run' ' command to the container process') parser.add_argument( '--stop-signal', - metavar='SIGTERM', + action=SignalAction, + default='TERM', help='Signal to stop a container') parser.add_argument( '--stop-timeout', @@ -374,11 +360,9 @@ class CreateArguments(): metavar='MOUNT', help='Create a tmpfs mount.' ' (default: rw,noexec,nosuid,nodev,size=65536k.)') - parser.add_argument( + parser.add_flag( '--tty', '-t', - action=BooleanAction, - default=False, help='Allocate a pseudo-TTY for standard input of container.') parser.add_argument( '--uidmap', @@ -394,15 +378,16 @@ class CreateArguments(): parser.add_argument( '--user', '-u', - help=('Sets the username or UID used and optionally' - ' the groupname or GID for the specified command.')) + help='Sets the username or UID used and optionally' + ' the groupname or GID for the specified command.') parser.add_argument( '--userns', metavar='NAMESPACE', help='Set the user namespace mode for the container') parser.add_argument( '--uts', - choices=['host', 'ns'], + choices=('host', 'ns'), + type=str.lower, help='Set the UTS mode for the container') parser.add_argument('--volume', '-v', help='Create a bind mount.') parser.add_argument( diff --git a/contrib/python/pypodman/pypodman/lib/actions/commit_action.py b/contrib/python/pypodman/pypodman/lib/actions/commit_action.py index 21665ad0b..c166e1aff 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/commit_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/commit_action.py @@ -2,7 +2,7 @@ import sys import podman -from pypodman.lib import AbstractActionBase, BooleanAction, ChangeAction +from pypodman.lib import AbstractActionBase, ChangeAction class Commit(AbstractActionBase): @@ -30,7 +30,8 @@ class Commit(AbstractActionBase): choices=('oci', 'docker'), default='oci', type=str.lower, - help='Set the format of the image manifest and metadata', + help='Set the format of the image manifest and metadata.' + ' (Ignored.)', ) parser.add_argument( '--iidfile', @@ -40,19 +41,17 @@ class Commit(AbstractActionBase): parser.add_argument( '--message', '-m', - help='Set commit message for committed image', + help='Set commit message for committed image' + ' (Only on docker images.)', ) - parser.add_argument( + parser.add_flag( '--pause', '-p', - action=BooleanAction, - default=True, help='Pause the container when creating an image', ) - parser.add_argument( + parser.add_flag( '--quiet', '-q', - action='store_true', help='Suppress output', ) parser.add_argument( @@ -80,8 +79,16 @@ class Commit(AbstractActionBase): flush=True) return 1 else: - ident = ctnr.commit(self.opts['image'][0], **self.opts) - print(ident) + ident = ctnr.commit( + self.opts['image'][0], + change=self.opts.get('change', None), + message=self.opts.get('message', None), + pause=self.opts['pause'], + author=self.opts.get('author', None), + ) + + if not self.opts['quiet']: + print(ident) except podman.ErrorOccurred as e: sys.stdout.flush() print( diff --git a/contrib/python/pypodman/pypodman/lib/actions/create_action.py b/contrib/python/pypodman/pypodman/lib/actions/create_action.py index d9631763a..26a312bb1 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/create_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/create_action.py @@ -21,7 +21,7 @@ class Create(AbstractActionBase): parser.add_argument('image', nargs=1, help='source image id') parser.add_argument( 'command', - nargs='*', + nargs=parent.REMAINDER, help='command and args to run.', ) parser.set_defaults(class_=cls, method='create') diff --git a/contrib/python/pypodman/pypodman/lib/actions/history_action.py b/contrib/python/pypodman/pypodman/lib/actions/history_action.py index f9aaa54f6..76c3ad756 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/history_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/history_action.py @@ -5,8 +5,7 @@ from collections import OrderedDict import humanize import podman -from pypodman.lib import (AbstractActionBase, BooleanAction, Report, - ReportColumn) +from pypodman.lib import AbstractActionBase, Report, ReportColumn class History(AbstractActionBase): @@ -17,13 +16,10 @@ class History(AbstractActionBase): """Add History command to parent parser.""" parser = parent.add_parser('history', help='report image history') super().subparser(parser) - parser.add_argument( + parser.add_flag( '--human', '-H', - action=BooleanAction, - default='True', - help='Display sizes and dates in human readable format.' - ' (default: %(default)s)') + help='Display sizes and dates in human readable format.') parser.add_argument( '--format', choices=('json', 'table'), diff --git a/contrib/python/pypodman/pypodman/lib/actions/images_action.py b/contrib/python/pypodman/pypodman/lib/actions/images_action.py index 29bf90dd2..21376eeeb 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/images_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/images_action.py @@ -24,11 +24,9 @@ class Images(AbstractActionBase): help=('Change sort ordered of displayed images.' ' (default: %(default)s)')) - group = parser.add_mutually_exclusive_group() - group.add_argument( + parser.add_flag( '--digests', - action='store_true', - help='Include digests with images. (default: %(default)s)') + help='Include digests with images.') parser.set_defaults(class_=cls, method='list') def __init__(self, args): diff --git a/contrib/python/pypodman/pypodman/lib/actions/info_action.py b/contrib/python/pypodman/pypodman/lib/actions/info_action.py index 988284541..3c854a358 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/info_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/info_action.py @@ -22,10 +22,6 @@ class Info(AbstractActionBase): " (default: yaml)") parser.set_defaults(class_=cls, method='info') - def __init__(self, args): - """Construct Info class.""" - super().__init__(args) - def info(self): """Report on Podman Service.""" try: diff --git a/contrib/python/pypodman/pypodman/lib/actions/inspect_action.py b/contrib/python/pypodman/pypodman/lib/actions/inspect_action.py index 514b4702a..ca5ad2215 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/inspect_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/inspect_action.py @@ -22,12 +22,9 @@ class Inspect(AbstractActionBase): type=str.lower, help='Type of object to inspect', ) - parser.add_argument( - 'size', - action='store_true', - default=True, - help='Display the total file size if the type is a container.' - ' Always True.') + parser.add_flag( + '--size', + help='Display the total file size if the type is a container.') parser.add_argument( 'objects', nargs='+', @@ -35,10 +32,6 @@ class Inspect(AbstractActionBase): ) parser.set_defaults(class_=cls, method='inspect') - def __init__(self, args): - """Construct Inspect class.""" - super().__init__(args) - def _get_container(self, ident): try: logging.debug("Getting container %s", ident) @@ -59,7 +52,7 @@ class Inspect(AbstractActionBase): def inspect(self): """Inspect provided podman objects.""" - output = {} + output = [] try: for ident in self._args.objects: obj = None @@ -78,7 +71,13 @@ class Inspect(AbstractActionBase): msg = 'Object "{}" not found'.format(ident) print(msg, file=sys.stderr, flush=True) else: - output.update(obj._asdict()) + fields = obj._asdict() + if not self._args.size: + try: + del fields['sizerootfs'] + except KeyError: + pass + output.append(fields) except podman.ErrorOccurred as e: sys.stdout.flush() print( diff --git a/contrib/python/pypodman/pypodman/lib/actions/kill_action.py b/contrib/python/pypodman/pypodman/lib/actions/kill_action.py index cb3d3f035..e8fb4e74d 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/kill_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/kill_action.py @@ -1,9 +1,8 @@ """Remote client command for signaling podman containers.""" -import signal import sys import podman -from pypodman.lib import AbstractActionBase +from pypodman.lib import AbstractActionBase, SignalAction class Kill(AbstractActionBase): @@ -16,10 +15,9 @@ class Kill(AbstractActionBase): parser.add_argument( '--signal', '-s', - choices=range(1, signal.NSIG), - metavar='[1,{}]'.format(signal.NSIG), + action=SignalAction, default=9, - help='Signal to send to the container. (default: 9)') + help='Signal to send to the container. (default: %(default)s)') parser.add_argument( 'containers', nargs='+', @@ -27,10 +25,6 @@ class Kill(AbstractActionBase): ) parser.set_defaults(class_=cls, method='kill') - def __init__(self, args): - """Construct Kill class.""" - super().__init__(args) - def kill(self): """Signal provided containers.""" try: diff --git a/contrib/python/pypodman/pypodman/lib/actions/pause_action.py b/contrib/python/pypodman/pypodman/lib/actions/pause_action.py index ab64d8b81..7dc02f7fe 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pause_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pause_action.py @@ -19,10 +19,6 @@ class Pause(AbstractActionBase): ) parser.set_defaults(class_=cls, method='pause') - def __init__(self, args): - """Construct Pause class.""" - super().__init__(args) - def pause(self): """Pause provided containers.""" try: diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/create_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/create_parser.py index 46c1e3e51..4e0bde777 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/create_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/create_parser.py @@ -2,7 +2,7 @@ import sys import podman -from pypodman.lib import AbstractActionBase, BooleanAction +from pypodman.lib import AbstractActionBase class CreatePod(AbstractActionBase): @@ -20,12 +20,9 @@ class CreatePod(AbstractActionBase): type=str, help='Path to cgroups under which the' ' cgroup for the pod will be created.') - parser.add_argument( + parser.add_flag( '--infra', - action=BooleanAction, - default=True, - help='Create an infra container and associate it with the pod' - '(default: %(default)s)') + help='Create an infra container and associate it with the pod.') parser.add_argument( '-l', '--label', diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/kill_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/kill_parser.py index 430ec34e0..9b6229939 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/kill_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/kill_parser.py @@ -3,7 +3,7 @@ import signal import sys import podman -from pypodman.lib import AbstractActionBase +from pypodman.lib import AbstractActionBase, SignalAction from pypodman.lib import query_model as query_pods @@ -15,18 +15,16 @@ class KillPod(AbstractActionBase): """Add Pod Kill command to parent parser.""" parser = parent.add_parser('kill', help='signal containers in pod') - parser.add_argument( - '-a', + parser.add_flag( '--all', - action='store_true', - help='Sends signal to all pods') + '-a', + help='Sends signal to all pods.') parser.add_argument( '-s', '--signal', - choices=range(1, signal.NSIG), - metavar='[1,{}]'.format(signal.NSIG), + action=SignalAction, default=9, - help='Signal to send to the pod. (default: 9)') + help='Signal to send to the pod. (default: %(default)s)') parser.add_argument('pod', nargs='*', help='pod(s) to signal') parser.set_defaults(class_=cls, method='kill') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/pause_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/pause_parser.py index daae028d4..c751314ca 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/pause_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/pause_parser.py @@ -13,8 +13,10 @@ class PausePod(AbstractActionBase): def subparser(cls, parent): """Add Pod Pause command to parent parser.""" parser = parent.add_parser('pause', help='pause containers in pod') - parser.add_argument( - '-a', '--all', action='store_true', help='Pause all pods') + parser.add_flag( + '--all', + '-a', + help='Pause all pods.') parser.add_argument('pod', nargs='*', help='pod(s) to pause.') parser.set_defaults(class_=cls, method='pause') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/processes_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/processes_parser.py index ecfcb883a..855e313c7 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/processes_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/processes_parser.py @@ -14,18 +14,15 @@ class ProcessesPod(AbstractActionBase): parser = parent.add_parser('ps', help='list processes of pod') super().subparser(parser) - parser.add_argument( + parser.add_flag( '--ctr-names', - action='store_true', - help='Include container name in the info field') - parser.add_argument( + help='Include container name in the info field.') + parser.add_flag( '--ctr-ids', - action='store_true', - help='Include container ID in the info field') - parser.add_argument( + help='Include container ID in the info field.') + parser.add_flag( '--ctr-status', - action='store_true', - help='Include container status in the info field') + help='Include container status in the info field.') parser.add_argument( '--format', choices=('json'), diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/remove_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/remove_parser.py index 40eeb7203..289325d14 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/remove_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/remove_parser.py @@ -13,13 +13,14 @@ class RemovePod(AbstractActionBase): def subparser(cls, parent): """Add Pod Rm command to parent parser.""" parser = parent.add_parser('rm', help='Delete pod and container(s)') - parser.add_argument( - '-a', '--all', action='store_true', help='Remove all pods') - parser.add_argument( - '-f', + parser.add_flag( + '--all', + '-a', + help='Remove all pods.') + parser.add_flag( '--force', - action='store_true', - help='Stop and remove container(s) then delete pod') + '-f', + help='Stop and remove container(s) then delete pod.') parser.add_argument( 'pod', nargs='*', help='Pod to remove. Or, use --all') parser.set_defaults(class_=cls, method='remove') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/restart_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/restart_parser.py index af489ad28..53f45b6de 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/restart_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/restart_parser.py @@ -13,8 +13,10 @@ class RestartPod(AbstractActionBase): def subparser(cls, parent): """Add Pod Restart command to parent parser.""" parser = parent.add_parser('restart', help='restart containers in pod') - parser.add_argument( - '-a', '--all', action='store_true', help='Restart all pods') + parser.add_flag( + '--all', + '-a', + help='Restart all pods.') parser.add_argument( 'pod', nargs='*', help='Pod to restart. Or, use --all') parser.set_defaults(class_=cls, method='restart') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/start_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/start_parser.py index 0ddc336bf..ff62b839e 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/start_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/start_parser.py @@ -14,8 +14,10 @@ class StartPod(AbstractActionBase): def subparser(cls, parent): """Add Pod Start command to parent parser.""" parser = parent.add_parser('start', help='start pod') - parser.add_argument( - '-a', '--all', action='store_true', help='Start all pods') + parser.add_flag( + '--all', + '-a', + help='Start all pods.') parser.add_argument( 'pod', nargs='*', help='Pod to start. Or, use --all') parser.set_defaults(class_=cls, method='start') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/stop_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/stop_parser.py index 7054fd38a..cbf2bf1e7 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/stop_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/stop_parser.py @@ -13,8 +13,10 @@ class StopPod(AbstractActionBase): def subparser(cls, parent): """Add Pod Stop command to parent parser.""" parser = parent.add_parser('stop', help='stop pod') - parser.add_argument( - '-a', '--all', action='store_true', help='Stop all pods') + parser.add_flag( + '--all', + '-a', + help='Stop all pods.') parser.add_argument( 'pod', nargs='*', help='Pod to stop. Or, use --all') parser.set_defaults(class_=cls, method='stop') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod/unpause_parser.py b/contrib/python/pypodman/pypodman/lib/actions/pod/unpause_parser.py index 90e1ddbe2..5186cf9cc 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod/unpause_parser.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod/unpause_parser.py @@ -13,8 +13,10 @@ class UnpausePod(AbstractActionBase): def subparser(cls, parent): """Add Pod Unpause command to parent parser.""" parser = parent.add_parser('unpause', help='unpause pod') - parser.add_argument( - '-a', '--all', action='store_true', help='Unpause all pods') + parser.add_flag( + '--all', + '-a', + help='Unpause all pods.') parser.add_argument( 'pod', nargs='*', help='Pod to unpause. Or, use --all') parser.set_defaults(class_=cls, method='unpause') diff --git a/contrib/python/pypodman/pypodman/lib/actions/pod_action.py b/contrib/python/pypodman/pypodman/lib/actions/pod_action.py index 046af34bb..4b8997a05 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/pod_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/pod_action.py @@ -5,6 +5,8 @@ import sys from pypodman.lib import AbstractActionBase +# pylint: disable=wildcard-import +# pylint: disable=unused-wildcard-import from .pod import * diff --git a/contrib/python/pypodman/pypodman/lib/actions/port_action.py b/contrib/python/pypodman/pypodman/lib/actions/port_action.py index d2a8ded46..6913f3813 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/port_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/port_action.py @@ -13,16 +13,13 @@ class Port(AbstractActionBase): """Add Port command to parent parser.""" parser = parent.add_parser( 'port', help='retrieve ports from containers') - parser.add_argument( + parser.add_flag( '--all', '-a', - action='store_true', - default=False, help='List all known port mappings for running containers') parser.add_argument( 'containers', nargs='*', - default=None, help='containers to list ports', ) parser.set_defaults(class_=cls, method='port') @@ -61,3 +58,4 @@ class Port(AbstractActionBase): file=sys.stderr, flush=True) return 1 + return 0 diff --git a/contrib/python/pypodman/pypodman/lib/actions/ps_action.py b/contrib/python/pypodman/pypodman/lib/actions/ps_action.py index cd7a7947d..62ceb2e67 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/ps_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/ps_action.py @@ -16,6 +16,7 @@ class Ps(AbstractActionBase): """Add Images command to parent parser.""" parser = parent.add_parser('ps', help='list containers') super().subparser(parser) + parser.add_argument( '--sort', choices=('createdat', 'id', 'image', 'names', 'runningfor', 'size', @@ -32,9 +33,9 @@ class Ps(AbstractActionBase): self.columns = OrderedDict({ 'id': - ReportColumn('id', 'CONTAINER ID', 14), + ReportColumn('id', 'CONTAINER ID', 12), 'image': - ReportColumn('image', 'IMAGE', 30), + ReportColumn('image', 'IMAGE', 31), 'command': ReportColumn('column', 'COMMAND', 20), 'createdat': @@ -49,10 +50,15 @@ class Ps(AbstractActionBase): def list(self): """List containers.""" + if self._args.all: + ictnrs = self.client.containers.list() + else: + ictnrs = filter( + lambda c: podman.FoldedString(c['status']) == 'running', + self.client.containers.list()) + # TODO: Verify sorting on dates and size - ctnrs = sorted( - self.client.containers.list(), - key=operator.attrgetter(self._args.sort)) + ctnrs = sorted(ictnrs, key=operator.attrgetter(self._args.sort)) if not ctnrs: return @@ -65,9 +71,6 @@ class Ps(AbstractActionBase): 'createdat': humanize.naturaldate(podman.datetime_parse(ctnr.createdat)), }) - - if self._args.truncate: - fields.update({'image': ctnr.image[-30:]}) rows.append(fields) with Report(self.columns, heading=self._args.heading) as report: diff --git a/contrib/python/pypodman/pypodman/lib/actions/push_action.py b/contrib/python/pypodman/pypodman/lib/actions/push_action.py index 0030cb5b9..8e86ca335 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/push_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/push_action.py @@ -15,12 +15,10 @@ class Push(AbstractActionBase): 'push', help='push image elsewhere', ) - parser.add_argument( + parser.add_flag( '--tlsverify', - action='store_true', - default=True, help='Require HTTPS and verify certificates when' - ' contacting registries (default: %(default)s)') + ' contacting registries.') parser.add_argument( 'image', nargs=1, help='name or id of image to push') parser.add_argument( @@ -30,10 +28,6 @@ class Push(AbstractActionBase): ) parser.set_defaults(class_=cls, method='push') - def __init__(self, args): - """Construct Push class.""" - super().__init__(args) - def pull(self): """Store image elsewhere.""" try: diff --git a/contrib/python/pypodman/pypodman/lib/actions/restart_action.py b/contrib/python/pypodman/pypodman/lib/actions/restart_action.py index d99d1ad65..415594920 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/restart_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/restart_action.py @@ -23,10 +23,6 @@ class Restart(AbstractActionBase): 'targets', nargs='+', help='container id(s) to restart') parser.set_defaults(class_=cls, method='restart') - def __init__(self, args): - """Construct Restart class.""" - super().__init__(args) - def restart(self): """Restart container(s).""" try: diff --git a/contrib/python/pypodman/pypodman/lib/actions/rm_action.py b/contrib/python/pypodman/pypodman/lib/actions/rm_action.py index e8074ef4e..99ff6c460 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/rm_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/rm_action.py @@ -12,20 +12,14 @@ class Rm(AbstractActionBase): def subparser(cls, parent): """Add Rm command to parent parser.""" parser = parent.add_parser('rm', help='delete container(s)') - parser.add_argument( - '-f', + parser.add_flag( '--force', - action='store_true', - help=('force delete of running container(s).' - ' (default: %(default)s)')) + '-f', + help='force delete of running container(s).') parser.add_argument( 'targets', nargs='+', help='container id(s) to delete') parser.set_defaults(class_=cls, method='remove') - def __init__(self, args): - """Construct Rm class.""" - super().__init__(args) - def remove(self): """Remove container(s).""" for ident in self._args.targets: diff --git a/contrib/python/pypodman/pypodman/lib/actions/rmi_action.py b/contrib/python/pypodman/pypodman/lib/actions/rmi_action.py index c6ba835cb..7c3d0bd79 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/rmi_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/rmi_action.py @@ -12,19 +12,13 @@ class Rmi(AbstractActionBase): def subparser(cls, parent): """Add Rmi command to parent parser.""" parser = parent.add_parser('rmi', help='delete image(s)') - parser.add_argument( - '-f', + parser.add_flag( '--force', - action='store_true', - help=('force delete of image(s) and associated containers.' - ' (default: %(default)s)')) + '-f', + help='force delete of image(s) and associated containers.') parser.add_argument('targets', nargs='+', help='image id(s) to delete') parser.set_defaults(class_=cls, method='remove') - def __init__(self, args): - """Construct Rmi class.""" - super().__init__(args) - def remove(self): """Remove image(s).""" for ident in self._args.targets: diff --git a/contrib/python/pypodman/pypodman/lib/actions/run_action.py b/contrib/python/pypodman/pypodman/lib/actions/run_action.py index a63eb7917..6a6b3cb2c 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/run_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/run_action.py @@ -21,7 +21,7 @@ class Run(AbstractActionBase): parser.add_argument('image', nargs=1, help='source image id.') parser.add_argument( 'command', - nargs='*', + nargs=parent.REMAINDER, help='command and args to run.', ) parser.set_defaults(class_=cls, method='run') diff --git a/contrib/python/pypodman/pypodman/lib/actions/search_action.py b/contrib/python/pypodman/pypodman/lib/actions/search_action.py index d2a585d92..b7b8b465d 100644 --- a/contrib/python/pypodman/pypodman/lib/actions/search_action.py +++ b/contrib/python/pypodman/pypodman/lib/actions/search_action.py @@ -4,8 +4,8 @@ import sys from collections import OrderedDict import podman -from pypodman.lib import (AbstractActionBase, BooleanValidate, - PositiveIntAction, Report, ReportColumn) +from pypodman.lib import (AbstractActionBase, PositiveIntAction, Report, + ReportColumn) class FilterAction(argparse.Action): @@ -58,16 +58,16 @@ class FilterAction(argparse.Action): if val < 0: parser.error(msg) elif opt == 'is-automated': - try: - val = BooleanValidate()(val) - except ValueError: + if val.capitalize() in ('True', 'False'): + val = bool(val) + else: msg = ('{} option "is-automated"' ' must be True or False.'.format(self.dest)) parser.error(msg) elif opt == 'is-official': - try: - val = BooleanValidate()(val) - except ValueError: + if val.capitalize() in ('True', 'False'): + val = bool(val) + else: msg = ('{} option "is-official"' ' must be True or False.'.format(self.dest)) parser.error(msg) diff --git a/contrib/python/pypodman/pypodman/lib/actions/start_action.py b/contrib/python/pypodman/pypodman/lib/actions/start_action.py new file mode 100644 index 000000000..5f88731dc --- /dev/null +++ b/contrib/python/pypodman/pypodman/lib/actions/start_action.py @@ -0,0 +1,71 @@ +"""Remote client command for starting containers.""" +import sys + +import podman +from pypodman.lib import AbstractActionBase + + +class Start(AbstractActionBase): + """Class for starting container.""" + + @classmethod + def subparser(cls, parent): + """Add Start command to parent parser.""" + parser = parent.add_parser('start', help='start container') + parser.add_flag( + '--attach', + '-a', + help="Attach container's STDOUT and STDERR.") + parser.add_argument( + '--detach-keys', + metavar='KEY(s)', + default=4, + help='Override the key sequence for detaching a container.' + ' (format: a single character [a-Z] or ctrl-<value> where' + ' <value> is one of: a-z, @, ^, [, , or _) (default: ^D)') + parser.add_flag( + '--interactive', + '-i', + help="Attach container's STDIN.") + # TODO: Implement sig-proxy + parser.add_flag( + '--sig-proxy', + help="Proxy received signals to the process." + ) + parser.add_argument( + 'containers', + nargs='+', + help='containers to start', + ) + parser.set_defaults(class_=cls, method='start') + + def start(self): + """Start provided containers.""" + stdin = sys.stdin if self.opts['interactive'] else None + stdout = sys.stdout if self.opts['attach'] else None + + try: + for ident in self._args.containers: + try: + ctnr = self.client.containers.get(ident) + ctnr.attach( + eot=self.opts['detach_keys'], + stdin=stdin, + stdout=stdout) + ctnr.start() + except podman.ContainerNotFound as e: + sys.stdout.flush() + print( + 'Container "{}" not found'.format(e.name), + file=sys.stderr, + flush=True) + else: + print(ident) + except podman.ErrorOccurred as e: + sys.stdout.flush() + print( + '{}'.format(e.reason).capitalize(), + file=sys.stderr, + flush=True) + return 1 + return 0 diff --git a/contrib/python/pypodman/pypodman/lib/actions/version_action.py b/contrib/python/pypodman/pypodman/lib/actions/version_action.py new file mode 100644 index 000000000..29a0cabe4 --- /dev/null +++ b/contrib/python/pypodman/pypodman/lib/actions/version_action.py @@ -0,0 +1,35 @@ +"""Remote client command for reporting on Podman service.""" +import sys + +import podman +from pypodman.lib import AbstractActionBase + + +class Version(AbstractActionBase): + """Class for reporting on Podman Service.""" + + @classmethod + def subparser(cls, parent): + """Add Version command to parent parser.""" + parser = parent.add_parser( + 'version', help='report version on podman service') + parser.set_defaults(class_=cls, method='version') + + def version(self): + """Report on Podman Service.""" + try: + info = self.client.system.info() + except podman.ErrorOccurred as e: + sys.stdout.flush() + print( + '{}'.format(e.reason).capitalize(), + file=sys.stderr, + flush=True) + return 1 + else: + version = info._asdict()['podman'] + host = info._asdict()['host'] + print("Version {}".format(version['podman_version'])) + print("Go Version {}".format(version['go_version'])) + print("Git Commit {}".format(version['git_commit'])) + print("OS/Arch {}/{}".format(host["os"], host["arch"])) diff --git a/contrib/python/pypodman/pypodman/lib/parser_actions.py b/contrib/python/pypodman/pypodman/lib/parser_actions.py index c10b85495..3ff12cab8 100644 --- a/contrib/python/pypodman/pypodman/lib/parser_actions.py +++ b/contrib/python/pypodman/pypodman/lib/parser_actions.py @@ -6,6 +6,7 @@ The constructors are very verbose but remain for IDE support. import argparse import copy import os +import signal # API defined by argparse.Action therefore shut up pylint # pragma pylint: disable=redefined-builtin @@ -13,22 +14,8 @@ import os # pragma pylint: disable=too-many-arguments -class BooleanValidate(): - """Validate value is boolean string.""" - - def __call__(self, value): - """Return True, False or raise ValueError.""" - val = value.capitalize() - if val == 'False': - return False - elif val == 'True': - return True - else: - raise ValueError('"{}" is not True or False'.format(value)) - - -class BooleanAction(argparse.Action): - """Convert and validate bool argument.""" +class ChangeAction(argparse.Action): + """Convert and validate change argument.""" def __init__(self, option_strings, @@ -37,11 +24,16 @@ class BooleanAction(argparse.Action): const=None, default=None, type=None, - choices=('True', 'False'), + choices=None, required=False, help=None, - metavar='{True,False}'): - """Create BooleanAction object.""" + metavar='OPT=VALUE'): + """Create ChangeAction object.""" + help = (help or '') + ('Apply change(s) to the new image.' + ' May be given multiple times.') + if default is None: + default = [] + super().__init__( option_strings=option_strings, dest=dest, @@ -56,32 +48,37 @@ class BooleanAction(argparse.Action): def __call__(self, parser, namespace, values, option_string=None): """Convert and Validate input.""" - try: - val = BooleanValidate()(values) - except ValueError: - parser.error('{} must be True or False.'.format(self.dest)) - else: - setattr(namespace, self.dest, val) + items = getattr(namespace, self.dest, None) or [] + items = copy.copy(items) + choices = ('CMD', 'ENTRYPOINT', 'ENV', 'EXPOSE', 'LABEL', 'ONBUILD', + 'STOPSIGNAL', 'USER', 'VOLUME', 'WORKDIR') -class ChangeAction(argparse.Action): - """Convert and validate change argument.""" + opt, _ = values.split('=', 1) + if opt not in choices: + parser.error('Option "{}" is not supported by argument "{}",' + ' valid options are: {}'.format( + opt, option_string, ', '.join(choices))) + items.append(values) + setattr(namespace, self.dest, items) + + +class SignalAction(argparse.Action): + """Validate input as a signal.""" def __init__(self, option_strings, dest, nargs=None, const=None, - default=[], - type=None, + default=None, + type=str, choices=None, required=False, - help=None, - metavar='OPT=VALUE'): - """Create ChangeAction object.""" - help = (help or '') + ('Apply change(s) to the new image.' - ' May be given multiple times.') - + help='The signal to send.' + ' It may be given as a name or a number.', + metavar='SIGNAL'): + """Create SignalAction object.""" super().__init__( option_strings=option_strings, dest=dest, @@ -94,22 +91,40 @@ class ChangeAction(argparse.Action): help=help, metavar=metavar) - def __call__(self, parser, namespace, values, option_string=None): - """Convert and Validate input.""" - print(self.dest) - items = getattr(namespace, self.dest, None) or [] - items = copy.copy(items) + if hasattr(signal, "Signals"): - choices = ('CMD', 'ENTRYPOINT', 'ENV', 'EXPOSE', 'LABEL', 'ONBUILD', - 'STOPSIGNAL', 'USER', 'VOLUME', 'WORKDIR') + def _signal_number(signame): + cooked = 'SIG{}'.format(signame) + try: + return signal.Signals[cooked].value + except ValueError: + pass + else: - opt, val = values.split('=', 1) - if opt not in choices: - parser.error('{} is not a supported "--change" option,' - ' valid options are: {}'.format( - opt, ', '.join(choices))) - items.append(values) - setattr(namespace, self.dest, items) + def _signal_number(signame): + cooked = 'SIG{}'.format(signame) + for n, v in sorted(signal.__dict__.items()): + if n != cooked: + continue + if n.startswith("SIG") and not n.startswith("SIG_"): + return v + + self._signal_number = _signal_number + + def __call__(self, parser, namespace, values, option_string=None): + """Validate input is a signal for platform.""" + if values.isdigit(): + signum = int(values) + if signal.SIGRTMIN <= signum >= signal.SIGRTMAX: + raise ValueError('"{}" is not a valid signal. {}-{}'.format( + values, signal.SIGRTMIN, signal.SIGRTMAX)) + else: + signum = self._signal_number(values) + if signum is None: + parser.error( + '"{}" is not a valid signal,' + ' see your platform documentation.'.format(values)) + setattr(namespace, self.dest, signum) class UnitAction(argparse.Action): @@ -127,8 +142,8 @@ class UnitAction(argparse.Action): help=None, metavar='UNIT'): """Create UnitAction object.""" - help = (help or metavar or dest - ) + ' (format: <number>[<unit>], where unit = b, k, m or g)' + help = (help or metavar or dest)\ + + ' (format: <number>[<unit>], where unit = b, k, m or g)' super().__init__( option_strings=option_strings, dest=dest, @@ -148,15 +163,15 @@ class UnitAction(argparse.Action): except ValueError: if not values[:-1].isdigit(): msg = ('{} must be a positive integer,' - ' with optional suffix').format(self.dest) + ' with optional suffix').format(option_string) parser.error(msg) if not values[-1] in ('b', 'k', 'm', 'g'): msg = '{} only supports suffices of: b, k, m, g'.format( - self.dest) + option_string) parser.error(msg) else: if val <= 0: - msg = '{} must be a positive integer'.format(self.dest) + msg = '{} must be a positive integer'.format(option_string) parser.error(msg) setattr(namespace, self.dest, values) @@ -174,19 +189,16 @@ class PositiveIntAction(argparse.Action): type=int, choices=None, required=False, - help=None, + help='Must be a positive integer.', metavar=None): """Create PositiveIntAction object.""" - self.message = '{} must be a positive integer'.format(dest) - help = help or self.message - super().__init__( option_strings=option_strings, dest=dest, nargs=nargs, const=const, default=default, - type=int, + type=type, choices=choices, required=required, help=help, @@ -198,7 +210,8 @@ class PositiveIntAction(argparse.Action): setattr(namespace, self.dest, values) return - parser.error(self.message) + msg = '{} must be a positive integer'.format(option_string) + parser.error(msg) class PathAction(argparse.Action): diff --git a/contrib/python/pypodman/pypodman/lib/podman_parser.py b/contrib/python/pypodman/pypodman/lib/podman_parser.py index d3c84224f..913546a91 100644 --- a/contrib/python/pypodman/pypodman/lib/podman_parser.py +++ b/contrib/python/pypodman/pypodman/lib/podman_parser.py @@ -48,6 +48,18 @@ class PodmanArgumentParser(argparse.ArgumentParser): super().__init__(**kwargs) + def add_flag(self, *args, **kwargs): + """Add flag to parser.""" + flags = [a for a in args if a[0] in self.prefix_chars] + dest = flags[0].lstrip(self.prefix_chars) + no_flag = '{0}{0}no-{1}'.format(self.prefix_chars, dest) + + group = self.add_mutually_exclusive_group(required=False) + group.add_argument(*flags, action='store_true', dest=dest, **kwargs) + group.add_argument(no_flag, action='store_false', dest=dest, **kwargs) + default = kwargs.get('default', False) + self.set_defaults(**{dest: default}) + def initialize_parser(self): """Initialize parser without causing recursion meltdown.""" self.add_argument( @@ -97,6 +109,8 @@ class PodmanArgumentParser(argparse.ArgumentParser): actions_parser = self.add_subparsers( dest='subparser_name', help='commands') + # For create/exec/run: don't process options intended for subcommand + actions_parser.REMAINDER = argparse.REMAINDER # import buried here to prevent import loops import pypodman.lib.actions # pylint: disable=cyclic-import @@ -152,7 +166,7 @@ class PodmanArgumentParser(argparse.ArgumentParser): reqattr( 'run_dir', getattr(args, 'run_dir') - or os.environ.get('RUN_DIR') + or os.environ.get('PODMAN_RUN_DIR') or config['default'].get('run_dir') or str(Path(args.xdg_runtime_dir, 'pypodman')) ) # yapf: disable @@ -161,23 +175,24 @@ class PodmanArgumentParser(argparse.ArgumentParser): args, 'host', getattr(args, 'host') - or os.environ.get('HOST') + or os.environ.get('PODMAN_HOST') or config['default'].get('host') ) # yapf:disable reqattr( 'username', getattr(args, 'username') + or os.environ.get('PODMAN_USER') + or config['default'].get('username') or os.environ.get('USER') or os.environ.get('LOGNAME') - or config['default'].get('username') or getpass.getuser() ) # yapf:disable reqattr( 'port', getattr(args, 'port') - or os.environ.get('PORT') + or os.environ.get('PODMAN_PORT') or config['default'].get('port', None) or 22 ) # yapf:disable @@ -185,7 +200,7 @@ class PodmanArgumentParser(argparse.ArgumentParser): reqattr( 'remote_socket_path', getattr(args, 'remote_socket_path') - or os.environ.get('REMOTE_SOCKET_PATH') + or os.environ.get('PODMAN_REMOTE_SOCKET_PATH') or config['default'].get('remote_socket_path') or '/run/podman/io.podman' ) # yapf:disable @@ -193,7 +208,7 @@ class PodmanArgumentParser(argparse.ArgumentParser): reqattr( 'log_level', getattr(args, 'log_level') - or os.environ.get('LOG_LEVEL') + or os.environ.get('PODMAN_LOG_LEVEL') or config['default'].get('log_level') or logging.WARNING ) # yapf:disable @@ -202,7 +217,7 @@ class PodmanArgumentParser(argparse.ArgumentParser): args, 'identity_file', getattr(args, 'identity_file') - or os.environ.get('IDENTITY_FILE') + or os.environ.get('PODMAN_IDENTITY_FILE') or config['default'].get('identity_file') or os.path.expanduser('~{}/.ssh/id_dsa'.format(args.username)) ) # yapf:disable diff --git a/contrib/spec/podman.spec.in b/contrib/spec/podman.spec.in index c2d8fc59d..20e2a84ea 100644 --- a/contrib/spec/podman.spec.in +++ b/contrib/spec/podman.spec.in @@ -39,7 +39,7 @@ %global shortcommit_conmon %(c=%{commit_conmon}; echo ${c:0:7}) Name: podman -Version: 0.10.2 +Version: 0.12.2 Release: #COMMITDATE#.git%{shortcommit0}%{?dist} Summary: Manage Pods, Containers and Container Images License: ASL 2.0 @@ -378,10 +378,6 @@ providing packages with %{import_path} prefix. %prep %autosetup -Sgit -n %{repo}-%{shortcommit0} -sed -i '/\/bin\/env/d' completions/bash/%{name} -sed -i 's/0.0.0/%{version}/' contrib/python/%{name}/setup.py -sed -i 's/0.0.0/%{version}/' contrib/python/py%{name}/setup.py -mv pkg/hooks/README.md pkg/hooks/README-hooks.md # untar cri-o tar zxf %{SOURCE1} @@ -416,15 +412,17 @@ popd %install install -dp %{buildroot}%{_unitdir} -%{__make} PREFIX=%{buildroot}%{_prefix} ETCDIR=%{buildroot}%{_sysconfdir} \ +PODMAN_VERSION=%{version} %{__make} PREFIX=%{buildroot}%{_prefix} ETCDIR=%{buildroot}%{_sysconfdir} \ install.bin \ install.man \ install.cni \ install.systemd \ install.completions +mv pkg/hooks/README.md pkg/hooks/README-hooks.md + %if %{with varlink} -%{__make} DESTDIR=%{buildroot} install.python +PODMAN_VERSION=%{version} %{__make} DESTDIR=%{buildroot} install.python %endif # varlink # install libpod.conf |