summaryrefslogtreecommitdiff
path: root/test/system/helpers.bash
Commit message (Collapse)AuthorAge
* Merge pull request #10199 from edsantiago/system_tests_with_runc_overrideOpenShift Merge Robot2021-05-04
|\ | | | | System tests: honor $OCI_RUNTIME (for CI)
| * System tests: honor $OCI_RUNTIME (for CI)Ed Santiago2021-05-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some CI systems set $OCI_RUNTIME as a way to override the default crun. Integration (e2e) tests honor this, but system tests were not aware of the convention; this means we haven't been testing system tests with runc, which means RHEL gating tests are now failing. The proper solution would be to edit containers.conf on CI systems. Sorry, that would involve too much CI-VM work. Instead, this PR detects $OCI_RUNTIME and creates a dummy containers.conf file using that runtime. Add: various skips for tests that don't work with runc. Refactor: add a helper function so we don't need to do the complicated 'podman info blah blah .OCIRuntime.blah' thing in many places. BUG: we leave a tmp file behind on exit. Signed-off-by: Ed Santiago <santiago@redhat.com>
* | system test image: add arm64v8 imageEd Santiago2021-05-03
|/ | | | | | | | | | | The RHEL multi-arch team informed me that we were missing aarch64; add it, using the new name (arm64v8). (This is from last week, so the image date tag does not match today's date. I was waiting for confirmation that things were working). Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: setup: better cleanup of stray imagesEd Santiago2021-04-13
| | | | | | | | | | | | | | | | Fix a corner case in basic_setup(), where we rmi stray images. If a test tags $IMAGE and fails to rmi by tag name, cleanup could rmi both tag name and IID, wiping out the desired image: podman tag $IMAGE foo ... cleanup: rmi foo $FOO_IID [this removes $IMAGE!] Solution: rmi by name, but only rmi by IID if != $IMAGE. TOTH to ypu for bringing this to my attention. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: friendier messages for 2-arg is()Ed Santiago2021-03-29
| | | | | | | | | | | | | | | | | The 'is' check was intended to be called with three arguments, the last one being a nice helpful test name. There's a fallback for two-argument calls, but it was a horrible FIXME. New fallback: the most recently run podman command. We keep track of it in each run_podman() invocation. This is not ideal, because it's theoretically possible to invoke 'is' on something other than the output of run_podman, but this at least fixes the by-far-most-common case. [NO TESTS NEEDED] Signed-off-by: Ed Santiago <santiago@redhat.com>
* Merge pull request #8749 from edsantiago/upgrade_testOpenShift Merge Robot2021-02-26
|\ | | | | podman upgrade tests
| * podman upgrade testsEd Santiago2021-02-23
| | | | | | | | | | | | | | | | Initial validation of using podman-in-podman to create an old-podman root, then use new-podman to play with the containers created therein. Signed-off-by: Ed Santiago <santiago@redhat.com>
* | system test image: build it multiarchEd Santiago2021-02-24
|/ | | | | | | | | | | | | | buildah now supports running under emulation[1] as well as creating manifests. Use those features to create a multiarch testimage that can be used to test podman on other arches. [1] on Fedora 33, this requires the qemu-user-static package We also build a new :00000001 image, replacing :00000000, because (sigh) some tests try to run 'true' in the container. Include instructions on building said image. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: set PODMAN_TIMEOUT to 120Valentin Rothberg2020-12-18
| | | | | | | | | | The new Ubuntu 20.04 VMs seem very slow and fail reproducibly in a build test (i.e, "wordir, cmd, env, label"). Bumping up the time out to 120 seconds will help get the CI green. See github.com/containers/podman/pull/8747. Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
* system tests: the catch-up gameEd Santiago2020-12-14
| | | | | | | | | | | | | | | | | | | | | | | | | - run test: minor cleanup to .containerenv test. Basically, make it do only two podman-runs (they're expensive) and tighten up the results checks - ps test: add ps -a --storage. Requires small tweak to run_podman helper, so we can have "timeout" be an expected result - sdnotify test: workaround for #8718 (seeing MAINPID=xxx as last output line instead of READY=1). As found by the newly-added debugging echos, what we are seeing is: MAINPID=103530 READY=1 MAINPID=103530 It's not supposed to be that way; it's supposed to be just the first two. But when faced with reality, we must bend to accommodate it, so let's accept READY=1 anywhere in the output stream, not just as the last line. Signed-off-by: Ed Santiago <santiago@redhat.com>
* RHEL gating tests: more journald exceptionsEd Santiago2020-12-14
| | | | | | | | | | | | | | | | | | | | Followup to #8284, due to my not having noticed #8096. RHEL gating tests are failing again due to rhbz#1895105, the one where we can't run journalctl rootless on RHEL. #8284 fixed this for some RHEL builds of older podman, but I missed #8096 which added yet another logs test. This brings us to three journalctl exceptions, which means it gets complicated because I have to refactor it all. **THIS IS NOT SUSTAINABLE**. We need some way to have a similar setup in CI, with a permission-less rootless login, so we don't add yet another logs test some day and discover, months later, that it doesn't work on RHEL and then have to go into crisis mode. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Add system test for shell completionPaul Holzinger2020-12-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There exists a unit test to ensure that shell completion functions are defined. However there was no check about the quality of the provided shell completions. Lets change that. The idea is to create a general test that makes sure we are suggesting containers,pods,images... for the correct commands. This works by reading the command use line and checking for each arg if we provide the correct suggestions for this arg. It includes the following tests: - flag suggestions if [options] is set - container, pod, image, network, volume, registry completion - path completion for the appropriate arg KEYWORDS (`PATH`,`CONTEXT`,etc.) - no completion if there are no args - completion for more than one arg if it ends with `...]` The test does not cover completion values for flags and not every arg KEYWORD is supported. This is still a huge improvement and covers most use cases. This test spotted several inconsistencies between the completion and the command use line. All of them have been adjusted to make the test pass. The biggest advantage is that the completions always match the latest command changes. So if someone changes the arguments for a command this ensures that the completions must be adjusted. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
* Add a system test to verify --runtime is preservedMatthew Heon2020-10-20
| | | | Signed-off-by: Matthew Heon <matthew.heon@pm.me>
* System tests: remove some misleading 'run'sEd Santiago2020-10-19
| | | | | | | | | | | | | | | The BATS 'run' directive is really quite obnoxious; for the most part we really don't want to use it. Remove some uses that snuck in last week, and remove one test (exists) that can more naturally be piggybacked into an rm test. While we're at it: in setup(), look for and delete stray external (buildah) containers. This will be important if any of the external-container tests fails; this way we don't leave behind a state that causes subsequent tests to fail. Signed-off-by: Ed Santiago <santiago@redhat.com>
* System test additionsEd Santiago2020-10-14
| | | | | | | | | | | | | | | | | | | | | | | | - run --userns=keep-id: confirm that $HOME gets set (#8013) - inspect: confirm that JSON output is a sane number of lines (10 or more), not an unreadable one-liner (#8011 and #8021). Do so with image, pod, network, volume because the code paths might be different. - cgroups: confirm that 'run' preserves cgroup manager (#7970) - sdnotify: reenable tests, and hope CI doesn't hang. This test was disabled on August 18 because CI jobs were hanging and timing out. My suspicion was that it was #7316, which in turn seems to have hinged on conmon #182. The latter was merged on Sep 16, so let's cross our fingers and see what happens. Also: remove inaccurate warning from a networking test. And, wow, fix is_cgroupsv2(), it has never actually worked. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: cleanup, and add more testsEd Santiago2020-10-07
| | | | | | | | | | | | | | | | | | | | - images test: add test for 'table' and '\t' formatting - image mount test: check output from 'umount', test repeat umount (NOP), and test invalid-umount - kill test: remove kludgy workaround for crun signal bug ref: #5004 -- code is no longer needed (fingers crossed), and the workaround involved pulling an expensive image. - selinux test: add new tests for shared context in: * pods , w/ and w/o infra container (ref: #7902) * containers with namespace sharing: --ipc, --pid, --net - selinux test: new test for --pid=host (disabled pending propagation of container-selinux-2.146, ref: #7939) Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: add podman run --tzEd Santiago2020-09-29
| | | | | | | | New tests for podman run --tz=EXPLICIT and =local. Requires updating our testimage by adding a fixed reference timestamp to a known file path. Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: reenable some skipped testsEd Santiago2020-09-28
| | | | | | | | | | | | | | | | | | | | | - pause test: enable when rootless + cgroups v2 (was previously disabled for all rootless) - run --pull: now works with podman-remote (in #7647, thank you @jwhonce) - various other run/volumes tests: try reenabling It looks like #7195 was fixed (by #7451? I'm not sure if I'm reading the conversation correctly). Anyway, remove all the skip()s on 7195. Only time will tell if it's really fixed) Also: - new test for podman image tree --whatrequires (because TIL). Doesn't work with podman-remote. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Merge pull request #7687 from edsantiago/batsOpenShift Merge Robot2020-09-25
|\ | | | | system tests: new tests
| * system tests: new testsEd Santiago2020-09-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - podman network create: new test - podman pull by-sha + podman images -a (#7651) - podman image mount: new test - podman pod: --infra-image and --infra-command (#7167) For convenience and robustness, build a new testimage containing a custom file /home/podman/testimage-id with contents YYYYMMDD (same as image tag). The image-mount test checks that this file exists and has the desired content. New testimage also includes a dummy 'pause' executable, for testing pod infra. Updates from testimage:20200902 to :20200917 Signed-off-by: Ed Santiago <santiago@redhat.com>
* | system tests: helpers: safer parse_tableEd Santiago2020-09-23
|/ | | | | | | | | | | | | | | | | The parse_table() helper has until now dumbly split lines on every single '|' character. This prevents us from running simple tests such as 'cgroupManager: (systemd|cgroupfs)'. We now use an ugly but robust sed expression to split on '|' but *only* when surrounded by spaces and/or beginning or end of line. This is safe because, for readability, all tables already keep the '|' symbols well separated from table content. Add tests. And, the whole reason behind this, add an actual real test for cgroupManager and cgroupVersion. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Migrate away from docker.ioEd Santiago2020-09-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CI and system tests currently pull some images from docker.io. Eliminate that, by: - building a custom image containing much of what we need for testing; and - copying other needed images to quay.io (Reason: effective 2020-11-01 docker.io will limit the number of image pulls). The principal change is to create a new quay.io/libpod/testimage, using the new test/system/build-testimage script, instead of relying on quay.io/libpod/alpine_labels. We also switch to using a hardcoded :YYYYMMDD tag, instead of :latest, in an attempt to futureproof our CI. This image includes 'httpd' from busybox-extras, which we use in our networking test (previously we had to pull and run busybox from docker.io). The testimage can and should be extended as needed for future tests, e.g. adding test file content or other useful tools. For the '--pull' tests which require actually pulling from the registry, I've created an image with the same name but tagged :00000000 so it will never be pulled by default. Since this image is only used minimally, it's just busybox. Unfortunately there remain two cases we cannot solve in this tiny alpine-based image: 1) docker registry 2) systemd For those, I've (manually) run: podman pull [ docker.io/library/registry:2.7 | registry.fedoraproject.org/fedora:31 ] podman tag !$ quay.io/... podman push !$ ...and amended the calling tests accordingly. I've tried to make the the smallest reasonable diff, not the smallest possible one. I hope it's a reasonable tradeoff. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Just use `rm` for helper command to remove storageSascha Grunert2020-08-28
| | | | | | This allows to use any kind of `rm` in `$PATH` for the system tests. Signed-off-by: Sascha Grunert <sgrunert@suse.com>
* Reenable remote system testsEd Santiago2020-08-03
| | | | | | | | | | | | | | | | | | | podman-remote is in better shape now. Let's see what needs to be done to reenable remote system tests. - logs test: skip multilog, it doesn't work remote - diff test: use -l only when local, not with remote - many other tests: skip_if_remote, with 'FIXME: pending #xxxx' where xxxx is a filed issue. Unrelated: added new helper to skip_if_remote and _if_rootless, where we check if the source message includes "remote"/"rootless" and insert it if missing. This is a minor usability enhancement to make it easier to understand at-a-glance why a skip triggers. Signed-off-by: Ed Santiago <santiago@redhat.com>
* BATS tests: more resilient remove_same_dev_warningEd Santiago2020-07-20
| | | | | | | | | | | | | | | | Some CI tests are flaking in the SELinux test, possibly because there's a new variation of the "multiple devices" warning I hadn't seen before: WARNING: Creating device "/dev/null" with same type, major and minor as existing "/dev/foodevdir/null". Solution: in remove_same_dev_warning(), remove "multiple" from the match string. Also: fix a Go test that wasn't cleaning up after itself. And add an actual test to it, not just check-exit-status. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: new tests for run, execEd Santiago2020-07-14
| | | | | | | | | | | | | | | | | | | | | | - Issue #6735 : problem with multiple namespaces; confirms combinations of --userns=keep-id, --privileged, --user=XX - Issue #6829 : --userns=keep-id will add a /etc/passwd entry - Issue #6593 : podman exec, with --userns=keep-id, errors (test is currently skipped because issue remains live) ...and, addendum: add new helper function, remove_same_dev_warning. Some CI systems issue a warning on podman run --privileged: WARNING: The same type, major and minor should not be used for multiple devices. We already had special-case code to ignore than in the SELinux test, but now we're seeing it in the new run tests I added, so I've refactored the "ignore this warning" code and written tests for the removal code. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: invoke with abs path to podmanEd Santiago2020-06-23
| | | | | | | | | | | | | | | | | | | | | | Reversion of one part of #6679: my handling of 'realpath' would not work when $PODMAN is 'podman-remote --url etc'. Trying to handle that case got unmaintainable; so instead let's just force 'make {local,remote}system' to invoke with a full PODMAN path. This breaks down if someone runs the tests with a manual 'bats' invocation, but I think I'm the only one who ever does that. Since podman path will now be very long in the logs, add code to logformatter to abbreviate it like we do for the ginkgo logs. And, one thing that has bugged me for a long time: in the error logs, show a different prompt ('#' vs '$') to distinguish root vs rootless. This should make it much easier to see at-a-glance whether a log file is root or not. Add tests for it. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: new rm, build testsEd Santiago2020-06-22
| | | | | | | | | | | | - rm: confirm 'rm' and 'rm -f' on running container - build: shotgun test of workdir, cmd, env, labels The new build test cd's to a temporary directory, which broke test invocations using a relative path (./bin/podman). Added code to detect relative paths and convert them to absolute. Signed-off-by: Ed Santiago <santiago@redhat.com>
* fix and enable systemd system testsValentin Rothberg2020-05-08
| | | | | | | | | | | The systemd unit test never ran in CI and was broken for various reasons. Fix the test to execute Podman in systemd units and to also run generated units files. Note: more tests will be added in the future. The simple check for now will prevent regressions. Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
* refactor infoBrent Baude2020-04-06
| | | | | | the current implementation of info, while typed, is very loosely done so. we need stronger types for our apiv2 implmentation and bindings. Signed-off-by: Brent Baude <bbaude@redhat.com>
* more BATS testsEd Santiago2020-01-13
| | | | | | | | | | | | | | | | | - run: --name (includes 'podman container exists' tests) - run: --pull (always, never, missing) - build: new test for ADD URL (#4420) - exec: new test for issue #4785 (pipe getting lost) - diff: new test - selinux (mostly copied from docker-autotest) Plus a bug fix: the wait_for_output() helper would continue checking, eventually timing out, even if the container had already exited (probably because of an error). Fix: as part of the loop, run 'podman inspect' and bail out if container is not running. Include exit code and logs. Signed-off-by: Ed Santiago <santiago@redhat.com>
* podman images history test - clean upEd Santiago2019-12-12
| | | | | | | | | | | | | | | As initially written the test does not work other than in a CI environment because it relies on an empty tag history. Rewrite so we can guarantee that, by creating a new image. Also add slightly more helpful tests: the initial tests would just show "expected 0, got 1" which is unhelpful. Tweak so we test on actual history contents, which will show more informative messages on failure. And, finally, clean up after ourselves. Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: make sure exec pid hash w/o leakingAlex Jia2019-10-18
| | | | | | | | | | podman exec leaks an exec_pid_<hash> file for every exec in tmpfs, it's known rhbz#1731117, this case makes sure leakage issue has been fixed. rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1731117 Signed-off-by: Alex Jia <chuanchang.jia@gmail.com>
* implement 'make remotesystem'Ed Santiago2019-08-08
| | | | | | | | | | | | | | | | | | | | | podman-remote rm now works; that's the only thing we were waiting for to enable podman-remote (varlink) system tests. Add a (too-complicated, sorry) Makefile target that will define a random socket path, start the podman varlink server, and run the test suite using podman-remote. Also: add two convenience functions, is_rootless and is_remote, and use those in skip_if_rootless/if_remote and elsewhere Also: workarounds for broken tests: - basic version test: podman-remote emits an empty 'Client' line. Just ignore it. - looks like 'podman-remote pod' doesn't work; skip test. Also: minor documentation update Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: resolve hang in rawhide rootlessEd Santiago2019-08-01
| | | | | | | | | | | | | | | | | | | Fedora CI tests are failing on rawhide under kernel 5.3.0-0.rc1.git3.1.fc31 (rhbz#1736758). But there's another insidious failure, a 4-hour hang in the rootless tests on the same CI system. The culprit line is in the podman build test, but it's actually BATS itself that hangs, not the build command -- which suggests that it's the usual FD 3 problem (see BATS README). It would seem that podman is forking a process that inherits fd 3 but that process is not getting cleaned up when podman crashes upon encountering the kernel bug. Today it's podman build, tomorrow it might be something else. Let's just run all podman invocations in run_podman with a non-bats FD 3. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Use conmon pidfile in generated systemd unit as PIDFile.Danila Kiver2019-07-04
| | | | | | | | | | | By default, podman points PIDFile in generated unit file to non-existent location. As a result, the unit file, generated by podman, is broken: an attempt to start this unit without prior modification results in a crash, because systemd can not find the pidfile of service's main process. Fix the value of "PIDFile" and add a system test for this case. Signed-off-by: Danila Kiver <danila.kiver@mail.ru>
* BATS tests: start supporting podman-remoteEd Santiago2019-04-10
| | | | | | | | | | | | | | podman-remote now supports rm! That's what we needed to start running BATS tests. Although most tests don't actually work, some do, and maybe the rest will start working over time. For now, disable them. The only significant difference found is that podman-remote strips fractional seconds from timestamps in JSON output. Probably not something worth caring about. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Implement review feedbackEd Santiago2019-03-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - document a recommended convention for fail-fast tests - document the requirement for jq. (And, add a fail-fast test for its presence; remove the duplicated checks in subtests) - add further sanity checks to 'help' test. Add missing documentation. Remove a no-longer-needed workaround for usage-message bug fixed in #2486 - add a documented TEMPLATE - and, since we're at 1.1, enable 'Remote API' check in version test - better diagnostics in setup/teardown; add vim filetype hint; better formatting of actual-vs-expect errors - new pod-top, logs, build tests - improve error messages - add $IMAGE alias for ridiculous $PODMAN_TEST_IMAGE_FQN - final cleanup, in prep for merge Signed-off-by: Ed Santiago <santiago@redhat.com>
* new system tests under BATSEd Santiago2019-03-07
Initial attempt at writing a framework for podman system tests. The idea is to define a useful set of primitives that will make it easy to write actual tests and to interpret results of failing ones. This is a proof-of-concept right now; only a small number of tests, by no means comprehensive. I am requesting review in order to find showstopper problems: reasons why this approach cannot work. Should there be none, we can work toward running these as gating tests for Fedora and RHEL8. Signed-off-by: Ed Santiago <santiago@redhat.com>