summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorOpenShift Merge Robot <openshift-merge-robot@users.noreply.github.com>2019-03-08 09:13:07 -0800
committerGitHub <noreply@github.com>2019-03-08 09:13:07 -0800
commit5fdb8bfcb346f41029b0876bd11477282811642c (patch)
tree4463ec7644ce74a8c6e8491153d1404011a2406a
parentc4815e8a618cb7824f0a8e8c84771c69ed2a8943 (diff)
parent53cd9bed95d7fdfc7712ed34071083042c81d5c1 (diff)
downloadpodman-5fdb8bfcb346f41029b0876bd11477282811642c.tar.gz
podman-5fdb8bfcb346f41029b0876bd11477282811642c.tar.bz2
podman-5fdb8bfcb346f41029b0876bd11477282811642c.zip
Merge pull request #2257 from cevich/system_test_beginning
[ci skip] System-test: Begin list of needed high-level tests
-rw-r--r--test/README.md4
-rw-r--r--test/system/TODO.md105
2 files changed, 109 insertions, 0 deletions
diff --git a/test/README.md b/test/README.md
index ef3bfbcf9..5e5a7da61 100644
--- a/test/README.md
+++ b/test/README.md
@@ -105,3 +105,7 @@ You can run the test with following command:
```
make localsystem
```
+
+## Contributing to system tests
+
+Please see [the TODO list of needed workflows/tests](system/TODO.md).
diff --git a/test/system/TODO.md b/test/system/TODO.md
new file mode 100644
index 000000000..f6110d2e9
--- /dev/null
+++ b/test/system/TODO.md
@@ -0,0 +1,105 @@
+![PODMAN logo](../../logo/podman-logo-source.svg)
+
+# Overview
+
+System tests exercise Podman in the context of a complete, composed environment from
+distribution packages. It should match as closely as possible to how an end-user
+would experience a fresh-install. Dependencies on external configuration and resources
+must be kept minimal, and the tests must be generic and vendor-neutral.
+
+The system-tests must execute cleanly on all tested platforms. They may optionally
+be executed during continuous-integration testing of code-changes, after all other
+testing completes successfully. For a list of tested platforms, please see [the
+CI configuration file.](../../.cirrus.yml)
+
+
+# Execution
+
+When working from a clone of [the libpod repository](https://github.com/containers/libpod),
+the main entry-point for humans and automation is `make localsystem`. When operating
+from a packaged version of the system-tests, the entry-point may vary as appropriate.
+Running the packaged system-tests assumes the version of Podman matches the test
+version, and all standard dependencies are installed.
+
+
+# Test Design and overview
+
+System-tests should be high-level and user work-flow oriented. For example, consider
+how multiple Podman invocations would be used together by an end-user. The set of
+related commands should be considered a single test. If one or more intermediate
+commands fail, the test could still pass if the end-result is still achieved.
+
+
+# *TODO*: List of needed System-tests
+
+***Note***: Common operations (like `rm` and `rmi` for cleanup/reset)
+have been omitted as they are verified by repeated implied use.
+
+- [ ] pull, build, run, attach, commit, diff, inspect
+
+ - Pull existing image from registry
+ - Build new image FROM explicitly pulled image
+ - Run built container in detached mode
+ - Attach to running container, execute command to modify storage.
+ - Commit running container to new image w/ changed ENV VAR
+ - Verify attach + commit using diff
+ - verify changed ENV VAR with inspect
+
+- [ ] Implied pull, create, start, exec, log, stop, wait, rm
+
+ - Create non-existing local image
+ - start stopped container
+ - exec simple command in running container
+ - verify exec result with log
+ - wait on running container
+ - stop running container with 2 second timeout
+ - verify wait in 4 seconds or less
+ - verify stopped by rm **without** --force
+
+- [ ] Implied pull, build, export, modify, import, tag, run, kill
+
+ - Build from Dockerfile FROM non-existing local image
+ - Export built container as tarball
+ - Modify tarball contents
+ - Import tarball
+ - Tag imported image
+ - Run imported image to confirm tarball modification, block on non-special signal
+ - Kill can send non-TERM/KILL signal to container to exit
+ - Confirm exit within timeout
+
+- [ ] Container runlabel, exists, checkpoint, exists, restore, stop, prune
+
+ - Using pre-existing remote image, start it with 'podman container runlabel --pull'
+ - Run a named container that exits immediatly
+ - Confirm 'container exists' zero exit (both containers)
+ - Checkpoint the running container
+ - Confirm 'container exists' non-zero exit (runlabel container)
+ - Confirm 'container exists' zero exit (named container)
+ - Run 'container restore'
+ - Confirm 'container exists' zero exit (both containers)
+ - Stop container
+ - Run 'container prune'
+ - Confirm `podman ps -a` lists no containers
+
+
+# TODO: List of commands to be combined into additional workflows above.
+
+- podman-remote (workflow TBD)
+- history
+- image
+- load
+- mount
+- pause
+- pod
+- port
+- login, push, & logout (difficult, save for last)
+- restart
+- save
+- search
+- stats
+- top
+- umount, unmount
+- unpause
+- volume
+- `--namespace`
+- `--storage-driver`