Merge pull request #2257 from cevich/system_test_beginning

[ci skip] System-test: Begin list of needed high-level tests

2 files changed, 109 insertions, 0 deletions

diff --git a/test/README.md b/test/README.md
index ef3bfbcf9..5e5a7da61 100644
--- a/test/README.md
+++ b/test/README.md
@@ -105,3 +105,7 @@ You can run the test with following command:
 ```
 make localsystem
 ```
+
+## Contributing to system tests
+
+Please see [the TODO list of needed workflows/tests](system/TODO.md).
diff --git a/test/system/TODO.md b/test/system/TODO.md
new file mode 100644
index 000000000..f6110d2e9
--- /dev/null
+++ b/test/system/TODO.md
@@ -0,0 +1,105 @@
+![PODMAN logo](../../logo/podman-logo-source.svg)
+
+# Overview
+
+System tests exercise Podman in the context of a complete, composed environment from
+distribution packages.  It should match as closely as possible to how an end-user
+would experience a fresh-install.  Dependencies on external configuration and resources
+must be kept minimal, and the tests must be generic and vendor-neutral.
+
+The system-tests must execute cleanly on all tested platforms.  They may optionally
+be executed during continuous-integration testing of code-changes, after all other
+testing completes successfully.  For a list of tested platforms, please see [the
+CI configuration file.](../../.cirrus.yml)
+
+
+# Execution
+
+When working from a clone of [the libpod repository](https://github.com/containers/libpod),
+the main entry-point for humans and automation is `make localsystem`.  When operating
+from a packaged version of the system-tests, the entry-point may vary as appropriate.
+Running the packaged system-tests assumes the version of Podman matches the test
+version, and all standard dependencies are installed.
+
+
+# Test Design and overview
+
+System-tests should be high-level and user work-flow oriented.  For example, consider
+how multiple Podman invocations would be used together by an end-user.  The set of
+related commands should be considered a single test.  If one or more intermediate
+commands fail, the test could still pass if the end-result is still achieved.
+
+
+# *TODO*: List of needed System-tests
+
+***Note***: Common operations (like `rm` and `rmi` for cleanup/reset)
+have been omitted as they are verified by repeated implied use.
+
+- [ ] pull, build, run, attach, commit, diff, inspect
+
+  - Pull existing image from registry
+  - Build new image FROM explicitly pulled image
+  - Run built container in detached mode
+  - Attach to running container, execute command to modify storage.
+  - Commit running container to new image w/ changed ENV VAR
+  - Verify attach + commit using diff
+  - verify changed ENV VAR with inspect
+
+- [ ] Implied pull, create, start, exec, log, stop, wait, rm
+
+  - Create non-existing local image
+  - start stopped container
+  - exec simple command in running container
+  - verify exec result with log
+  - wait on running container
+  - stop running container with 2 second timeout
+  - verify wait in 4 seconds or less
+  - verify stopped by rm **without** --force
+
+- [ ] Implied pull, build, export, modify, import, tag, run, kill
+
+  - Build from Dockerfile FROM non-existing local image
+  - Export built container as tarball
+  - Modify tarball contents
+  - Import tarball
+  - Tag imported image
+  - Run imported image to confirm tarball modification, block on non-special signal
+  - Kill can send non-TERM/KILL signal to container to exit
+  - Confirm exit within timeout
+
+- [ ] Container runlabel, exists, checkpoint, exists, restore, stop, prune
+
+  - Using pre-existing remote image, start it with 'podman container runlabel --pull'
+  - Run a named container that exits immediatly
+  - Confirm 'container exists' zero exit (both containers)
+  - Checkpoint the running container
+  - Confirm 'container exists' non-zero exit (runlabel container)
+  - Confirm 'container exists' zero exit (named container)
+  - Run 'container restore'
+  - Confirm 'container exists' zero exit (both containers)
+  - Stop container
+  - Run 'container prune'
+  - Confirm `podman ps -a` lists no containers
+
+
+# TODO: List of commands to be combined into additional workflows above.
+
+- podman-remote (workflow TBD)
+- history
+- image
+- load
+- mount
+- pause
+- pod
+- port
+- login, push, & logout (difficult, save for last)
+- restart
+- save
+- search
+- stats
+- top
+- umount, unmount
+- unpause
+- volume
+- `--namespace`
+- `--storage-driver`