1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
# Overview
System tests exercise Podman in the context of a complete, composed environment from
distribution packages. It should match as closely as possible to how an end-user
would experience a fresh-install. Dependencies on external configuration and resources
must be kept minimal, and the tests must be generic and vendor-neutral.
The system-tests must execute cleanly on all tested platforms. They may optionally
be executed during continuous-integration testing of code-changes, after all other
testing completes successfully. For a list of tested platforms, please see [the
CI configuration file.](../../.cirrus.yml)
# Execution
When working from a clone of [the libpod repository](https://github.com/containers/podman),
the main entry-point for humans and automation is `make localsystem`. When operating
from a packaged version of the system-tests, the entry-point may vary as appropriate.
Running the packaged system-tests assumes the version of Podman matches the test
version, and all standard dependencies are installed.
# Test Design and overview
System-tests should be high-level and user work-flow oriented. For example, consider
how multiple Podman invocations would be used together by an end-user. The set of
related commands should be considered a single test. If one or more intermediate
commands fail, the test could still pass if the end-result is still achieved.
# *TODO*: List of needed System-tests
***Note***: Common operations (like `rm` and `rmi` for cleanup/reset)
have been omitted as they are verified by repeated implied use.
- [ ] pull, build, run, attach, commit, diff, inspect
- Pull existing image from registry
- Build new image FROM explicitly pulled image
- Run built container in detached mode
- Attach to running container, execute command to modify storage.
- Commit running container to new image w/ changed ENV VAR
- Verify attach + commit using diff
- verify changed ENV VAR with inspect
- [ ] Implied pull, create, start, exec, log, stop, wait, rm
- Create non-existing local image
- start stopped container
- exec simple command in running container
- verify exec result with log
- wait on running container
- stop running container with 2 second timeout
- verify wait in 4 seconds or less
- verify stopped by rm **without** --force
- [ ] Implied pull, build, export, modify, import, tag, run, kill
- Build from Dockerfile FROM non-existing local image
- Export built container as tarball
- Modify tarball contents
- Import tarball
- Tag imported image
- Run imported image to confirm tarball modification, block on non-special signal
- Kill can send non-TERM/KILL signal to container to exit
- Confirm exit within timeout
- [ ] Container runlabel, exists, checkpoint, exists, restore, stop, prune
- Using pre-existing remote image, start it with 'podman container runlabel --pull'
- Run a named container that exits immediately
- Confirm 'container exists' zero exit (both containers)
- Checkpoint the running container
- Confirm 'container exists' non-zero exit (runlabel container)
- Confirm 'container exists' zero exit (named container)
- Run 'container restore'
- Confirm 'container exists' zero exit (both containers)
- Stop container
- Run 'container prune'
- Confirm `podman ps -a` lists no containers
# TODO: List of commands to be combined into additional workflows above.
- podman-remote (workflow TBD)
- history
- image
- load
- mount
- pause
- pod
- port
- login, push, & logout (difficult, save for last)
- restart
- save
- search
- stats
- top
- umount, unmount
- unpause
- volume
- `--namespace`
- `--storage-driver`
|
