summaryrefslogtreecommitdiff
path: root/test/system/015-help.bats
Commit message (Collapse)AuthorAge
* system tests: add assert(), and start using itEd Santiago2022-04-20
| | | | | | | | | | | | | | | | | | | | | Problem: the system test 'is()' checker was poorly thought out. For example, there is no way to check for inequality or for absence of a substring. Solution, step 1: introduce new assert(), copied almost verbatim from buildah, where it has been successful in addressing the gaps in is(). The logical next step is to search the tests for 'die' and for 'run', looking for negative assertions which we can replace with assert(). There were a lot, and in the process I found a number of ugly bugs in the tests themselves. I've taken the liberty of fixing these. Important note: at this time we have both assert() and is(). Replacing all instances of is() would be impossible to review. Signed-off-by: Ed Santiago <santiago@redhat.com>
* System tests: Usage checks: better error messagesEd Santiago2022-04-13
| | | | | | | | Current error messages are really awful, and cause great suffering every time someone adds a new subcommand. Let's see if these are better. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Merge pull request #11953 from markusthoemmes/help-defaultOpenShift Merge Robot2021-11-08
|\ | | | | Display help text on empty subcommand by default
| * Keep error semantics intactMarkus Thömmes2021-11-05
| | | | | | | | Signed-off-by: Markus Thömmes <markusthoemmes@me.com>
| * Adjust tests to verify all subcommands show the help messageMarkus Thömmes2021-10-14
| | | | | | | | Signed-off-by: Markus Thömmes <markusthoemmes@me.com>
* | System tests: confirm that -a and -l clashEd Santiago2021-10-20
|/ | | | | | ...and fix one instance where there was no check Signed-off-by: Ed Santiago <santiago@redhat.com>
* Add system test for shell completionPaul Holzinger2020-12-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There exists a unit test to ensure that shell completion functions are defined. However there was no check about the quality of the provided shell completions. Lets change that. The idea is to create a general test that makes sure we are suggesting containers,pods,images... for the correct commands. This works by reading the command use line and checking for each arg if we provide the correct suggestions for this arg. It includes the following tests: - flag suggestions if [options] is set - container, pod, image, network, volume, registry completion - path completion for the appropriate arg KEYWORDS (`PATH`,`CONTEXT`,etc.) - no completion if there are no args - completion for more than one arg if it ends with `...]` The test does not cover completion values for flags and not every arg KEYWORD is supported. This is still a huge improvement and covers most use cases. This test spotted several inconsistencies between the completion and the command use line. All of them have been adjusted to make the test pass. The biggest advantage is that the completions always match the latest command changes. So if someone changes the arguments for a command this ensures that the completions must be adjusted. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
* Switch use of Flags to OptionsDaniel J Walsh2020-10-21
| | | | | | | | Want to have man pages match commands, since we have lots of printed man pages with using Options, we will change the command line to use Options in --help. Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
* Usability: prevent "-l" with argumentsEd Santiago2020-09-14
| | | | | | | | | | | Add new system check confirming that "podman foo -l arg" throws an error; and fix lots of instances where code was not doing this check. I'll probably need to add something similar for --all but that can wait. Signed-off-by: Ed Santiago <santiago@redhat.com>
* system tests: enable more remote tests; cleanupEd Santiago2020-08-19
| | | | | | | | | | | | | | | | | | | | | | | | info, images, run, networking tests: remove some skip_if_remote()s that were added in the varlink days. All of these tests now seem to work with APIv2. help test: check that first output line from 'podman --help' is the program description (regression check for #7273). load test: clean up stray images, rewrite test to make it conform to existing convention. In the process, discover and file #7337 exec test (and networking): file #7360, and add FIXME comment to skip()s suggesting evaluating those tests once that is fixed. pod test: now that #6328 is fixed, use 'podman pod inspect --format' instead of relying on jq Various other tests: add an explanation of why test is disabled so we can more easily distinguish "this will never be meaningful under remote" vs "hey, doesn't work for now, but maybe someday". Signed-off-by: Ed Santiago <santiago@redhat.com>
* BATS help-message test: improve diagnosticsEd Santiago2020-07-21
| | | | | | | | | | | The error messages from the 'podman xxx --help' cross-check test are unhelpful, and cause much wasted time when they trigger. Solution: instead of using the built-in exit-status check in run_podman, do an explicit check outside of run_podman. This lets us die() with a custom, hopefully useful, message. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Docs: consistency between man / --helpEd Santiago2020-06-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | New functionality in hack/man-page-checker: start cross- referencing the man page 'Synopsis' line against the output of 'podman foo --help'. This is part 1, flag/option consistency. Part 2 (arg consistency) is too big and will have to wait for later. flag/option consistency means: if 'podman foo --help' includes the string '[flags]' in the Usage message, make sure the man page includes '[*options*]' in its Synopsis line, and vice-versa. This found several inconsistencies, which I've fixed. While doing this I realized that Cobra automatically includes a 'Flags:' subsection in its --help output for all subcommands that have defined flags. This is great - it lets us cross-check against the usage synopsis, and make sure that '[flags]' is present or absent as needed, without fear of human screwups. If a flag-less subcommand ever gets extended with flags, but the developer forgets to add '[flags]' and remove DisableFlagsInUseLine, we now have a test that will catch that. (This, too, caught two instances which I fixed). I don't actually know if the new man-page-checker functionality will work in CI: I vaguely recall that it might run before 'make podman' does; and also vaguely recall that some steps were taken to remedy that. Signed-off-by: Ed Santiago <santiago@redhat.com>
* BATS tests: new too-many-arguments testEd Santiago2020-06-23
| | | | | | | | | | | | | | | | | | | | | | | | | ...plus a few others. And fixes to actual parsing. If a command's usage message includes '...' in the argument list, assume it can take unlimited arguments. Nothing we can check. For all others, though, the ALL-CAPS part on the right-hand side of the usage message will define an upper bound on the number of arguments accepted by the command. So in our 'podman --help' test, generate N+1 args and run that command. We expect a 125 exit status and a suitably helpful error message. Not all podman commands or subcommands were checking, so I fixed that. And, fixed some broken usage messages (all-caps FLAGS, and '[flags]' at the end of 'ARGS'). Add new checks to the help test to prevent those in the future. Plus a little refactoring/cleanup where necessary. Signed-off-by: Ed Santiago <santiago@redhat.com>
* BATS help test: check usage stringEd Santiago2020-05-05
| | | | | | | | | | Now that we've agreed that usage messages should match what the user typed, confirm it. IOW 'podman foo --help' should not issue a usage message for 'podman container foo'. Fix one broken instance, 'unpause'. Signed-off-by: Ed Santiago <santiago@redhat.com>
* fix commands without inputbaude2020-04-30
| | | | | | in cases where commands require input and we dont provide it, we often would segv. This can be attributed in many cases to the subcommand not picked up the cobra Args attribute or neither had them. Signed-off-by: baude <bbaude@redhat.com>
* System tests: help messages: check required-argEd Santiago2020-04-30
| | | | | | | | | | | | | | | | | | | If a usage message is of the form '... [flags] ARGNAME', where ARGNAME is all-caps and not in brackets, it must be a required argument. Try running podman subcommand without ARGNAME, and make sure that podman bails out with an informative message. (Since this message is freeform in each subcommand, not Cobra-generated, we have a lot of possible variations to check for). Fix podman login/logout Use messages to indicate that REGISTRY is now optional (as of #5233). This test has actually been in place for over a year but due to a typo on my part -- a missing space -- it was not being run. "For want of a space, much testing was lost". Signed-off-by: Ed Santiago <santiago@redhat.com>
* stats: list all running containers unless specified otherwiseValentin Rothberg2019-10-23
| | | | | | | | | | Unless specified otherwise by --all, --latest or via arguments, list all running containers. This matches the behaviour of Docker and is also illustrated in the man pages where containers and options are marked to be optional. Fixes: #4274 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
* BATS tests: start supporting podman-remoteEd Santiago2019-04-10
| | | | | | | | | | | | | | podman-remote now supports rm! That's what we needed to start running BATS tests. Although most tests don't actually work, some do, and maybe the rest will start working over time. For now, disable them. The only significant difference found is that podman-remote strips fractional seconds from timestamps in JSON output. Probably not something worth caring about. Signed-off-by: Ed Santiago <santiago@redhat.com>
* BATS: new tests, and improvements to existing onesEd Santiago2019-03-18
| | | | | | | | | | | | | | | | | | | | | New: - podman exec - podman load (requires #2674) - CLI parsing (regression test for #2574) Improved: - help: test "podman NoSuchCommand", and subcommands - help: test "podman cmd" without required args - pod: start with --infra=false; this allows running rootless - log: also run 'logs' after container is run - log: test -f with two containers Also, use helpful descriptions for skip_if_rootless Tested on f29, root and rootless. As soon as podman-remote supports rm, I'll start testing that too. Signed-off-by: Ed Santiago <santiago@redhat.com>
* Implement review feedbackEd Santiago2019-03-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - document a recommended convention for fail-fast tests - document the requirement for jq. (And, add a fail-fast test for its presence; remove the duplicated checks in subtests) - add further sanity checks to 'help' test. Add missing documentation. Remove a no-longer-needed workaround for usage-message bug fixed in #2486 - add a documented TEMPLATE - and, since we're at 1.1, enable 'Remote API' check in version test - better diagnostics in setup/teardown; add vim filetype hint; better formatting of actual-vs-expect errors - new pod-top, logs, build tests - improve error messages - add $IMAGE alias for ridiculous $PODMAN_TEST_IMAGE_FQN - final cleanup, in prep for merge Signed-off-by: Ed Santiago <santiago@redhat.com>
* new system tests under BATSEd Santiago2019-03-07
Initial attempt at writing a framework for podman system tests. The idea is to define a useful set of primitives that will make it easy to write actual tests and to interpret results of failing ones. This is a proof-of-concept right now; only a small number of tests, by no means comprehensive. I am requesting review in order to find showstopper problems: reasons why this approach cannot work. Should there be none, we can work toward running these as gating tests for Fedora and RHEL8. Signed-off-by: Ed Santiago <santiago@redhat.com>