summaryrefslogtreecommitdiff
path: root/test/buildah-bud
Commit message (Collapse)AuthorAge
* buildah-bud tests: simplifyEd Santiago2021-04-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Experience this week has shown that managing .diff files is too difficult for humans, and too fragile. Opportunities for errors abound. So, let's try to minimize the diffs. We can't eliminate the diffs to helpers.bash: those are true code changes that are absolutely required for running tests using podman instead of buildah. We need to carry those ourselves: they are not appropriate for the buildah repo itself. What we can do is simplify the patching of bud.bats. That is fragile, because bud.bats changes often, and context- sensitive git patch files can easily get confused. Recognizing that the changes to bud.bats fall under two types: - tests that are skipped - tests in which podman error messages differ from buildah's ...we now have a new script, apply-podman-deltas, which is (I hope) much user-friendlier. It understands two directives: errmsg - alter the expected error message skip - skip a test Both operate based on a bats test name. The test name must match exactly. These directives use 'sed' to update bud.bats. If any directive fails, the script will keep going (so you get as many errors as possible in a run), then exits failure. Instructions (README.md) now explain the process for dealing with all expected test failures. (Sneak checkin: add '--filter=NAME' option to test runner, allowing for targeted and much shorter test runs). Signed-off-by: Ed Santiago <santiago@redhat.com>
* Update buildah-bud diffsEd Santiago2021-04-07
| | | | Signed-off-by: Ed Santiago <santiago@redhat.com>
* buildah-bud tests: handle go pseudoversions, plus...Ed Santiago2021-04-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Handle go pseudoversions, e.g. a custom non-released buildah used during testing of a PR. This will be something like: v1.20.1-0.20210402144408-36a37402d0c8 ...and it makes it impossible (AFAIK) to do a shallow checkout; we need to do a full clone of buildah, then git-checkout the SHA (last element of the long string above). FIXME: this is great for testing, but we almost certainly want some way to block this PR from merging, don't we? And, while testing this, found and fixed three bugs: - quote "$failhint" when echoing it on failure; otherwise we lose original whitespace. - invoke git-am with --reject! This makes it SO MUCH EASIER to identify the failing part of our patch! - sigh: generate the make-new-buildah-diffs helper *BEFORE* we try git-am! Otherwise, duh, if git-am fails we have no way to help the developer create a new diff file. Signed-off-by: Ed Santiago <santiago@redhat.com>
* buildah-bud tests: reenable pull-never testEd Santiago2021-03-29
| | | | | | | Issue #9573 (podman build --pull-never is a NOP) is fixed. Remove the 'skip' in the buildah-bud pull-never test. Signed-off-by: Ed Santiago <santiago@redhat.com>
* [NO TESTS NEEDED] Vendor in containers/buildah v1.20.0Daniel J Walsh2021-03-26
| | | | Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
* WIP: run buildah bud tests using podmanEd Santiago2021-03-21
Set of scripts to run buildah's bud.bats test using podman build in podman CI. podman build is not 100% compatible with buildah bud. In particular: * podman defaults to --layers=true; buildah to false * podman defaults to --force-rm=true; buildah to false * podman error exit status is 125; buildah is 2 * differences in error messages, command-line arguments Some of the above can be dealt with programmatically, by tweaking the buildah helpers.bash (BATS helpers). Some need to be tweaked by patching bud.bats itself. This PR includes a patch that will, I fear, need to be periodically maintained over time. There will likely be failures when vendoring in a new buildah, possibly because new tests were added for new features that don't exist in podman, possibly (I hope unlikely) if existing tests are changed in ways that make the patch file fail to apply. I've tried to write good instructions and to write the run script in such a way that it will offer helpful hints on failure. My instructions and code will be imperfect; I hope they will be good enough to merit continued use of this test (possibly with improvements to the instructions as we learn more about real-world failures). Signed-off-by: Ed Santiago <santiago@redhat.com>