buildah-bud tests: simplify

Experience this week has shown that managing .diff files
is too difficult for humans, and too fragile. Opportunities
for errors abound. So, let's try to minimize the diffs.

We can't eliminate the diffs to helpers.bash: those are
true code changes that are absolutely required for running
tests using podman instead of buildah. We need to carry
those ourselves: they are not appropriate for the buildah
repo itself.

What we can do is simplify the patching of bud.bats. That
is fragile, because bud.bats changes often, and context-
sensitive git patch files can easily get confused.

Recognizing that the changes to bud.bats fall under two types:

  - tests that are skipped
  - tests in which podman error messages differ from buildah's

...we now have a new script, apply-podman-deltas, which
is (I hope) much user-friendlier. It understands two directives:

  errmsg - alter the expected error message
  skip   - skip a test

Both operate based on a bats test name. The test name must
match exactly. These directives use 'sed' to update bud.bats.
If any directive fails, the script will keep going (so you
get as many errors as possible in a run), then exits failure.

Instructions (README.md) now explain the process for dealing
with all expected test failures.

(Sneak checkin: add '--filter=NAME' option to test runner,
allowing for targeted and much shorter test runs).

Signed-off-by: Ed Santiago <santiago@redhat.com>

1 files changed, 47 insertions, 23 deletions

diff --git a/test/buildah-bud/README.md b/test/buildah-bud/README.md
index 88e4bbc3c..ba776c4d9 100644
--- a/test/buildah-bud/README.md
+++ b/test/buildah-bud/README.md
@@ -1,5 +1,4 @@
-buildah-bud tests under podman
-==============================
+# buildah-bud tests under podman
 
 This directory contains tools for running 'buildah bud' tests
 under podman. The key concept of the workflow is:
@@ -13,8 +12,7 @@ It's a teeny bit more complicated than that, but that's really most of
 what you need to know for most purposes. The tests run in podman CI,
 and for the most part are expected to just pass.
 
-Troubleshooting
----------------
+## Troubleshooting
 
 If you're reading this, it's probably because something went wrong.
 At the time of this writing (March 2021, initial commit) it is
@@ -26,8 +24,7 @@ my prediction is that they will fit one of two categories:
 
 Let's examine those in reverse order:
 
-Failure when not vendoring
---------------------------
+## Failure when not vendoring
 
 Aside from flakes, my only guess here is that you broke 'podman build'.
 If this is the case, it is very likely that you are aware of what you
@@ -41,13 +38,12 @@ If neither of those is the case, then I'm sorry, you're on your own.
 When you figure it out, please remember to update these instructions.
 
 
-Failure when vendoring new buildah
-----------------------------------
+## Failure when vendoring new buildah
 
 This is what I predict will be the usual case; and I predict that
 failures will fall into one of two bins:
 
-* failure to apply the patch
+* failure to apply the patches; and/or
 * failure because there are new buildah tests for functionality not in podman
 
 In either case, the process for solving is the same:
@@ -59,24 +55,52 @@ Presumably, something will fail here. Whatever the failure, your next step is:
 
 * `cd test-buildah-v<TAB>` (this is a new directory created by the script)
 
-If the failure was in `git am`, solve it (left as exercise for the reader).
+Now there are three possible failures:
 
-If the failure was in tests run, solve it (either by adding `skip`s to
-failing tests in bud.bats, or less preferably, by making other tweaks
-to the test code).
+### Failure in `git am`
 
-You now have modified files. THOSE SHOULD ONLY BE test/bud.bats or
-test/helpers.bash! If you changed any other file, that is a sign that
-something is very wrong!
+If the failure was in `git am`, it probably means that buildah
+`tests/helpers.bash` got updated in such a way as to cause a conflict
+with the patches we apply. Your best bet is to:
 
-Commit your changes: `git commit --all --amend`
+* Look at `tests/*.rej`
+* For each rejected patch, try to figure out where it should go and how to apply it. Do so.
+* `git add tests/helpers.bash` - this is for `git am`, next
+* `git am --continue` - this continues the failed patch. Make sure it succeeds.
+* `./make-new-buildah-diffs` - this updates your podman working directory
+* `cd ..; git diff test/buildah-bud`. This will show you a diff of a .diff file, which is really painful to read. I'm sorry. Just try to confirm that the changes look like what you expect.
 
-Push those changes to the podman repo: `./make-new-buildah-diffs`
+Proceed with 'In all cases' below.
 
-cd back up to the podman repo
+### Failure when applying podman-custom deltas
 
-As necessary, rerun `run-buildah-bud-tests`. You can use `--no-checkout`
-to run tests immediately, without rerunning the git checkout.
+Failure in the `apply-podman-deltas` script means that one of the
+hand-crafted exceptions was not found, e.g., there's a `skip` or
+`errmsg` looking for a specific `@test` in `bud.bats` that is
+no longer there.
 
-If you're happy with the diffs, `git add` the modified `.diff` file
-and submit it as part of your PR.
+Solution:
+* Inspect the error message(s) from `apply-podman-deltas`. Each message will list a specific `@test` name.
+* Look at the diffs in `tests/bud.bats` between master and your PR. (I'm really sorry; there's no quick easy command-line way to do that. You will need a checked-out buildah tree, and you will need to know the old and new buildah tags).
+  * In those diffs, look for changes related to each `@test` listed as an error. For example, a test being renamed or even removed.
+  * Update `test/buildah-bud/apply-podman-deltas` accordingly.
+
+Proceed with 'In all cases' below.
+
+### Failure when running tests
+
+If the failure was in tests run, and you're vendoring, your only real choice is to add a new `skip`:
+
+* Identify the failing test(s)
+* File a new podman issue, e.g. "podman build fails buildah XYZ test"
+* Edit `test/buildah/bud/apply-podman-deltas`. Search for "actual podman bugs" near the bottom, and add a new `skip` line with the reason (INCLUDE THE ISSUE NUMBER!) and the test name.
+
+### In all cases
+
+You will probably want to rerun `run-buildah-bud-tests` to save yourself
+the hassle of having it fail in CI. (`rm -rf test-buildah-v<TAB>` first).
+If you're debugging problems that run on a specific test, you can
+use `--filter="pattern"` to run only tests that match "pattern".
+
+If everything passes, `git commit --amend` your PR, adding the
+files you changed under `test/buildah-bud`, then `git push --force`.