WIP: run buildah bud tests using podman

Set of scripts to run buildah's bud.bats test using
podman build in podman CI.

podman build is not 100% compatible with buildah bud.
In particular:

  * podman defaults to --layers=true; buildah to false
  * podman defaults to --force-rm=true; buildah to false
  * podman error exit status is 125; buildah is 2
  * differences in error messages, command-line arguments

Some of the above can be dealt with programmatically,
by tweaking the buildah helpers.bash (BATS helpers).
Some need to be tweaked by patching bud.bats itself.
This PR includes a patch that will, I fear, need to
be periodically maintained over time.

There will likely be failures when vendoring in a
new buildah, possibly because new tests were added
for new features that don't exist in podman, possibly
(I hope unlikely) if existing tests are changed in
ways that make the patch file fail to apply. I've
tried to write good instructions and to write the run
script in such a way that it will offer helpful hints
on failure. My instructions and code will be imperfect;
I hope they will be good enough to merit continued use
of this test (possibly with improvements to the instructions
as we learn more about real-world failures).

Signed-off-by: Ed Santiago <santiago@redhat.com>

1 files changed, 82 insertions, 0 deletions

diff --git a/test/buildah-bud/README.md b/test/buildah-bud/README.md
new file mode 100644
index 000000000..88e4bbc3c
--- /dev/null
+++ b/test/buildah-bud/README.md
@@ -0,0 +1,82 @@
+buildah-bud tests under podman
+==============================
+
+This directory contains tools for running 'buildah bud' tests
+under podman. The key concept of the workflow is:
+
+* Pull buildah @ version specified in go.mod
+* Apply a small set of patches to buildah's tests directory, such that
+  * BATS will use 'podman build' instead of 'buildah bud'; and
+  * some not-applicable-under-podman tests are skipped
+
+It's a teeny bit more complicated than that, but that's really most of
+what you need to know for most purposes. The tests run in podman CI,
+and for the most part are expected to just pass.
+
+Troubleshooting
+---------------
+
+If you're reading this, it's probably because something went wrong.
+At the time of this writing (March 2021, initial commit) it is
+impossible to foresee what typical failures will look like, but
+my prediction is that they will fit one of two categories:
+
+* Failure when vendoring new buildah (e.g., by dependabot)
+* Other failure
+
+Let's examine those in reverse order:
+
+Failure when not vendoring
+--------------------------
+
+Aside from flakes, my only guess here is that you broke 'podman build'.
+If this is the case, it is very likely that you are aware of what you
+did; and if this is the case, your change likely falls into one of
+these two categories:
+
+* "OOPS! I didn't mean to break that". Solution: fix it!
+* "Uh, yeah, this is deliberate, and we choose to be incompatible with buildah". In this case, you'll need to skip or edit the failing test(s); see below.
+
+If neither of those is the case, then I'm sorry, you're on your own.
+When you figure it out, please remember to update these instructions.
+
+
+Failure when vendoring new buildah
+----------------------------------
+
+This is what I predict will be the usual case; and I predict that
+failures will fall into one of two bins:
+
+* failure to apply the patch
+* failure because there are new buildah tests for functionality not in podman
+
+In either case, the process for solving is the same:
+
+* Start with a checked-out podman tree with the failing PR applied
+* run `./test/buildah-bud/run-buildah-bud-tests`
+
+Presumably, something will fail here. Whatever the failure, your next step is:
+
+* `cd test-buildah-v<TAB>` (this is a new directory created by the script)
+
+If the failure was in `git am`, solve it (left as exercise for the reader).
+
+If the failure was in tests run, solve it (either by adding `skip`s to
+failing tests in bud.bats, or less preferably, by making other tweaks
+to the test code).
+
+You now have modified files. THOSE SHOULD ONLY BE test/bud.bats or
+test/helpers.bash! If you changed any other file, that is a sign that
+something is very wrong!
+
+Commit your changes: `git commit --all --amend`
+
+Push those changes to the podman repo: `./make-new-buildah-diffs`
+
+cd back up to the podman repo
+
+As necessary, rerun `run-buildah-bud-tests`. You can use `--no-checkout`
+to run tests immediately, without rerunning the git checkout.
+
+If you're happy with the diffs, `git add` the modified `.diff` file
+and submit it as part of your PR.