buildah-bud tests: simplify

Experience this week has shown that managing .diff files
is too difficult for humans, and too fragile. Opportunities
for errors abound. So, let's try to minimize the diffs.

We can't eliminate the diffs to helpers.bash: those are
true code changes that are absolutely required for running
tests using podman instead of buildah. We need to carry
those ourselves: they are not appropriate for the buildah
repo itself.

What we can do is simplify the patching of bud.bats. That
is fragile, because bud.bats changes often, and context-
sensitive git patch files can easily get confused.

Recognizing that the changes to bud.bats fall under two types:

  - tests that are skipped
  - tests in which podman error messages differ from buildah's

...we now have a new script, apply-podman-deltas, which
is (I hope) much user-friendlier. It understands two directives:

  errmsg - alter the expected error message
  skip   - skip a test

Both operate based on a bats test name. The test name must
match exactly. These directives use 'sed' to update bud.bats.
If any directive fails, the script will keep going (so you
get as many errors as possible in a run), then exits failure.

Instructions (README.md) now explain the process for dealing
with all expected test failures.

(Sneak checkin: add '--filter=NAME' option to test runner,
allowing for targeted and much shorter test runs).

Signed-off-by: Ed Santiago <santiago@redhat.com>

1 files changed, 34 insertions, 4 deletions

diff --git a/test/buildah-bud/run-buildah-bud-tests b/test/buildah-bud/run-buildah-bud-tests
index 1265e67d5..b08b854ee 100755
--- a/test/buildah-bud/run-buildah-bud-tests
+++ b/test/buildah-bud/run-buildah-bud-tests
@@ -14,16 +14,25 @@ BASE_TAG=buildah-bud-in-podman
 # END   user-customizable section
 ###############################################################################
 
-usage="Usage: $ME [--help] [--no-checkout] [--no-test]
+usage="Usage: $ME [--help] [--no-checkout] [--no-test] [--filter=TESTNAME]
+
+Flags, useful for manual debugging:
+
+  --no-checkout   Skip checkout step, go directly to running tests
+  --no-test       Do checkout only, but do not run tests
+  --filter=NAME   Passed on to bats; runs only tests that match NAME
 "
 
 # Parse command-line options (used in development only, not in CI)
 do_checkout=y
 do_test=y
+declare -a bats_filter=()
 for i; do
+    value=$(expr "$i" : '[^=]*=\(.*\)')
     case "$i" in
         --no-checkout)  do_checkout= ; shift;;
         --no-test)      do_test=     ; shift;;
+        --filter=*)     bats_filter=("--filter" "$value"); shift;;
         -h|--help)      echo "$usage"; exit 0;;
         *)              echo "$ME: Unrecognized option '$i'" >&2; exit 1;;
     esac
@@ -150,12 +159,33 @@ if [[ -n $do_checkout ]]; then
     chmod 755 make-new-buildah-diffs
 
     # Apply custom patches. We do this _after_ building, although it shouldn't
-    # matter because these patches should only apply to test scripts.
+    # matter because these patches should only apply to test scripts and not
+    # to any buildah sources.
     failhint="
 Error applying patch file. This can happen when you vendor in a new buildah.
+You will want to:
 
-Look for '*.rej' files to resolve the conflict(s) manually."
+  - look for 'test/*.rej'
+  - resolve conflicts manually
+  - git add test/helpers.bash
+  - git am --continue
+  - ./make-new-buildah-diffs
+"
     (set -x;git am --reject <$PATCHES)
+
+    # Now apply our custom skips and error-message changes. This is maintained
+    # in a custom script, not a .diff file, because diffs are WAY too hard for
+    # humans to read and update.
+    APPLY=apply-podman-deltas
+    failhint="
+Error applying podman-specific deltas. This sometimes happens when you
+vendor in a new buildah. You will want to:
+
+  - inspect the errors shown above
+  - find the corresponding lines in $BUD_TEST_DIR_REL/$APPLY
+  - edit/delete them as necessary
+"
+    (set -x;$BUD_TEST_DIR/$APPLY)
 else
     # Called with --no-checkout
     test -d $buildah_dir || die "Called with --no-checkout, but $buildah_dir does not exist"
@@ -183,5 +213,5 @@ review the test failure and double-check your changes.
     (set -x;sudo env TMPDIR=/var/tmp \
                  PODMAN_BINARY=$PODMAN_BINARY \
                  BUILDAH_BINARY=$(pwd)/bin/buildah \
-                 bats tests/bud.bats)
+                 bats "${bats_filter[@]}" tests/bud.bats)
 fi