buildah-bud tests: simplify

Experience this week has shown that managing .diff files
is too difficult for humans, and too fragile. Opportunities
for errors abound. So, let's try to minimize the diffs.

We can't eliminate the diffs to helpers.bash: those are
true code changes that are absolutely required for running
tests using podman instead of buildah. We need to carry
those ourselves: they are not appropriate for the buildah
repo itself.

What we can do is simplify the patching of bud.bats. That
is fragile, because bud.bats changes often, and context-
sensitive git patch files can easily get confused.

Recognizing that the changes to bud.bats fall under two types:

  - tests that are skipped
  - tests in which podman error messages differ from buildah's

...we now have a new script, apply-podman-deltas, which
is (I hope) much user-friendlier. It understands two directives:

  errmsg - alter the expected error message
  skip   - skip a test

Both operate based on a bats test name. The test name must
match exactly. These directives use 'sed' to update bud.bats.
If any directive fails, the script will keep going (so you
get as many errors as possible in a run), then exits failure.

Instructions (README.md) now explain the process for dealing
with all expected test failures.

(Sneak checkin: add '--filter=NAME' option to test runner,
allowing for targeted and much shorter test runs).

Signed-off-by: Ed Santiago <santiago@redhat.com>

1 files changed, 3 insertions, 3 deletions

diff --git a/test/buildah-bud/make-new-buildah-diffs b/test/buildah-bud/make-new-buildah-diffs
index 1191f4597..11987e376 100644
--- a/test/buildah-bud/make-new-buildah-diffs
+++ b/test/buildah-bud/make-new-buildah-diffs
@@ -37,11 +37,11 @@ if [[ $n_commits -gt 1 ]]; then
     die "Please squash your commits"
 fi
 
-# Scope check: make sure the only files changed are under tests/
-changes=$(git diff --name-status [BASETAG]..HEAD | egrep -v '\stests/')
+# Scope check: make sure the only files changed is helpers.bash
+changes=$(git diff --name-status [BASETAG]..HEAD | egrep -v '\stests/helpers.bash')
 if [[ -n "$changes" ]]; then
     echo $changes
-    die "Found modified files other than under 'tests/'"
+    die "Found modified files other than 'tests/helpers.bash'"
 fi
 
 ###############################################################################