# buildah-bud tests under podman

This directory contains tools for running 'buildah bud' tests
under podman. The key concept of the workflow is:

* Pull buildah @ version specified in go.mod
* Apply a small set of patches to buildah's tests directory, such that
  * BATS will use 'podman build' instead of 'buildah bud'; and
  * some not-applicable-under-podman tests are skipped

It's a teeny bit more complicated than that, but that's really most of
what you need to know for most purposes. The tests run in podman CI,
and for the most part are expected to just pass.

## Troubleshooting

If you're reading this, it's probably because something went wrong.
At the time of this writing (March 2021, initial commit) it is
impossible to foresee what typical failures will look like, but
my prediction is that they will fit one of two categories:

* Failure when vendoring new buildah (e.g., by dependabot)
* Other failure

Let's examine those in reverse order:

## Failure when not vendoring

Aside from flakes, my only guess here is that you broke 'podman build'.
If this is the case, it is very likely that you are aware of what you
did; and if this is the case, your change likely falls into one of
these two categories:

* "OOPS! I didn't mean to break that". Solution: fix it!
* "Uh, yeah, this is deliberate, and we choose to be incompatible with buildah". In this case, you'll need to skip or edit the failing test(s); see below.

If neither of those is the case, then I'm sorry, you're on your own.
When you figure it out, please remember to update these instructions.


## Failure when vendoring new buildah

This is what I predict will be the usual case; and I predict that
failures will fall into one of two bins:

* failure to apply the patches; and/or
* failure because there are new buildah tests for functionality not in podman

In either case, the process for solving is the same:

* Start with a checked-out podman tree with the failing PR applied
* run `./test/buildah-bud/run-buildah-bud-tests`

Presumably, something will fail here. Whatever the failure, your next step is:

* `cd test-buildah-v<TAB>` (this is a new directory created by the script)

Now there are three possible failures:

### Failure in `git am`

If the failure was in `git am`, it probably means that buildah
`tests/helpers.bash` got updated in such a way as to cause a conflict
with the patches we apply. Your best bet is to:

* Look at `tests/*.rej`
* For each rejected patch, try to figure out where it should go and how to apply it. Do so.
* `git add tests/helpers.bash` - this is for `git am`, next
* `git am --continue` - this continues the failed patch. Make sure it succeeds.
* `./make-new-buildah-diffs` - this updates your podman working directory
* `cd ..; git diff test/buildah-bud`. This will show you a diff of a .diff file, which is really painful to read. I'm sorry. Just try to confirm that the changes look like what you expect.

Proceed with 'In all cases' below.

### Failure when applying podman-custom deltas

Failure in the `apply-podman-deltas` script means that one of the
hand-crafted exceptions was not found, e.g., there's a `skip` or
`errmsg` looking for a specific `@test` in `bud.bats` that is
no longer there.

Solution:
* Inspect the error message(s) from `apply-podman-deltas`. Each message will list a specific `@test` name.
* Look at the diffs in `tests/bud.bats` between master and your PR. (I'm really sorry; there's no quick easy command-line way to do that. You will need a checked-out buildah tree, and you will need to know the old and new buildah tags).
  * In those diffs, look for changes related to each `@test` listed as an error. For example, a test being renamed or even removed.
  * Update `test/buildah-bud/apply-podman-deltas` accordingly.

Proceed with 'In all cases' below.

### Failure when running tests

If the failure was in tests run, and you're vendoring, your only real choice is to add a new `skip`:

* Identify the failing test(s)
* File a new podman issue, e.g. "podman build fails buildah XYZ test"
* Edit `test/buildah/bud/apply-podman-deltas`. Search for "actual podman bugs" near the bottom, and add a new `skip` line with the reason (INCLUDE THE ISSUE NUMBER!) and the test name.

### In all cases

You will probably want to rerun `run-buildah-bud-tests` to save yourself
the hassle of having it fail in CI. (`rm -rf test-buildah-v<TAB>` first).
If you're debugging problems that run on a specific test, you can
use `--filter="pattern"` to run only tests that match "pattern".

If everything passes, `git commit --amend` your PR, adding the
files you changed under `test/buildah-bud`, then `git push --force`.