Restart failed containers in tests

When we're waiting for a container to come up with healthchecks,
and it's not even running, there's no point to waiting further.
Instead, let's restart the container and continue waiting.

This may fix some flakes we're seeing with 'podman port' tests.
Then again, all the tests there seem to fail, not just a single
test flaking - so I bet there's some other underlying cause.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>

1 files changed, 13 insertions, 0 deletions

diff --git a/test/e2e/common_test.go b/test/e2e/common_test.go
index c3a37236b..21afc4b84 100644
--- a/test/e2e/common_test.go
+++ b/test/e2e/common_test.go
@@ -530,6 +530,19 @@ func (p *PodmanTestIntegration) RunHealthCheck(cid string) error {
 		if hc.ExitCode() == 0 {
 			return nil
 		}
+		// Restart container if it's not running
+		ps := p.Podman([]string{"ps", "--no-trunc", "--q", "--filter", fmt.Sprintf("id=%s", cid)})
+		ps.WaitWithDefaultTimeout()
+		if ps.ExitCode() == 0 {
+			if !strings.Contains(ps.OutputToString(), cid) {
+				fmt.Printf("Container %s is not running, restarting", cid)
+				restart := p.Podman([]string{"restart", cid})
+				restart.WaitWithDefaultTimeout()
+				if restart.ExitCode() != 0 {
+					return errors.Errorf("unable to restart %s", cid)
+				}
+			}
+		}
 		fmt.Printf("Waiting for %s to pass healthcheck\n", cid)
 		time.Sleep(1 * time.Second)
 	}