From 0162f678c0e68e9ef0756f8cf521cf14d637be29 Mon Sep 17 00:00:00 2001 From: Valentin Rothberg Date: Mon, 11 Apr 2022 13:32:23 +0200 Subject: benchmarking Podman: proof of concept MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a proof of concept for benchmarking Podman. The benchmarks are implemented by means of the end-to-end test suite but hidden behind a `benchmarks` build tag. Running `make localbenchmarks` will run `test/e2e` with the specific build tag and set ginkgo's "focus" to the specific "Podman Benchmark Suite" to only run this spec and skip all others. ginkgo will print a report before terminating listing the CPU and memory stats for each benchmark. New benchmarks can easily be added via the `newBenchmark` function that also supports adding an `init()` function to each benchmark which allows for performing certain setups for the specific benchmark. For instance, benchmarking `podman start` requires creating a container beforehand. Podman may be called more than once in the main function of a benchmark but note that the displayed memory consumption is then a sum of all Podman invocations. The memory consumption is collected via `/usr/bin/time`. A benchmark's report is split into CPU and memory as displayed below: ``` [CPU] podman images: Fastest Time: 0.146s Slowest Time: 0.187s Average Time: 0.180s ± 0.015s [MEM] podman images: Smallest: 41892.0KB Largest: 42792.0KB Average: 42380.7KB ± 286.4KB ``` Note that the benchmarks are not wired into the CI yet. They are meant as a proof of concept. More benchmarks and the plumbing into CI will happen in a later change. Signed-off-by: Valentin Rothberg --- Makefile | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) (limited to 'Makefile') diff --git a/Makefile b/Makefile index af9a2b7f6..e94d4cf73 100644 --- a/Makefile +++ b/Makefile @@ -553,8 +553,8 @@ test: localunit localintegration remoteintegration localsystem remotesystem ## .PHONY: ginkgo-run ginkgo-run: - $(GOBIN)/ginkgo version - $(GOBIN)/ginkgo -v $(TESTFLAGS) -tags "$(TAGS)" $(GINKGOTIMEOUT) -cover -flakeAttempts 3 -progress -trace -noColor -nodes 3 -debug test/e2e/. $(HACK) + ACK_GINKGO_RC=true $(GOBIN)/ginkgo version + ACK_GINKGO_RC=true $(GOBIN)/ginkgo -v $(TESTFLAGS) -tags "$(TAGS)" $(GINKGOTIMEOUT) -cover -flakeAttempts 3 -progress -trace -noColor -nodes 3 -debug test/e2e/. $(HACK) .PHONY: ginkgo ginkgo: @@ -570,6 +570,14 @@ localintegration: test-binaries ginkgo .PHONY: remoteintegration remoteintegration: test-binaries ginkgo-remote +.PHONY: localbenchmarks +localbenchmarks: test-binaries + ACK_GINKGO_RC=true $(GOBIN)/ginkgo \ + -focus "Podman Benchmark Suite" \ + -tags "$(BUILDTAGS) benchmarks" -noColor \ + -noisySkippings=false -noisyPendings=false \ + test/e2e/. + .PHONY: localsystem localsystem: # Wipe existing config, database, and cache: start with clean slate. -- cgit v1.2.3-54-g00ecf