diff options
78 files changed, 810 insertions, 331 deletions
@@ -5,7 +5,7 @@ Podman (the POD MANager) is a tool for managing containers and images, volumes mounted into those containers, and pods made from groups of containers. Podman is based on libpod, a library for container lifecycle management that is also contained in this repository. The libpod library provides APIs for managing containers, pods, container images, and volumes. -* [Latest Version: 3.3.1](https://github.com/containers/podman/releases/latest) +* [Latest Version: 3.4.0](https://github.com/containers/podman/releases/latest) * Latest Remote client for Windows * Latest Remote client for macOS * Latest Static Remote client for Linux diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index b9b94dbb3..ef48df291 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,5 +1,102 @@ # Release Notes +## 3.4.0 +### Features +- Pods now support init containers! Init containers are containers which run before the rest of the pod starts. There are two types of init containers: "always", which always run before the pod is started, and "once", which only run the first time the pod starts and are subsequently removed. They can be added using the `podman create` command's `--init-ctr` option. +- Support for init containers has also been added to `podman play kube` and `podman generate kube` - init containers contained in Kubernetes YAML will be created as Podman init containers, and YAML generated by Podman will include any init containers created. +- The `podman play kube` command now supports building images. If the `--build` option is given and a directory with the name of the specified image exists in the current working directory and contains a valid Containerfile or Dockerfile, the image will be built and used for the container. +- The `podman play kube` command now supports a new option, `--teardown`, which removes any pods and containers created by the given Kubernetes YAML. +- The `podman generate kube` command now generates annotations for SELinux mount options on volume (`:z` and `:Z`) that are respected by the `podman play kube` command. +- A new command has been added, `podman pod logs`, to return logs for all containers in a pod at the same time. +- Two new commands have been added, `podman volume export` (to export a volume to a tar file) and `podman volume import`) (to populate a volume from a given tar file). +- The `podman auto-update` command now supports simple rollbacks. If a container fails to start after an automatic update, it will be rolled back to the previous image and restarted again. +- Pods now share their user namespace by default, and the `podman pod create` command now supports the `--userns` option. This allows rootless pods to be created with the `--userns=keep-id` option. +- The `podman pod ps` command now supports a new filter with its `--filter` option, `until`, which returns pods created before a given timestamp. +- The `podman image scp` command has been added. This command allows images to be transferred between different hosts. +- The `podman stats` command supports a new option, `--interval`, to specify the amount of time before the information is refreshed. +- The `podman inspect` command now includes ports exposed (but not published) by containers (e.g. ports from `--expose` when `--publish-all` is not specified). +- The `podman inspect` command now has a new boolean value, `Checkpointed`, which indicates that a container was stopped as a result of a `podman container checkpoint` operation. +- Volumes created by `podman volume create` now support setting quotas when run atop XFS. The `size` and `inode` options allow the maximum size and maximum number of inodes consumed by a volume to be limited. +- The `podman info` command now outputs information on what log drivers, network drivers, and volume plugins are available for use ([#11265](https://github.com/containers/podman/issues/11265)). +- The `podman info` command now outputs the current log driver in use, and the variant and codename of the distribution in use. +- The parameters of the VM created by `podman machine init` (amount of disk space, memory, CPUs) can now be set in `containers.conf`. +- The `podman machine ls` command now shows additional information (CPUs, memory, disk size) about VMs managed by `podman machine`. +- The `podman ps` command now includes healthcheck status in container state for containers that have healthchecks ([#11527](https://github.com/containers/podman/issues/11527)). + +### Changes +- The `podman build` command has a new alias, `podman buildx`, to improve compatibility with Docker. We have already added support for many `docker buildx` flags to `podman build` and aim to continue to do so. +- Cases where Podman is run without a user session or a writable temporary files directory will now produce better error messages. +- The default log driver has been changed from `file` to `journald`. The `file` driver did not properly support log rotation, so this should lead to a better experience. If journald is not available on the system, Podman will automatically revert to the `file`. +- Podman no longer depends on `ip` for removing networks ([#11403](https://github.com/containers/podman/issues/11403)). +- The deprecated `--macvlan` flag to `podman network create` now warns when it is used. It will be removed entirely in the Podman 4.0 release. +- The `podman machine start` command now prints a message when the VM is successfully started. +- The `podman stats` command can now be used on containers that are paused. +- The `podman unshare` command will now return the exit code of the command that was run in the user namespace (assuming the command was successfully run). +- Successful healthchecks will no longer add a `healthy` line to the system log to reduce log spam. +- As a temporary workaround for a lack of shortname prompts in the Podman remote client, VMs created by `podman machine` now default to only using the `docker.io` registry. + +### Bugfixes +- Fixed a bug where whitespace in the definition of sysctls (particularly default sysctls specified in `containers.conf`) would cause them to be parsed incorrectly. +- Fixed a bug where the Windows remote client improperly validated volume paths ([#10900](https://github.com/containers/podman/issues/10900)). +- Fixed a bug where the first line of logs from a container run with the `journald` log driver could be skipped. +- Fixed a bug where images created by `podman commit` did not include ports exposed by the container. +- Fixed a bug where the `podman auto-update` command would ignore the `io.containers.autoupdate.authfile` label when pulling images ([#11171](https://github.com/containers/podman/issues/11171)). +- Fixed a bug where the `--workdir` option to `podman create` and `podman run` could not be set to a directory where a volume was mounted ([#11352](https://github.com/containers/podman/issues/11352)). +- Fixed a bug where systemd socket-activation did not properly work with systemd-managed Podman containers ([#10443](https://github.com/containers/podman/issues/10443)). +- Fixed a bug where environment variable secrets added to a container were not available to exec sessions launched in the container. +- Fixed a bug where rootless containers could fail to start the `rootlessport` port-forwarding service when `XDG_RUNTIME_DIR` was set to a long path. +- Fixed a bug where arguments to the `--systemd` option to `podman create` and `podman run` were case-sensitive ([#11387](https://github.com/containers/podman/issues/11387)). +- Fixed a bug where the `podman manifest rm` command would also remove images referenced by the manifest, not just the manifest itself ([#11344](https://github.com/containers/podman/issues/11344)). +- Fixed a bug where the Podman remote client on OS X would not function properly if the `TMPDIR` environment variable was not set ([#11418](https://github.com/containers/podman/issues/11418)). +- Fixed a bug where the `/etc/hosts` file was not guaranteed to contain an entry for `localhost` (this is still not guaranteed if `--net=host` is used; such containers will exactly match the host's `/etc/hosts`) ([#11411](https://github.com/containers/podman/issues/11411)). +- Fixed a bug where the `podman machine start` command could print warnings about unsupported CPU features ([#11421](https://github.com/containers/podman/issues/11421)). +- Fixed a bug where the `podman info` command could segfault when accessing cgroup information. +- Fixed a bug where the `podman logs -f` command could hang when a container exited ([#11461](https://github.com/containers/podman/issues/11461)). +- Fixed a bug where the `podman generate systemd` command could not be used on containers that specified a restart policy ([#11438](https://github.com/containers/podman/issues/11438)). +- Fixed a bug where the remote Podman client's `podman build` command would fail to build containers if the UID and GID on the client were higher than 65536 ([#11474](https://github.com/containers/podman/issues/11474)). +- Fixed a bug where the remote Podman client's `podman build` command would fail to build containers if the context directory was a symlink ([#11732](https://github.com/containers/podman/issues/11732)). +- Fixed a bug where the `--network` flag to `podman play kube` was not properly parsed when a non-bridge network configuration was specified. +- Fixed a bug where the `podman inspect` command could error when the container being inspected was removed as it was being inspected ([#11392](https://github.com/containers/podman/issues/11392)). +- Fixed a bug where the `podman play kube` command ignored the default pod infra image specified in `containers.conf`. +- Fixed a bug where the `--format` option to `podman inspect` was nonfunctional under some circumstances ([#8785](https://github.com/containers/podman/issues/8785)). +- Fixed a bug where the remote Podman client's `podman run` and `podman exec` commands could skip a byte of output every 8192 bytes ([#11496](https://github.com/containers/podman/issues/11496)). +- Fixed a bug where the `podman stats` command would print nonsensical results if the container restarted while it was running ([#11469](https://github.com/containers/podman/issues/11469)). +- Fixed a bug where the remote Podman client would error when STDOUT was redirected on a Windows client ([#11444](https://github.com/containers/podman/issues/11444)). +- Fixed a bug where the `podman run` command could return 0 when the application in the container exited with 125 ([#11540](https://github.com/containers/podman/issues/11540)). +- Fixed a bug where containers with `--restart=always` set using the rootlessport port-forwarding service could not be restarted automatically. +- Fixed a bug where the `--cgroups=split` option to `podman create` and `podman run` was silently discarded if the container was part of a pod. +- Fixed a bug where the `podman container runlabel` command could fail if the image name given included a tag. +- Fixed a bug where Podman could add an extra `127.0.0.1` entry to `/etc/hosts` under some circumstances ([#11596](https://github.com/containers/podman/issues/11596)). +- Fixed a bug where the remote Podman client's `podman untag` command did not properly handle tags including a digest ([#11557](https://github.com/containers/podman/issues/11557)). +- Fixed a bug where the `--format` option to `podman ps` did not properly support the `table` argument for tabular output. +- Fixed a bug where the `--filter` option to `podman ps` did not properly handle filtering by healthcheck status ([#11687](https://github.com/containers/podman/issues/11687)). +- Fixed a bug where the `podman run` and `podman start --attach` commands could race when retrieving the exit code of a container that had already been removed resulting in an error (e.g. by an external `podman rm -f`) ([#11633](https://github.com/containers/podman/issues/11633)). +- Fixed a bug where the `podman generate kube` command would add default environment variables to generated YAML. +- Fixed a bug where the `podman generate kube` command would add the default CMD from the image to generated YAML ([#11672](https://github.com/containers/podman/issues/11672)). +- Fixed a bug where the `podman rm --storage` command could fail to remove containers under some circumstances ([#11207](https://github.com/containers/podman/issues/11207)). +- Fixed a bug where the `podman machine ssh` command could fail when run on Linux ([#11731](https://github.com/containers/podman/issues/11731)). +- Fixed a bug where the `podman stop` command would error when used on a container that was already stopped ([#11740](https://github.com/containers/podman/issues/11740)). +- Fixed a bug where renaming a container in a pod using the `podman rename` command, then removing the pod using `podman pod rm`, could cause Podman to believe the new name of the container was permanently in use, despite the container being removed ([#11750](https://github.com/containers/podman/issues/11750)). + +### API +- The Libpod Pull endpoint for Images now has a new query parameter, `quiet`, which (when set to true) suppresses image pull progress reports ([#10612](https://github.com/containers/podman/issues/10612)). +- The Compat Events endpoint now includes several deprecated fields from the Docker v1.21 API for improved compatibility with older clients. +- The Compat List and Inspect endpoints for Images now prefix image IDs with `sha256:` for improved Docker compatibility ([#11623](https://github.com/containers/podman/issues/11623)). +- The Compat Create endpoint for Containers now properly sets defaults for healthcheck-related fields ([#11225](https://github.com/containers/podman/issues/11225)). +- The Compat Create endpoint for Containers now supports volume options provided by the `Mounts` field ([#10831](https://github.com/containers/podman/issues/10831)). +- The Compat List endpoint for Secrets now supports a new query parameter, `filter`, which allows returned results to be filtered. +- The Compat Auth endpoint now returns the correct response code (500 instead of 400) when logging into a registry fails. +- The Version endpoint now includes information about the OCI runtime and Conmon in use ([#11227](https://github.com/containers/podman/issues/11227)). +- Fixed a bug where the X-Registry-Config header was not properly handled, leading to errors when pulling images ([#11235](https://github.com/containers/podman/issues/11235)). +- Fixed a bug where invalid query parameters could cause a null pointer dereference when creating error messages. +- Logging of API requests and responses at trace level has been greatly improved, including the addition of an X-Reference-Id header to correlate requests and responses ([#10053](https://github.com/containers/podman/issues/10053)). + +### Misc +- Updated Buildah to v1.23.0 +- Updated the containers/storage library to v1.36.0 +- Updated the containers/image library to v5.16.0 +- Updated the containers/common library to v0.44.0 + ## 3.3.1 ### Bugfixes - Fixed a bug where unit files created by `podman generate systemd` could not cleanup shut down containers when stopped by `systemctl stop` ([#11304](https://github.com/containers/podman/issues/11304)). diff --git a/cmd/podman/common/create.go b/cmd/podman/common/create.go index a969e17e9..e490fa121 100644 --- a/cmd/podman/common/create.go +++ b/cmd/podman/common/create.go @@ -421,7 +421,7 @@ func DefineCreateFlags(cmd *cobra.Command, cf *entities.ContainerCreateOptions, pidsLimitFlagName := "pids-limit" createFlags.Int64( pidsLimitFlagName, pidsLimit(), - "Tune container pids limit (set 0 for unlimited, -1 for server defaults)", + "Tune container pids limit (set -1 for unlimited)", ) _ = cmd.RegisterFlagCompletionFunc(pidsLimitFlagName, completion.AutocompleteNone) diff --git a/cmd/podman/containers/create.go b/cmd/podman/containers/create.go index 2593b4c44..bfeeb7ebe 100644 --- a/cmd/podman/containers/create.go +++ b/cmd/podman/containers/create.go @@ -235,6 +235,10 @@ func CreateInit(c *cobra.Command, vals entities.ContainerCreateOptions, isInfra if c.Flags().Changed("pids-limit") { val := c.Flag("pids-limit").Value.String() + // Convert -1 to 0, so that -1 maps to unlimited pids limit + if val == "-1" { + val = "0" + } pidsLimit, err := strconv.ParseInt(val, 10, 32) if err != nil { return vals, err diff --git a/contrib/cirrus/setup_environment.sh b/contrib/cirrus/setup_environment.sh index ef1f83024..1f594000f 100755 --- a/contrib/cirrus/setup_environment.sh +++ b/contrib/cirrus/setup_environment.sh @@ -256,9 +256,9 @@ case "$TEST_FLAVOR" in # Need to re-build lists (removed during image production) ooe.sh apt-get -qq -y update - msg "Installing previously downloaded/cached packages" + msg "Installing docker and containerd" # N/B: Tests check/expect `docker info` output, and this `!= podman info` - ooe.sh apt-get install --yes --no-download --ignore-missing containerd.io docker-ce docker-ce-cli + ooe.sh apt-get install --yes containerd.io docker-ce docker-ce-cli msg "Disabling docker service and socket activation" systemctl stop docker.service docker.socket diff --git a/docs/source/markdown/podman-create.1.md b/docs/source/markdown/podman-create.1.md index 0630c8be9..ee52bfd13 100644 --- a/docs/source/markdown/podman-create.1.md +++ b/docs/source/markdown/podman-create.1.md @@ -732,7 +732,7 @@ Default is to create a private PID namespace for the container #### **--pids-limit**=*limit* -Tune the container's pids limit. Set `0` to have unlimited pids for the container. (default "4096" on systems that support PIDS cgroups). +Tune the container's pids limit. Set `-1` to have unlimited pids for the container. (default "4096" on systems that support PIDS cgroups). #### **--platform**=*OS/ARCH* diff --git a/docs/source/markdown/podman-run.1.md b/docs/source/markdown/podman-run.1.md index 43b6d5cc6..5cc17f470 100644 --- a/docs/source/markdown/podman-run.1.md +++ b/docs/source/markdown/podman-run.1.md @@ -756,7 +756,7 @@ The default is to create a private PID namespace for the container. #### **--pids-limit**=*limit* -Tune the container's pids limit. Set to **0** to have unlimited pids for the container. The default is **4096** on systems that support "pids" cgroup controller. +Tune the container's pids limit. Set to **-1** to have unlimited pids for the container. The default is **4096** on systems that support "pids" cgroup controller. #### **--platform**=*OS/ARCH* @@ -17,7 +17,7 @@ require ( github.com/containers/image/v5 v5.16.0 github.com/containers/ocicrypt v1.1.2 github.com/containers/psgo v1.7.1 - github.com/containers/storage v1.36.0 + github.com/containers/storage v1.36.1-0.20210929132900-162a0bf730ce github.com/coreos/go-systemd/v22 v22.3.2 github.com/coreos/stream-metadata-go v0.0.0-20210225230131-70edb9eb47b3 github.com/cyphar/filepath-securejoin v0.2.3 @@ -41,7 +41,7 @@ require ( github.com/hpcloud/tail v1.0.0 github.com/json-iterator/go v1.1.12 github.com/mattn/go-colorable v0.1.8 // indirect - github.com/mattn/go-isatty v0.0.12 + github.com/mattn/go-isatty v0.0.14 github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 github.com/mrunalp/fileutils v0.5.0 github.com/onsi/ginkgo v1.16.4 @@ -267,8 +267,9 @@ github.com/containers/psgo v1.7.1 h1:2N6KADeFvBm1aI2iXxu6+/Xh7CCkdh8p8F3F/cpIU5I github.com/containers/psgo v1.7.1/go.mod h1:mWGpFzW73qWFA+blhF6l7GuKzbrACkYgr/ajiNQR+RM= github.com/containers/storage v1.23.5/go.mod h1:ha26Q6ngehFNhf3AWoXldvAvwI4jFe3ETQAf/CeZPyM= github.com/containers/storage v1.35.0/go.mod h1:qzYhasQP2/V9D9XdO+vRwkHBhsBO0oznMLzzRDQ8s20= -github.com/containers/storage v1.36.0 h1:OelxllCW19tnNngYuZw2ty/zLabVMG5rSs3KSwO1Lzc= github.com/containers/storage v1.36.0/go.mod h1:vbd3SKVQNHdmU5qQI6hTEcKPxnZkGqydG4f6uwrI5a8= +github.com/containers/storage v1.36.1-0.20210929132900-162a0bf730ce h1:6YOfANEWtL7+Q4RmnAfloGLIJNtt17MEHjvlHXz0vVY= +github.com/containers/storage v1.36.1-0.20210929132900-162a0bf730ce/go.mod h1:b7OGxODIyB3XpvCSWR91lllT9fv9DXeC8yfnaUocWJU= github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= @@ -590,8 +591,9 @@ github.com/klauspost/compress v1.11.0/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYs github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.13.4/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= -github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4= github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= +github.com/klauspost/compress v1.13.6 h1:P76CopJELS0TiO2mebmnzgWaajssP/EszplttgQxcgc= +github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -623,8 +625,9 @@ github.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= +github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= +github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI= github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk= @@ -19,6 +19,8 @@ $0 is a wrapper for invoking podman system tests. version of bats installed, runs with '--filter pattern' which runs only subtests that match 'pattern' + -T Passed on to bats, which will then show timing data + --help display usage message By default, tests ./bin/podman. To test a different podman, do: @@ -60,6 +62,8 @@ REMOTE= ROOT_ONLY= ROOTLESS_ONLY= +declare -a bats_opts=() + declare -a bats_filter=() for i;do @@ -69,6 +73,7 @@ for i;do --root) ROOT_ONLY=1 ;; --rootless) ROOTLESS_ONLY=1 ;; --remote) REMOTE=remote; echo "--remote is TBI"; exit 1;; + --ts|-T) bats_opts+=("-T") ;; */*.bats) TESTS=$i ;; *) if [[ $i =~ : ]]; then @@ -94,7 +99,7 @@ if [ -z "$ROOTLESS_ONLY" ]; then sudo --preserve-env=PODMAN \ --preserve-env=PODMAN_TEST_DEBUG \ --preserve-env=OCI_RUNTIME \ - bats "${bats_filter[@]}" $TESTS + bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS rc=$? fi @@ -102,7 +107,7 @@ fi echo "--------------------------------------------------" if [ -z "$ROOT_ONLY" ]; then echo "\$ bats ${bats_filter[@]} $TESTS" - bats "${bats_filter[@]}" $TESTS + bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS rc=$((rc | $?)) fi diff --git a/libpod/define/config.go b/libpod/define/config.go index 7a0d39e42..a5cf07afc 100644 --- a/libpod/define/config.go +++ b/libpod/define/config.go @@ -90,3 +90,6 @@ const ( // DefaultRlimitValue is the value set by default for nofile and nproc const RLimitDefaultValue = uint64(1048576) + +// BindMountPrefix distinguishes its annotations from others +const BindMountPrefix = "bind-mount-options:" diff --git a/libpod/kube.go b/libpod/kube.go index 57d99f3ef..bf86a9d16 100644 --- a/libpod/kube.go +++ b/libpod/kube.go @@ -241,11 +241,13 @@ func (p *Pod) podWithContainers(ctx context.Context, containers []*Container, po isInit := ctr.IsInitCtr() - ctr, volumes, _, err := containerToV1Container(ctx, ctr) + ctr, volumes, _, annotations, err := containerToV1Container(ctx, ctr) if err != nil { return nil, err } - + for k, v := range annotations { + podAnnotations[define.BindMountPrefix+k] = v + } // Since port bindings for the pod are handled by the // infra container, wipe them here. ctr.Ports = nil @@ -271,7 +273,7 @@ func (p *Pod) podWithContainers(ctx context.Context, containers []*Container, po deDupPodVolumes[vol.Name] = &vol } } else { - _, _, infraDNS, err := containerToV1Container(ctx, ctr) + _, _, infraDNS, _, err := containerToV1Container(ctx, ctr) if err != nil { return nil, err } @@ -359,17 +361,19 @@ func simplePodWithV1Containers(ctx context.Context, ctrs []*Container) (*v1.Pod, if !ctr.HostNetwork() { hostNetwork = false } - kubeCtr, kubeVols, ctrDNS, err := containerToV1Container(ctx, ctr) + kubeCtr, kubeVols, ctrDNS, annotations, err := containerToV1Container(ctx, ctr) if err != nil { return nil, err } + for k, v := range annotations { + kubeAnnotations[define.BindMountPrefix+k] = v + } if isInit { kubeInitCtrs = append(kubeInitCtrs, kubeCtr) } else { kubeCtrs = append(kubeCtrs, kubeCtr) } kubeVolumes = append(kubeVolumes, kubeVols...) - // Combine DNS information in sum'd structure if ctrDNS != nil { // nameservers @@ -415,42 +419,44 @@ func simplePodWithV1Containers(ctx context.Context, ctrs []*Container) (*v1.Pod, // containerToV1Container converts information we know about a libpod container // to a V1.Container specification. -func containerToV1Container(ctx context.Context, c *Container) (v1.Container, []v1.Volume, *v1.PodDNSConfig, error) { +func containerToV1Container(ctx context.Context, c *Container) (v1.Container, []v1.Volume, *v1.PodDNSConfig, map[string]string, error) { kubeContainer := v1.Container{} kubeVolumes := []v1.Volume{} + annotations := make(map[string]string) kubeSec, err := generateKubeSecurityContext(c) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } // NOTE: a privileged container mounts all of /dev/*. if !c.Privileged() && len(c.config.Spec.Linux.Devices) > 0 { // TODO Enable when we can support devices and their names kubeContainer.VolumeDevices = generateKubeVolumeDeviceFromLinuxDevice(c.config.Spec.Linux.Devices) - return kubeContainer, kubeVolumes, nil, errors.Wrapf(define.ErrNotImplemented, "linux devices") + return kubeContainer, kubeVolumes, nil, annotations, errors.Wrapf(define.ErrNotImplemented, "linux devices") } if len(c.config.UserVolumes) > 0 { - volumeMounts, volumes, err := libpodMountsToKubeVolumeMounts(c) + volumeMounts, volumes, localAnnotations, err := libpodMountsToKubeVolumeMounts(c) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, nil, err } + annotations = localAnnotations kubeContainer.VolumeMounts = volumeMounts kubeVolumes = append(kubeVolumes, volumes...) } envVariables, err := libpodEnvVarsToKubeEnvVars(c.config.Spec.Process.Env) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } portmappings, err := c.PortMappings() if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } ports, err := ocicniPortMappingToContainerPort(portmappings) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } // Handle command and arguments. @@ -469,11 +475,11 @@ func containerToV1Container(ctx context.Context, c *Container) (v1.Container, [] kubeContainer.Stdin = c.Stdin() img, _, err := c.runtime.libimageRuntime.LookupImage(image, nil) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } imgData, err := img.Inspect(ctx, false) if err != nil { - return kubeContainer, kubeVolumes, nil, err + return kubeContainer, kubeVolumes, nil, annotations, err } if reflect.DeepEqual(imgData.Config.Cmd, kubeContainer.Command) { kubeContainer.Command = nil @@ -555,7 +561,7 @@ func containerToV1Container(ctx context.Context, c *Container) (v1.Container, [] } dns.Options = dnsOptions } - return kubeContainer, kubeVolumes, &dns, nil + return kubeContainer, kubeVolumes, &dns, annotations, nil } // ocicniPortMappingToContainerPort takes an ocicni portmapping and converts @@ -606,16 +612,23 @@ func libpodEnvVarsToKubeEnvVars(envs []string) ([]v1.EnvVar, error) { } // libpodMountsToKubeVolumeMounts converts the containers mounts to a struct kube understands -func libpodMountsToKubeVolumeMounts(c *Container) ([]v1.VolumeMount, []v1.Volume, error) { +func libpodMountsToKubeVolumeMounts(c *Container) ([]v1.VolumeMount, []v1.Volume, map[string]string, error) { namedVolumes, mounts := c.sortUserVolumes(c.config.Spec) vms := make([]v1.VolumeMount, 0, len(mounts)) vos := make([]v1.Volume, 0, len(mounts)) + annotations := make(map[string]string) var suffix string for index, m := range mounts { + for _, opt := range m.Options { + if opt == "Z" || opt == "z" { + annotations[m.Source] = opt + break + } + } vm, vo, err := generateKubeVolumeMount(m) if err != nil { - return vms, vos, err + return vms, vos, annotations, err } // Name will be the same, so use the index as suffix suffix = fmt.Sprintf("-%d", index) @@ -629,7 +642,7 @@ func libpodMountsToKubeVolumeMounts(c *Container) ([]v1.VolumeMount, []v1.Volume vms = append(vms, vm) vos = append(vos, vo) } - return vms, vos, nil + return vms, vos, annotations, nil } // generateKubePersistentVolumeClaim converts a ContainerNamedVolume to a Kubernetes PersistentVolumeClaim diff --git a/libpod/reset.go b/libpod/reset.go index 96fa44c2f..7b25ed680 100644 --- a/libpod/reset.go +++ b/libpod/reset.go @@ -123,7 +123,7 @@ func (r *Runtime) Reset(ctx context.Context) error { if storageConfPath, err := storage.DefaultConfigFile(rootless.IsRootless()); err == nil { if _, err = os.Stat(storageConfPath); err == nil { fmt.Printf("A storage.conf file exists at %s\n", storageConfPath) - fmt.Println("You should remove this file if you did not modified the configuration.") + fmt.Println("You should remove this file if you did not modify the configuration.") } } else { if prevError != nil { diff --git a/libpod/runtime_cstorage.go b/libpod/runtime_cstorage.go index 58bd67e6d..5694967aa 100644 --- a/libpod/runtime_cstorage.go +++ b/libpod/runtime_cstorage.go @@ -121,7 +121,7 @@ func (r *Runtime) removeStorageContainer(idOrName string, force bool) error { } if err := r.store.DeleteContainer(ctr.ID); err != nil { - if errors.Cause(err) == storage.ErrContainerUnknown { + if errors.Cause(err) == storage.ErrNotAContainer || errors.Cause(err) == storage.ErrContainerUnknown { // Container again gone, no error logrus.Infof("Storage for container %s already removed", ctr.ID) return nil diff --git a/libpod/storage.go b/libpod/storage.go index ad78fe191..5c265df40 100644 --- a/libpod/storage.go +++ b/libpod/storage.go @@ -184,8 +184,12 @@ func (r *storageService) DeleteContainer(idOrName string) error { } err = r.store.DeleteContainer(container.ID) if err != nil { - logrus.Debugf("Failed to delete container %q: %v", container.ID, err) - return err + if errors.Cause(err) == storage.ErrNotAContainer || errors.Cause(err) == storage.ErrContainerUnknown { + logrus.Infof("Storage for container %s already removed", container.ID) + } else { + logrus.Debugf("Failed to delete container %q: %v", container.ID, err) + return err + } } return nil } diff --git a/pkg/domain/infra/abi/play.go b/pkg/domain/infra/abi/play.go index 35389ec5e..cf72a6253 100644 --- a/pkg/domain/infra/abi/play.go +++ b/pkg/domain/infra/abi/play.go @@ -319,8 +319,8 @@ func (ic *ContainerEngine) playKubePod(ctx context.Context, podName string, podY if err != nil { return nil, err } - specgenOpts := kube.CtrSpecGenOptions{ + Annotations: annotations, Container: initCtr, Image: pulledImage, Volumes: volumes, diff --git a/pkg/machine/qemu/machine.go b/pkg/machine/qemu/machine.go index d64dec07b..727b3cda4 100644 --- a/pkg/machine/qemu/machine.go +++ b/pkg/machine/qemu/machine.go @@ -392,7 +392,12 @@ func (v *MachineVM) Stop(name string, _ machine.StopOptions) error { logrus.Warn(err) } // Remove socket - return os.Remove(qemuSocketFile) + if err := os.Remove(qemuSocketFile); err != nil { + return err + } + + fmt.Printf("Successfully stopped machine: %s", name) + return nil } // NewQMPMonitor creates the monitor subsection of our vm diff --git a/pkg/specgen/generate/kube/kube.go b/pkg/specgen/generate/kube/kube.go index c01d7a1f0..9389b1a20 100644 --- a/pkg/specgen/generate/kube/kube.go +++ b/pkg/specgen/generate/kube/kube.go @@ -12,6 +12,7 @@ import ( "github.com/containers/common/pkg/parse" "github.com/containers/common/pkg/secrets" "github.com/containers/image/v5/manifest" + "github.com/containers/podman/v3/libpod/define" "github.com/containers/podman/v3/libpod/network/types" ann "github.com/containers/podman/v3/pkg/annotations" "github.com/containers/podman/v3/pkg/domain/entities" @@ -86,6 +87,8 @@ func ToPodOpt(ctx context.Context, podName string, p entities.PodCreateOptions, } type CtrSpecGenOptions struct { + // Annotations from the Pod + Annotations map[string]string // Container as read from the pod yaml Container v1.Container // Image available to use (pulled or found local) @@ -157,7 +160,7 @@ func ToSpecGen(ctx context.Context, opts *CtrSpecGenOptions) (*specgen.SpecGener return nil, errors.Wrap(err, "Failed to set CPU quota") } if milliCPU > 0 { - period, quota := util.CoresToPeriodAndQuota(float64(milliCPU) / 1000) + period, quota := util.CoresToPeriodAndQuota(float64(milliCPU)) s.ResourceLimits.CPU = &spec.LinuxCPU{ Quota: "a, Period: &period, @@ -289,6 +292,14 @@ func ToSpecGen(ctx context.Context, opts *CtrSpecGenOptions) (*specgen.SpecGener volume.MountPath = dest switch volumeSource.Type { case KubeVolumeTypeBindMount: + // If the container has bind mounts, we need to check if + // a selinux mount option exists for it + for k, v := range opts.Annotations { + // Make sure the z/Z option is not already there (from editing the YAML) + if strings.Replace(k, define.BindMountPrefix, "", 1) == volumeSource.Source && !util.StringInSlice("z", options) && !util.StringInSlice("Z", options) { + options = append(options, v) + } + } mount := spec.Mount{ Destination: volume.MountPath, Source: volumeSource.Source, diff --git a/pkg/specgen/generate/validate.go b/pkg/specgen/generate/validate.go index 50efe7fa3..b0d84825e 100644 --- a/pkg/specgen/generate/validate.go +++ b/pkg/specgen/generate/validate.go @@ -72,10 +72,9 @@ func verifyContainerResourcesCgroupV1(s *specgen.SpecGenerator) ([]string, error // Pids checks if s.ResourceLimits.Pids != nil { - pids := s.ResourceLimits.Pids // TODO: Should this be 0, or checking that ResourceLimits.Pids // is set at all? - if pids.Limit > 0 && !sysInfo.PidsLimit { + if s.ResourceLimits.Pids.Limit >= 0 && !sysInfo.PidsLimit { warnings = append(warnings, "Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded.") s.ResourceLimits.Pids = nil } diff --git a/pkg/specgen/specgen.go b/pkg/specgen/specgen.go index 7aa27487a..70b2aa1ef 100644 --- a/pkg/specgen/specgen.go +++ b/pkg/specgen/specgen.go @@ -411,7 +411,7 @@ type ContainerNetworkConfig struct { // Expose is a number of ports that will be forwarded to the container // if PublishExposedPorts is set. // Expose is a map of uint16 (port number) to a string representing - // protocol. Allowed protocols are "tcp", "udp", and "sctp", or some + // protocol i.e map[uint16]string. Allowed protocols are "tcp", "udp", and "sctp", or some // combination of the three separated by commas. // If protocol is set to "" we will assume TCP. // Only available if NetNS is set to Bridge or Slirp, and diff --git a/test/e2e/generate_kube_test.go b/test/e2e/generate_kube_test.go index cb556991c..cb987e139 100644 --- a/test/e2e/generate_kube_test.go +++ b/test/e2e/generate_kube_test.go @@ -6,6 +6,8 @@ import ( "path/filepath" "strconv" + "github.com/containers/podman/v3/libpod/define" + "github.com/containers/podman/v3/pkg/util" . "github.com/containers/podman/v3/test/utils" "github.com/ghodss/yaml" @@ -555,6 +557,15 @@ var _ = Describe("Podman generate kube", func() { kube.WaitWithDefaultTimeout() Expect(kube).Should(Exit(0)) + b, err := ioutil.ReadFile(outputFile) + Expect(err).ShouldNot(HaveOccurred()) + pod := new(v1.Pod) + err = yaml.Unmarshal(b, pod) + Expect(err).To(BeNil()) + val, found := pod.Annotations[define.BindMountPrefix+vol1] + Expect(found).To(BeTrue()) + Expect(val).To(HaveSuffix("z")) + rm := podmanTest.Podman([]string{"pod", "rm", "-f", "test1"}) rm.WaitWithDefaultTimeout() Expect(rm).Should(Exit(0)) diff --git a/test/e2e/play_kube_test.go b/test/e2e/play_kube_test.go index 0d5b9d52c..83ce751e6 100644 --- a/test/e2e/play_kube_test.go +++ b/test/e2e/play_kube_test.go @@ -2320,6 +2320,39 @@ MemoryReservation: {{ .HostConfig.MemoryReservation }}`}) } }) + It("podman play kube allows setting resource limits with --cpus 1", func() { + SkipIfContainerized("Resource limits require a running systemd") + SkipIfRootless("CPU limits require root") + podmanTest.CgroupManager = "systemd" + + var ( + expectedCpuLimit string = "1" + ) + + deployment := getDeployment( + withPod(getPod(withCtr(getCtr( + withCpuLimit(expectedCpuLimit), + ))))) + err := generateKubeYaml("deployment", deployment, kubeYaml) + Expect(err).To(BeNil()) + + kube := podmanTest.Podman([]string{"play", "kube", kubeYaml}) + kube.WaitWithDefaultTimeout() + Expect(kube).Should(Exit(0)) + + for _, pod := range getPodNamesInDeployment(deployment) { + inspect := podmanTest.Podman([]string{"inspect", getCtrNameInPod(&pod), "--format", `{{ .HostConfig.CpuPeriod }}:{{ .HostConfig.CpuQuota }}`}) + + inspect.WaitWithDefaultTimeout() + Expect(inspect).Should(Exit(0)) + + parts := strings.Split(strings.Trim(inspect.OutputToString(), "\n"), ":") + Expect(parts).To(HaveLen(2)) + + Expect(parts[0]).To(Equal(parts[1])) + } + }) + It("podman play kube reports invalid image name", func() { invalidImageName := "./myimage" diff --git a/test/system/001-basic.bats b/test/system/001-basic.bats index 888c075b8..2e5ebe4a3 100644 --- a/test/system/001-basic.bats +++ b/test/system/001-basic.bats @@ -101,7 +101,9 @@ function setup() { fi run_podman 125 --remote - is "$output" "Error: missing command 'podman COMMAND'" "podman remote show usage message without running endpoint" + is "$output" "Error: missing command 'podman COMMAND' +Try 'podman --help' for more information." \ + "podman --remote show usage message without running endpoint" } # This is for development only; it's intended to make sure our timeout diff --git a/test/system/005-info.bats b/test/system/005-info.bats index 0ea0f8356..045045fa6 100644 --- a/test/system/005-info.bats +++ b/test/system/005-info.bats @@ -37,7 +37,7 @@ cgroupVersion: v[12] # FIXME: if we're ever able to get package versions on Debian, # add '-[0-9]' to all '*.package' queries below. tests=" -host.buildahVersion | [0-9.] +host.buildahVersion | [1-9][0-9]*\.[0-9.]\\\+.* host.conmon.path | $expr_path host.conmon.package | .*conmon.* host.cgroupManager | \\\(systemd\\\|cgroupfs\\\) diff --git a/test/system/010-images.bats b/test/system/010-images.bats index 649987dfd..1e9d5f181 100644 --- a/test/system/010-images.bats +++ b/test/system/010-images.bats @@ -19,8 +19,8 @@ load helpers @test "podman images - custom formats" { tests=" -{{.ID}} | [0-9a-f]\\\{12\\\} -{{.ID| upper}} | [0-9A-F]\\\{12\\\} +{{.ID}} | [0-9a-f]\\\{12\\\}\\\$ +{{.ID| upper}} | [0-9A-F]\\\{12\\\}\\\$ {{.Repository}}:{{.Tag}} | $PODMAN_TEST_IMAGE_FQN {{.Labels.created_by}} | test/system/build-testimage {{.Labels.created_at}} | 20[0-9-]\\\+T[0-9:]\\\+Z @@ -28,7 +28,7 @@ load helpers parse_table "$tests" | while read fmt expect; do run_podman images --format "$fmt" - is "$output" "$expect\$" "podman images $fmt" + is "$output" "$expect" "podman images --format '$fmt'" done run_podman images --format "{{.ID}}" --no-trunc diff --git a/test/system/030-run.bats b/test/system/030-run.bats index 3d9d834b3..4080f08b4 100644 --- a/test/system/030-run.bats +++ b/test/system/030-run.bats @@ -517,7 +517,8 @@ json-file | f cid="$output" run_podman inspect --format "{{.ImageName}}" $cid - is "$output" "$newtag" "container .ImageName is the container-create name" + is "$output" "$newtag:latest" \ + "container .ImageName is the container-create name" # Same thing, but now with a :tag, and making sure it works with --name newtag2="${newtag}:$(random_string 6|tr A-Z a-z)" @@ -526,7 +527,8 @@ json-file | f cname="$(random_string 14|tr A-Z a-z)" run_podman create --name $cname $newtag2 run_podman inspect --format "{{.ImageName}}" $cname - is "$output" "$newtag2" "container .ImageName is the container-create name" + is "$output" "$newtag2" \ + "container .ImageName is the container-create name, with :tag" # Clean up. run_podman rm $cid $cname @@ -718,7 +720,7 @@ EOF run_podman 125 run --device-cgroup-rule="b 7:2" --rm $IMAGE is "$output" 'Error: invalid device cgroup rule requires type, major:Minor, and access rules: "b 7:2"' run_podman 125 run --device-cgroup-rule="x 7:* rmw" --rm $IMAGE - is "$output" "Error: invalid device type in device-access-add:" + is "$output" "Error: invalid device type in device-access-add: x" run_podman 125 run --device-cgroup-rule="a a:* rmw" --rm $IMAGE is "$output" "Error: strconv.ParseInt: parsing \"a\": invalid syntax" } diff --git a/test/system/075-exec.bats b/test/system/075-exec.bats index b7367d153..de767a1e8 100644 --- a/test/system/075-exec.bats +++ b/test/system/075-exec.bats @@ -77,7 +77,7 @@ load helpers is "${lines[1]}" "3000+0 records out" "dd: number of records out" # Verify sha. '% *' strips off the path, keeping only the SHA run_podman exec $cid sha512sum /tmp/bigfile - is "${output% *}" "$expect" "SHA of file in container" + is "${output% *}" "$expect " "SHA of file in container" # Clean up run_podman exec $cid touch /stop diff --git a/test/system/120-load.bats b/test/system/120-load.bats index 8be9ed5c5..e9959271f 100644 --- a/test/system/120-load.bats +++ b/test/system/120-load.bats @@ -11,7 +11,7 @@ load helpers # initialize, read image ID and name get_iid_and_name() { run_podman images -a --format '{{.ID}} {{.Repository}}:{{.Tag}}' - read iid img_name < <(echo "$output") + read iid img_name <<<"$output" archive=$PODMAN_TMPDIR/myimage-$(random_string 8).tar } @@ -62,7 +62,7 @@ verify_iid_and_name() { # FIXME: cannot compare IID, see #7371, so we check only the tag run_podman images $fqin --format '{{.Repository}}:{{.Tag}}' - is "$output" "$fqin" "image preserves name across save/load" + is "${lines[0]}" "$fqin" "image preserves name across save/load" # Load with a new tag local new_name=x1$(random_string 14 | tr A-Z a-z) diff --git a/test/system/125-import.bats b/test/system/125-import.bats index 5995d71bf..5b8d84a2f 100644 --- a/test/system/125-import.bats +++ b/test/system/125-import.bats @@ -20,26 +20,26 @@ load helpers # Simple import run_podman import -q $archive iid="$output" - run_podman run -t --rm $iid cat /random.txt + run_podman run --rm $iid cat /random.txt is "$output" "$random_content" "simple import" run_podman rmi -f $iid # Simple import via stdin run_podman import -q - < <(cat $archive) iid="$output" - run_podman run -t --rm $iid cat /random.txt + run_podman run --rm $iid cat /random.txt is "$output" "$random_content" "simple import via stdin" run_podman rmi -f $iid # Tagged import run_podman import -q $archive $fqin - run_podman run -t --rm $fqin cat /random.txt + run_podman run --rm $fqin cat /random.txt is "$output" "$random_content" "tagged import" run_podman rmi -f $fqin # Tagged import via stdin run_podman import -q - $fqin < <(cat $archive) - run_podman run -t --rm $fqin cat /random.txt + run_podman run --rm $fqin cat /random.txt is "$output" "$random_content" "tagged import via stdin" run_podman rmi -f $fqin } @@ -100,7 +100,7 @@ EOF # Confirm exit within timeout run_podman ps -a --filter name=$a_cnt --format '{{.Status}}' - is "$output" "Exited (33)" "Exit by non-TERM/KILL" + is "$output" "Exited (33) .*" "Exit by non-TERM/KILL" run_podman rm -f $a_cnt run_podman rmi $b_img $a_img diff --git a/test/system/150-login.bats b/test/system/150-login.bats index ed925044c..c003a0409 100644 --- a/test/system/150-login.bats +++ b/test/system/150-login.bats @@ -245,7 +245,7 @@ function _test_skopeo_credential_sharing() { is "$status" "0" "skopeo inspect - exit status" got_name=$(jq -r .Name <<<"$output") - is "$got_name" "$registry/$dest_name" "skopeo inspect -> Name" + is "$got_name" "$registry/$destname" "skopeo inspect -> Name" # Now try without a valid login; it should fail run_podman logout "$@" $registry diff --git a/test/system/160-volumes.bats b/test/system/160-volumes.bats index e21be9ea4..c02525e0d 100644 --- a/test/system/160-volumes.bats +++ b/test/system/160-volumes.bats @@ -191,7 +191,7 @@ EOF run_podman run --rm -v my_vol:/data $IMAGE sh -c "echo hello >> /data/test" run_podman volume create my_vol2 - tarfile=hello$(random_string | tr A-Z a-z).tar + tarfile=${PODMAN_TMPDIR}/hello$(random_string | tr A-Z a-z).tar run_podman volume export my_vol --output=$tarfile # we want to use `run_podman volume export my_vol` but run_podman is wrapping EOF run_podman volume import my_vol2 - < $tarfile @@ -261,7 +261,8 @@ EOF # prune should remove v4 run_podman volume prune --force - is "$output" "${v[4]}" "volume prune, with 1, 2, 3 in use, deletes only 4" + is "$(echo $(sort <<<$output))" "${v[4]} ${v[5]} ${v[6]}" \ + "volume prune, with 1, 2, 3 in use, deletes only 4, 5, 6" # Remove the container using v2 and v3. Prune should now remove those. # The 'echo sort' is to get the output sorted and in one line. diff --git a/test/system/260-sdnotify.bats b/test/system/260-sdnotify.bats index 0dae569a8..395e6f94f 100644 --- a/test/system/260-sdnotify.bats +++ b/test/system/260-sdnotify.bats @@ -70,7 +70,7 @@ function _stop_socat() { # Check that MAINPID=xxxxx points to a running conmon process function _assert_mainpid_is_conmon() { - local mainpid=$(expr "$1" : "MAINPID=\([0-9]\+\)") + local mainpid=$(expr "$1" : ".*MAINPID=\([0-9]\+\)") test -n "$mainpid" || die "Could not parse '$1' as 'MAINPID=nnnn'" test -d /proc/$mainpid || die "sdnotify MAINPID=$mainpid - but /proc/$mainpid does not exist" @@ -121,7 +121,7 @@ function _assert_mainpid_is_conmon() { # we look for READY=1 _anywhere_ in the output, not just the last line. is "$output" ".*READY=1.*" "sdnotify sent READY=1" - _assert_mainpid_is_conmon "${lines[0]}" + _assert_mainpid_is_conmon "$output" # Done. Stop container, clean up. run_podman exec $cid touch /stop @@ -163,7 +163,7 @@ function _assert_mainpid_is_conmon() { is "$output" ".*READY=1" "received READY=1 through notify socket" - _assert_mainpid_is_conmon "${lines[0]}" + _assert_mainpid_is_conmon "$output" # Done. Stop container, clean up. run_podman exec $cid touch /stop diff --git a/test/system/330-corrupt-images.bats b/test/system/330-corrupt-images.bats index 86da06cb0..7f2b81835 100644 --- a/test/system/330-corrupt-images.bats +++ b/test/system/330-corrupt-images.bats @@ -74,7 +74,7 @@ function _corrupt_image_test() { # Corruptify, and confirm that 'podman images' throws an error rm -v ${PODMAN_CORRUPT_TEST_WORKDIR}/root/*-images/$id/${rm_path} run_podman 125 images - is "$output" "Error: error retrieving label for image \"$id\": you may need to remove the image to resolve the error" + is "$output" "Error: error retrieving label for image \"$id\": you may need to remove the image to resolve the error.*" # Run the requested command. Confirm it succeeds, with suitable warnings run_podman $* diff --git a/test/system/410-selinux.bats b/test/system/410-selinux.bats index 5ee0e0715..7f7f23000 100644 --- a/test/system/410-selinux.bats +++ b/test/system/410-selinux.bats @@ -29,7 +29,7 @@ function check_label() { if [ -n "$2" ]; then # e.g. from the above example -> "s0:c45,c745" range=$(cut -d: -f4,5 <<<"$context") - is "$range" "$2" "SELinux range" + is "$range" "$2^@" "SELinux range" fi } @@ -101,7 +101,7 @@ function check_label() { --security-opt label=level:s0 \ $IMAGE sh -c 'while test ! -e /stop; do sleep 0.1; done' run_podman inspect --format='{{ .HostConfig.SecurityOpt }}' myc - is "$output" "\[label=type:spc_t,label=level:s0 seccomp=unconfined]" \ + is "$output" "[label=type:spc_t,label=level:s0 seccomp=unconfined]" \ "'podman inspect' preserves all --security-opts" run_podman exec myc touch /stop @@ -113,6 +113,14 @@ function check_label() { @test "podman selinux: shared context in (some) namespaces" { skip_if_no_selinux + # rootless users have no usable cgroups with cgroupsv1, so containers + # must use a pid namespace and not join an existing one. + skip_if_rootless_cgroupsv1 + + if [[ $(podman_runtime) == "runc" ]]; then + skip "some sort of runc bug, not worth fixing (#11784)" + fi + run_podman run -d --name myctr $IMAGE top run_podman exec myctr cat -v /proc/self/attr/current context_c1="$output" @@ -221,24 +229,25 @@ function check_label() { run_podman run -v $tmpdir:/test $IMAGE cat /proc/self/attr/current run ls -dZ ${tmpdir} - is "$output" ${LABEL} "No Relabel Correctly" + is "$output" "${LABEL} ${tmpdir}" "No Relabel Correctly" run_podman run -v $tmpdir:/test:z --security-opt label=disable $IMAGE cat /proc/self/attr/current run ls -dZ $tmpdir - is "$output" ${RELABEL} "Privileged Relabel Correctly" + is "$output" "${RELABEL} $tmpdir" "Privileged Relabel Correctly" run_podman run -v $tmpdir:/test:z --privileged $IMAGE cat /proc/self/attr/current run ls -dZ $tmpdir - is "$output" ${RELABEL} "Privileged Relabel Correctly" + is "$output" "${RELABEL} $tmpdir" "Privileged Relabel Correctly" run_podman run -v $tmpdir:/test:Z $IMAGE cat /proc/self/attr/current level=$(secon -l $output) run ls -dZ $tmpdir - is "$output" "system_u:object_r:container_file_t:$level" "Confined Relabel Correctly" + is "$output" "system_u:object_r:container_file_t:$level $tmpdir" \ + "Confined Relabel Correctly" run_podman run -v $tmpdir:/test:z $IMAGE cat /proc/self/attr/current run ls -dZ $tmpdir - is "$output" ${RELABEL} "Shared Relabel Correctly" + is "$output" "${RELABEL} $tmpdir" "Shared Relabel Correctly" } # vim: filetype=sh diff --git a/test/system/420-cgroups.bats b/test/system/420-cgroups.bats index 89c81a742..025a20012 100644 --- a/test/system/420-cgroups.bats +++ b/test/system/420-cgroups.bats @@ -8,9 +8,7 @@ load helpers @test "podman run, preserves initial --cgroup-manager" { skip_if_remote "podman-remote does not support --cgroup-manager" - if is_rootless && is_cgroupsv1; then - skip "not supported as rootless under cgroups v1" - fi + skip_if_rootless_cgroupsv1 # Find out our default cgroup manager, and from that, get the non-default run_podman info --format '{{.Host.CgroupManager}}' diff --git a/test/system/450-interactive.bats b/test/system/450-interactive.bats index 47bdff9ab..55c2afcd4 100644 --- a/test/system/450-interactive.bats +++ b/test/system/450-interactive.bats @@ -55,9 +55,11 @@ function teardown() { cols=$(( 15 + RANDOM % 60 & 126 )) stty rows $rows cols $cols <$PODMAN_TEST_PTY + CR=$'\r' + # ...and make sure stty under podman reads that. run_podman run -it --name mystty $IMAGE stty size <$PODMAN_TEST_PTY - is "$output" "$rows $cols" "stty under podman run reads the correct dimensions" + is "$output" "$rows $cols$CR" "stty under podman run reads the correct dimensions" run_podman rm -f mystty @@ -75,7 +77,7 @@ function teardown() { @test "podman load - will not read from tty" { run_podman 125 load <$PODMAN_TEST_PTY is "$output" \ - "Error: cannot read from terminal. Use command-line redirection" \ + "Error: cannot read from terminal. Use command-line redirection or the --input flag." \ "Diagnostic from 'podman load' without redirection or -i" } @@ -84,14 +86,15 @@ function teardown() { run_podman run --tty -i --rm $IMAGE echo hello < /dev/null is "$output" ".*The input device is not a TTY.*" "-it _without_ a tty" + CR=$'\r' run_podman run --tty -i --rm $IMAGE echo hello <$PODMAN_TEST_PTY - is "$output" "hello" "-it _with_ a pty" + is "$output" "hello$CR" "-it _with_ a pty" run_podman run --tty=false -i --rm $IMAGE echo hello < /dev/null is "$output" "hello" "-tty=false: no warning" run_podman run --tty -i=false --rm $IMAGE echo hello < /dev/null - is "$output" "hello" "-i=false: no warning" + is "$output" "hello$CR" "-i=false: no warning" } # vim: filetype=sh diff --git a/test/system/500-networking.bats b/test/system/500-networking.bats index 548f2d764..29fa309f3 100644 --- a/test/system/500-networking.bats +++ b/test/system/500-networking.bats @@ -417,7 +417,7 @@ load helpers # check network alias for container short id run_podman inspect $cid --format "{{(index .NetworkSettings.Networks \"$netname\").Aliases}}" - is "$output" "\[${cid:0:12}\]" "short container id in network aliases" + is "$output" "[${cid:0:12}]" "short container id in network aliases" run_podman network disconnect $netname $cid @@ -449,7 +449,7 @@ load helpers # check network2 alias for container short id run_podman inspect $cid --format "{{(index .NetworkSettings.Networks \"$netname2\").Aliases}}" - is "$output" "\[${cid:0:12}\]" "short container id in network aliases" + is "$output" "[${cid:0:12}]" "short container id in network aliases" # curl should work run curl --max-time 3 -s $SERVER/index.txt diff --git a/test/system/700-play.bats b/test/system/700-play.bats index 0785bffdf..f41c50e4c 100644 --- a/test/system/700-play.bats +++ b/test/system/700-play.bats @@ -30,7 +30,7 @@ spec: containers: - command: - sleep - - "100" + - \"100\" env: - name: PATH value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin @@ -49,7 +49,7 @@ spec: capabilities: {} privileged: false seLinuxOptions: - level: "s0:c1,c2" + level: \"s0:c1,c2\" readOnlyRootFilesystem: false volumeMounts: - mountPath: /testdir:z @@ -73,7 +73,7 @@ RELABEL="system_u:object_r:container_file_t:s0" run_podman play kube - < $PODMAN_TMPDIR/test.yaml if [ -e /usr/sbin/selinuxenabled -a /usr/sbin/selinuxenabled ]; then run ls -Zd $TESTDIR - is "$output" ${RELABEL} "selinux relabel should have happened" + is "$output" "${RELABEL} $TESTDIR" "selinux relabel should have happened" fi run_podman stop -a -t 0 @@ -88,7 +88,7 @@ RELABEL="system_u:object_r:container_file_t:s0" run_podman play kube $PODMAN_TMPDIR/test.yaml if [ -e /usr/sbin/selinuxenabled -a /usr/sbin/selinuxenabled ]; then run ls -Zd $TESTDIR - is "$output" ${RELABEL} "selinux relabel should have happened" + is "$output" "${RELABEL} $TESTDIR" "selinux relabel should have happened" fi run_podman stop -a -t 0 diff --git a/test/system/helpers.bash b/test/system/helpers.bash index 28ea924bb..03e1ab82b 100644 --- a/test/system/helpers.bash +++ b/test/system/helpers.bash @@ -428,6 +428,18 @@ function skip_if_cgroupsv1() { fi } +###################### +# skip_if_rootless_cgroupsv1 # ...with an optional message +###################### +function skip_if_rootless_cgroupsv1() { + if is_rootless; then + if ! is_cgroupsv2; then + local msg=$(_add_label_if_missing "$1" "rootless cgroupvs1") + skip "${msg:-not supported as rootless under cgroupsv1}" + fi + fi +} + ################################## # skip_if_journald_unavailable # rhbz#1895105: rootless journald permissions ################################## @@ -466,13 +478,30 @@ function is() { local expect="$2" local testname="${3:-${MOST_RECENT_PODMAN_COMMAND:-[no test name given]}}" + local is_expr= if [ -z "$expect" ]; then if [ -z "$actual" ]; then + # Both strings are empty. return fi expect='[no output]' - elif expr "$actual" : "$expect" >/dev/null; then + elif [[ "$actual" = "$expect" ]]; then + # Strings are identical. return + else + # Strings are not identical. Are there wild cards in our expect string? + if expr "$expect" : ".*[^\\][\*\[]" >/dev/null; then + # There is a '[' or '*' without a preceding backslash. + is_expr=' (using expr)' + elif [[ "${expect:0:1}" = '[' ]]; then + # String starts with '[', e.g. checking seconds like '[345]' + is_expr=' (using expr)' + fi + if [[ -n "$is_expr" ]]; then + if expr "$actual" : "$expect" >/dev/null; then + return + fi + fi fi # This is a multi-line message, which may in turn contain multi-line @@ -481,7 +510,7 @@ function is() { readarray -t actual_split <<<"$actual" printf "#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv\n" >&2 printf "#| FAIL: $testname\n" >&2 - printf "#| expected: '%s'\n" "$expect" >&2 + printf "#| expected: '%s'%s\n" "$expect" "$is_expr" >&2 printf "#| actual: '%s'\n" "${actual_split[0]}" >&2 local line for line in "${actual_split[@]:1}"; do diff --git a/vendor/github.com/containers/storage/.cirrus.yml b/vendor/github.com/containers/storage/.cirrus.yml index 20bede452..d080d790c 100644 --- a/vendor/github.com/containers/storage/.cirrus.yml +++ b/vendor/github.com/containers/storage/.cirrus.yml @@ -20,16 +20,14 @@ env: FEDORA_NAME: "fedora-34" PRIOR_FEDORA_NAME: "fedora-33" UBUNTU_NAME: "ubuntu-2104" - PRIOR_UBUNTU_NAME: "ubuntu-2010" # GCE project where images live IMAGE_PROJECT: "libpod-218412" # VM Image built in containers/automation_images - _BUILT_IMAGE_SUFFIX: "c6248193773010944" + _BUILT_IMAGE_SUFFIX: "c6431352024203264" FEDORA_CACHE_IMAGE_NAME: "fedora-${_BUILT_IMAGE_SUFFIX}" PRIOR_FEDORA_CACHE_IMAGE_NAME: "prior-fedora-${_BUILT_IMAGE_SUFFIX}" UBUNTU_CACHE_IMAGE_NAME: "ubuntu-${_BUILT_IMAGE_SUFFIX}" - PRIOR_UBUNTU_CACHE_IMAGE_NAME: "prior-ubuntu-${_BUILT_IMAGE_SUFFIX}" #### #### Command variables to help avoid duplication @@ -115,15 +113,6 @@ ubuntu_testing_task: &ubuntu_testing TEST_DRIVER: "overlay" -prior_ubuntu_testing_task: - <<: *ubuntu_testing - alias: prior_ubuntu_testing - name: *std_test_name - env: - OS_NAME: "${PRIOR_UBUNTU_NAME}" - VM_IMAGE: "${PRIOR_UBUNTU_CACHE_IMAGE_NAME}" - - lint_task: env: CIRRUS_WORKING_DIR: "/go/src/github.com/containers/storage" @@ -153,7 +142,6 @@ meta_task: ${FEDORA_CACHE_IMAGE_NAME} ${PRIOR_FEDORA_CACHE_IMAGE_NAME} ${UBUNTU_CACHE_IMAGE_NAME} - ${PRIOR_UBUNTU_CACHE_IMAGE_NAME} BUILDID: "${CIRRUS_BUILD_ID}" REPOREF: "${CIRRUS_CHANGE_IN_REPO}" GCPJSON: ENCRYPTED[244a93fe8b386b48b96f748342bf741350e43805eee81dd04b45093bdf737e540b993fc735df41f131835fa0f9b65826] @@ -181,7 +169,6 @@ success_task: - fedora_testing - prior_fedora_testing - ubuntu_testing - - prior_ubuntu_testing - meta - vendor container: diff --git a/vendor/github.com/containers/storage/VERSION b/vendor/github.com/containers/storage/VERSION index 39fc130ef..afc132ffa 100644 --- a/vendor/github.com/containers/storage/VERSION +++ b/vendor/github.com/containers/storage/VERSION @@ -1 +1 @@ -1.36.0 +1.36.0+dev diff --git a/vendor/github.com/containers/storage/drivers/aufs/aufs.go b/vendor/github.com/containers/storage/drivers/aufs/aufs.go index 76f12ec3b..a566fbffa 100644 --- a/vendor/github.com/containers/storage/drivers/aufs/aufs.go +++ b/vendor/github.com/containers/storage/drivers/aufs/aufs.go @@ -730,14 +730,14 @@ func useDirperm() bool { enableDirpermLock.Do(func() { base, err := ioutil.TempDir("", "storage-aufs-base") if err != nil { - logrus.Errorf("error checking dirperm1: %v", err) + logrus.Errorf("Checking dirperm1: %v", err) return } defer os.RemoveAll(base) union, err := ioutil.TempDir("", "storage-aufs-union") if err != nil { - logrus.Errorf("error checking dirperm1: %v", err) + logrus.Errorf("Checking dirperm1: %v", err) return } defer os.RemoveAll(union) @@ -748,7 +748,7 @@ func useDirperm() bool { } enableDirperm = true if err := Unmount(union); err != nil { - logrus.Errorf("error checking dirperm1: failed to unmount %v", err) + logrus.Errorf("Checking dirperm1: failed to unmount %v", err) } }) return enableDirperm diff --git a/vendor/github.com/containers/storage/drivers/fsdiff.go b/vendor/github.com/containers/storage/drivers/fsdiff.go index c52788509..a534630df 100644 --- a/vendor/github.com/containers/storage/drivers/fsdiff.go +++ b/vendor/github.com/containers/storage/drivers/fsdiff.go @@ -180,7 +180,7 @@ func (gdw *NaiveDiffDriver) ApplyDiff(id, parent string, options ApplyDiffOpts) start := time.Now().UTC() logrus.Debug("Start untar layer") if size, err = ApplyUncompressedLayer(layerFs, options.Diff, tarOptions); err != nil { - logrus.Errorf("Error while applying layer: %s", err) + logrus.Errorf("While applying layer: %s", err) return } logrus.Debugf("Untar time: %vs", time.Now().UTC().Sub(start).Seconds()) diff --git a/vendor/github.com/containers/storage/drivers/overlay/overlay.go b/vendor/github.com/containers/storage/drivers/overlay/overlay.go index f546f9b10..62130c73e 100644 --- a/vendor/github.com/containers/storage/drivers/overlay/overlay.go +++ b/vendor/github.com/containers/storage/drivers/overlay/overlay.go @@ -174,21 +174,21 @@ func checkSupportVolatile(home, runhome string) (bool, error) { var usingVolatile bool if err == nil { if volatileCacheResult { - logrus.Debugf("cached value indicated that volatile is being used") + logrus.Debugf("Cached value indicated that volatile is being used") } else { - logrus.Debugf("cached value indicated that volatile is not being used") + logrus.Debugf("Cached value indicated that volatile is not being used") } usingVolatile = volatileCacheResult } else { usingVolatile, err = doesVolatile(home) if err == nil { if usingVolatile { - logrus.Debugf("overlay test mount indicated that volatile is being used") + logrus.Debugf("overlay: test mount indicated that volatile is being used") } else { - logrus.Debugf("overlay test mount indicated that volatile is not being used") + logrus.Debugf("overlay: test mount indicated that volatile is not being used") } if err = cachedFeatureRecord(runhome, feature, usingVolatile, ""); err != nil { - return false, errors.Wrap(err, "error recording volatile-being-used status") + return false, errors.Wrap(err, "recording volatile-being-used status") } } } @@ -206,9 +206,9 @@ func checkAndRecordOverlaySupport(fsMagic graphdriver.FsMagic, home, runhome str overlayCacheResult, overlayCacheText, err := cachedFeatureCheck(runhome, feature) if err == nil { if overlayCacheResult { - logrus.Debugf("cached value indicated that overlay is supported") + logrus.Debugf("Cached value indicated that overlay is supported") } else { - logrus.Debugf("cached value indicated that overlay is not supported") + logrus.Debugf("Cached value indicated that overlay is not supported") } supportsDType = overlayCacheResult if !supportsDType { @@ -225,12 +225,12 @@ func checkAndRecordOverlaySupport(fsMagic graphdriver.FsMagic, home, runhome str } err = errors.Wrap(err, "kernel does not support overlay fs") if err2 := cachedFeatureRecord(runhome, feature, false, err.Error()); err2 != nil { - return false, errors.Wrapf(err2, "error recording overlay not being supported (%v)", err) + return false, errors.Wrapf(err2, "recording overlay not being supported (%v)", err) } return false, err } if err = cachedFeatureRecord(runhome, feature, supportsDType, ""); err != nil { - return false, errors.Wrap(err, "error recording overlay support status") + return false, errors.Wrap(err, "recording overlay support status") } } return supportsDType, nil @@ -310,24 +310,24 @@ func Init(home string, options graphdriver.Options) (graphdriver.Driver, error) metacopyCacheResult, _, err := cachedFeatureCheck(runhome, feature) if err == nil { if metacopyCacheResult { - logrus.Debugf("cached value indicated that metacopy is being used") + logrus.Debugf("Cached value indicated that metacopy is being used") } else { - logrus.Debugf("cached value indicated that metacopy is not being used") + logrus.Debugf("Cached value indicated that metacopy is not being used") } usingMetacopy = metacopyCacheResult } else { usingMetacopy, err = doesMetacopy(home, opts.mountOptions) if err == nil { if usingMetacopy { - logrus.Debugf("overlay test mount indicated that metacopy is being used") + logrus.Debugf("overlay: test mount indicated that metacopy is being used") } else { - logrus.Debugf("overlay test mount indicated that metacopy is not being used") + logrus.Debugf("overlay: test mount indicated that metacopy is not being used") } if err = cachedFeatureRecord(runhome, feature, usingMetacopy, ""); err != nil { - return nil, errors.Wrap(err, "error recording metacopy-being-used status") + return nil, errors.Wrap(err, "recording metacopy-being-used status") } } else { - logrus.Infof("overlay test mount did not indicate whether or not metacopy is being used: %v", err) + logrus.Infof("overlay: test mount did not indicate whether or not metacopy is being used: %v", err) return nil, err } } @@ -548,7 +548,7 @@ func SupportsNativeOverlay(graphroot, rundir string) (bool, error) { } switch contents { case "true": - logrus.Debugf("overlay storage already configured with a mount-program") + logrus.Debugf("overlay: storage already configured with a mount-program") return false, nil default: needsMountProgram, err := scanForMountProgramIndicators(home) @@ -640,17 +640,17 @@ func supportsOverlay(home string, homeMagic graphdriver.FsMagic, rootUID, rootGI flags = fmt.Sprintf("%s,userxattr", flags) } if err := syscall.Mknod(filepath.Join(upperDir, "whiteout"), syscall.S_IFCHR|0600, int(unix.Mkdev(0, 0))); err != nil { - logrus.Debugf("unable to create kernel-style whiteout: %v", err) + logrus.Debugf("Unable to create kernel-style whiteout: %v", err) return supportsDType, errors.Wrapf(err, "unable to create kernel-style whiteout") } if len(flags) < unix.Getpagesize() { err := unix.Mount("overlay", mergedDir, "overlay", 0, flags) if err == nil { - logrus.Debugf("overlay test mount with multiple lowers succeeded") + logrus.Debugf("overlay: test mount with multiple lowers succeeded") return supportsDType, nil } - logrus.Debugf("overlay test mount with multiple lowers failed %v", err) + logrus.Debugf("overlay: test mount with multiple lowers failed %v", err) } flags = fmt.Sprintf("lowerdir=%s,upperdir=%s,workdir=%s", lower1Dir, upperDir, workDir) if selinux.GetEnabled() { @@ -659,10 +659,10 @@ func supportsOverlay(home string, homeMagic graphdriver.FsMagic, rootUID, rootGI if len(flags) < unix.Getpagesize() { err := unix.Mount("overlay", mergedDir, "overlay", 0, flags) if err == nil { - logrus.StandardLogger().Logf(logLevel, "overlay test mount with multiple lowers failed, but succeeded with a single lower") + logrus.StandardLogger().Logf(logLevel, "overlay: test mount with multiple lowers failed, but succeeded with a single lower") return supportsDType, errors.Wrap(graphdriver.ErrNotSupported, "kernel too old to provide multiple lowers feature for overlay") } - logrus.Debugf("overlay test mount with a single lower failed %v", err) + logrus.Debugf("overlay: test mount with a single lower failed %v", err) } logrus.StandardLogger().Logf(logLevel, "'overlay' is not supported over %s at %q", backingFs, home) return supportsDType, errors.Wrapf(graphdriver.ErrIncompatibleFS, "'overlay' is not supported over %s at %q", backingFs, home) @@ -682,9 +682,9 @@ func (d *Driver) useNaiveDiff() bool { nativeDiffCacheResult, nativeDiffCacheText, err := cachedFeatureCheck(d.runhome, feature) if err == nil { if nativeDiffCacheResult { - logrus.Debugf("cached value indicated that native-diff is usable") + logrus.Debugf("Cached value indicated that native-diff is usable") } else { - logrus.Debugf("cached value indicated that native-diff is not being used") + logrus.Debugf("Cached value indicated that native-diff is not being used") logrus.Info(nativeDiffCacheText) } useNaiveDiffOnly = !nativeDiffCacheResult @@ -821,7 +821,7 @@ func (d *Driver) CreateReadWrite(id, parent string, opts *graphdriver.CreateOpts opts.StorageOpt["inodes"] = strconv.FormatUint(d.options.quota.Inodes, 10) } - return d.create(id, parent, opts) + return d.create(id, parent, opts, false) } // Create is used to create the upper, lower, and merge directories required for overlay fs for a given id. @@ -831,15 +831,16 @@ func (d *Driver) Create(id, parent string, opts *graphdriver.CreateOpts) (retErr if _, ok := opts.StorageOpt["size"]; ok { return fmt.Errorf("--storage-opt size is only supported for ReadWrite Layers") } + if _, ok := opts.StorageOpt["inodes"]; ok { return fmt.Errorf("--storage-opt inodes is only supported for ReadWrite Layers") } } - return d.create(id, parent, opts) + return d.create(id, parent, opts, true) } -func (d *Driver) create(id, parent string, opts *graphdriver.CreateOpts) (retErr error) { +func (d *Driver) create(id, parent string, opts *graphdriver.CreateOpts, disableQuota bool) (retErr error) { dir := d.dir(id) uidMaps := d.uidMaps @@ -880,7 +881,7 @@ func (d *Driver) create(id, parent string, opts *graphdriver.CreateOpts) (retErr } }() - if d.quotaCtl != nil { + if d.quotaCtl != nil && !disableQuota { quota := quota.Quota{} if opts != nil && len(opts.StorageOpt) > 0 { driver := &Driver{} @@ -994,7 +995,7 @@ func (d *Driver) getLower(parent string) (string, error) { } logrus.Warnf("Can't read parent link %q because it does not exist. Going through storage to recreate the missing links.", path.Join(parentDir, "link")) if err := d.recreateSymlinks(); err != nil { - return "", errors.Wrap(err, "error recreating the links") + return "", errors.Wrap(err, "recreating the links") } parentLink, err = ioutil.ReadFile(path.Join(parentDir, "link")) if err != nil { @@ -1038,7 +1039,7 @@ func (d *Driver) getLowerDirs(id string) ([]string, error) { if os.IsNotExist(err) { logrus.Warnf("Can't read link %q because it does not exist. A storage corruption might have occurred, attempting to recreate the missing symlinks. It might be best wipe the storage to avoid further errors due to storage corruption.", lower) if err := d.recreateSymlinks(); err != nil { - return nil, fmt.Errorf("error recreating the missing symlinks: %v", err) + return nil, fmt.Errorf("recreating the missing symlinks: %v", err) } // let's call Readlink on lower again now that we have recreated the missing symlinks lp, err = os.Readlink(lower) @@ -1121,7 +1122,7 @@ func (d *Driver) recreateSymlinks() error { // List all the directories under the home directory dirs, err := ioutil.ReadDir(d.home) if err != nil { - return fmt.Errorf("error reading driver home directory %q: %v", d.home, err) + return fmt.Errorf("reading driver home directory %q: %v", d.home, err) } linksDir := filepath.Join(d.home, "l") // This makes the link directory if it doesn't exist @@ -1148,7 +1149,7 @@ func (d *Driver) recreateSymlinks() error { // Read the "link" file under each layer to get the name of the symlink data, err := ioutil.ReadFile(path.Join(d.dir(dir.Name()), "link")) if err != nil { - errs = multierror.Append(errs, errors.Wrapf(err, "error reading name of symlink for %q", dir)) + errs = multierror.Append(errs, errors.Wrapf(err, "reading name of symlink for %q", dir)) continue } linkPath := path.Join(d.home, linkDir, strings.Trim(string(data), "\n")) @@ -1162,7 +1163,7 @@ func (d *Driver) recreateSymlinks() error { } madeProgress = true } else if err != nil { - errs = multierror.Append(errs, errors.Wrapf(err, "error trying to stat %q", linkPath)) + errs = multierror.Append(errs, err) continue } } @@ -1170,7 +1171,7 @@ func (d *Driver) recreateSymlinks() error { // that each symlink we have corresponds to one. links, err := ioutil.ReadDir(linksDir) if err != nil { - errs = multierror.Append(errs, errors.Wrapf(err, "error reading links directory %q", linksDir)) + errs = multierror.Append(errs, err) continue } // Go through all of the symlinks in the "l" directory @@ -1178,7 +1179,7 @@ func (d *Driver) recreateSymlinks() error { // Read the symlink's target, which should be "../$layer/diff" target, err := os.Readlink(filepath.Join(linksDir, link.Name())) if err != nil { - errs = multierror.Append(errs, errors.Wrapf(err, "error reading target of link %q", link)) + errs = multierror.Append(errs, err) continue } targetComponents := strings.Split(target, string(os.PathSeparator)) @@ -1196,7 +1197,7 @@ func (d *Driver) recreateSymlinks() error { data, err := ioutil.ReadFile(linkFile) if err != nil || string(data) != link.Name() { if err := ioutil.WriteFile(linkFile, []byte(link.Name()), 0644); err != nil { - errs = multierror.Append(errs, errors.Wrapf(err, "error correcting link for layer %q", targetID)) + errs = multierror.Append(errs, errors.Wrapf(err, "correcting link for layer %s", targetID)) continue } madeProgress = true @@ -1241,7 +1242,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO if unshare.IsRootless() { logLevel = logrus.DebugLevel } - logrus.StandardLogger().Logf(logLevel, "ignoring metacopy option from storage.conf, not supported with booted kernel") + logrus.StandardLogger().Logf(logLevel, "Ignoring metacopy option from storage.conf, not supported with booted kernel") } } } @@ -1275,7 +1276,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO } logrus.Warnf("Can't read parent link %q because it does not exist. Going through storage to recreate the missing links.", path.Join(dir, "link")) if err := d.recreateSymlinks(); err != nil { - return "", errors.Wrap(err, "error recreating the links") + return "", errors.Wrap(err, "recreating the links") } link, err = ioutil.ReadFile(path.Join(dir, "link")) if err != nil { @@ -1330,7 +1331,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO if lower == "" && os.IsNotExist(err) { logrus.Warnf("Can't stat lower layer %q because it does not exist. Going through storage to recreate the missing symlinks.", newpath) if err := d.recreateSymlinks(); err != nil { - return "", fmt.Errorf("error recreating the missing symlinks: %v", err) + return "", fmt.Errorf("Recreating the missing symlinks: %v", err) } lower = newpath } else if lower == "" { @@ -1381,7 +1382,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO if retErr != nil { if c := d.ctr.Decrement(mergedDir); c <= 0 { if mntErr := unix.Unmount(mergedDir, 0); mntErr != nil { - logrus.Errorf("error unmounting %v: %v", mergedDir, mntErr) + logrus.Errorf("Unmounting %v: %v", mergedDir, mntErr) } } } @@ -1473,7 +1474,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO flags, data := mount.ParseOptions(mountData) logrus.Debugf("overlay: mount_data=%s", mountData) if err := mountFunc("overlay", mountTarget, "overlay", uintptr(flags), data); err != nil { - return "", fmt.Errorf("error creating overlay mount to %s, mount_data=%q: %v", mountTarget, mountData, err) + return "", fmt.Errorf("creating overlay mount to %s, mount_data=%q: %v", mountTarget, mountData, err) } return mergedDir, nil @@ -1820,7 +1821,7 @@ func (d *Driver) UpdateLayerIDMap(id string, toContainer, toHost *idtools.IDMapp err = graphdriver.ChownPathByMaps(layerFs, toContainer, toHost) if err != nil { if err2 := d.Put(id); err2 != nil { - logrus.Errorf("%v; error unmounting %v: %v", err, id, err2) + logrus.Errorf("%v; unmounting %v: %v", err, id, err2) } return err } @@ -1923,7 +1924,7 @@ func (d *Driver) releaseAdditionalLayerByID(id string) { if al, err := d.getAdditionalLayerPathByID(id); err == nil { notifyReleaseAdditionalLayer(al) } else if !os.IsNotExist(err) { - logrus.Warnf("unexpected error on reading Additional Layer Store pointer %v", err) + logrus.Warnf("Unexpected error on reading Additional Layer Store pointer %v", err) } } @@ -2004,10 +2005,10 @@ func notifyUseAdditionalLayer(al string) { } else if err == nil { f.Close() if err := os.Remove(useFile); err != nil { - logrus.Warnf("failed to remove use file") + logrus.Warnf("Failed to remove use file") } } - logrus.Warnf("unexpected error by Additional Layer Store %v during use; GC doesn't seem to be supported", err) + logrus.Warnf("Unexpected error by Additional Layer Store %v during use; GC doesn't seem to be supported", err) } // notifyReleaseAdditionalLayer notifies Additional Layer Store that we don't use the specified @@ -2024,7 +2025,7 @@ func notifyReleaseAdditionalLayer(al string) { if os.IsNotExist(err) { return } - logrus.Warnf("unexpected error by Additional Layer Store %v during release; GC doesn't seem to be supported", err) + logrus.Warnf("Unexpected error by Additional Layer Store %v during release; GC doesn't seem to be supported", err) } // redirectDiffIfAdditionalLayer checks if the passed diff path is Additional Layer and diff --git a/vendor/github.com/containers/storage/drivers/overlay/randomid.go b/vendor/github.com/containers/storage/drivers/overlay/randomid.go index fc565ef0b..736c48b9c 100644 --- a/vendor/github.com/containers/storage/drivers/overlay/randomid.go +++ b/vendor/github.com/containers/storage/drivers/overlay/randomid.go @@ -47,7 +47,7 @@ func generateID(l int) string { if retryOnError(err) && retries < maxretries { count += n retries++ - logrus.Errorf("error generating version 4 uuid, retrying: %v", err) + logrus.Errorf("Generating version 4 uuid, retrying: %v", err) continue } diff --git a/vendor/github.com/containers/storage/drivers/template.go b/vendor/github.com/containers/storage/drivers/template.go index 5d80b8865..d40d71cfc 100644 --- a/vendor/github.com/containers/storage/drivers/template.go +++ b/vendor/github.com/containers/storage/drivers/template.go @@ -31,7 +31,7 @@ func NaiveCreateFromTemplate(d TemplateDriver, id, template string, templateIDMa diff, err := d.Diff(template, templateIDMappings, parent, parentIDMappings, opts.MountLabel) if err != nil { if err2 := d.Remove(id); err2 != nil { - logrus.Errorf("error removing layer %q: %v", id, err2) + logrus.Errorf("Removing layer %q: %v", id, err2) } return err } @@ -44,7 +44,7 @@ func NaiveCreateFromTemplate(d TemplateDriver, id, template string, templateIDMa } if _, err = d.ApplyDiff(id, parent, applyOptions); err != nil { if err2 := d.Remove(id); err2 != nil { - logrus.Errorf("error removing layer %q: %v", id, err2) + logrus.Errorf("Removing layer %q: %v", id, err2) } return err } diff --git a/vendor/github.com/containers/storage/go.mod b/vendor/github.com/containers/storage/go.mod index 911469182..ac7f45c3f 100644 --- a/vendor/github.com/containers/storage/go.mod +++ b/vendor/github.com/containers/storage/go.mod @@ -12,14 +12,13 @@ require ( github.com/google/go-intervals v0.0.2 github.com/google/uuid v1.2.0 // indirect github.com/hashicorp/go-multierror v1.1.1 - github.com/json-iterator/go v1.1.11 - github.com/klauspost/compress v1.13.5 + github.com/json-iterator/go v1.1.12 + github.com/klauspost/compress v1.13.6 github.com/klauspost/pgzip v1.2.5 github.com/mattn/go-shellwords v1.0.12 github.com/mistifyio/go-zfs v2.1.2-0.20190413222219-f784269be439+incompatible github.com/moby/sys/mountinfo v0.4.1 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect - github.com/modern-go/reflect2 v1.0.1 // indirect github.com/opencontainers/go-digest v1.0.0 github.com/opencontainers/runc v1.0.2 github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 diff --git a/vendor/github.com/containers/storage/go.sum b/vendor/github.com/containers/storage/go.sum index 5cc5da6d3..3da64ce8c 100644 --- a/vendor/github.com/containers/storage/go.sum +++ b/vendor/github.com/containers/storage/go.sum @@ -115,15 +115,16 @@ github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= -github.com/json-iterator/go v1.1.11 h1:uVUAXhF2To8cbw/3xN3pxj6kk7TYKs98NIrTqPlMWAQ= -github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4= github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= +github.com/klauspost/compress v1.13.6 h1:P76CopJELS0TiO2mebmnzgWaajssP/EszplttgQxcgc= +github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -148,9 +149,8 @@ github.com/moby/sys/mountinfo v0.4.1/go.mod h1:rEr8tzG/lsIZHBtN/JjGG+LMYx9eXgW2J github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= -github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/mrunalp/fileutils v0.5.0/go.mod h1:M1WthSahJixYnrXQl/DFQuteStB1weuxD2QJNHXfbSQ= github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= diff --git a/vendor/github.com/containers/storage/layers.go b/vendor/github.com/containers/storage/layers.go index 32ba20685..fbf6ad362 100644 --- a/vendor/github.com/containers/storage/layers.go +++ b/vendor/github.com/containers/storage/layers.go @@ -1557,7 +1557,7 @@ func (r *layerStore) applyDiffWithOptions(to string, layerOptions *LayerOptions, compressor = pgzip.NewWriter(&tsdata) } if err := compressor.SetConcurrency(1024*1024, 1); err != nil { // 1024*1024 is the hard-coded default; we're not changing that - logrus.Infof("error setting compression concurrency threads to 1: %v; ignoring", err) + logrus.Infof("Error setting compression concurrency threads to 1: %v; ignoring", err) } metadata := storage.NewJSONPacker(compressor) uncompressed, err := archive.DecompressStream(defragmented) diff --git a/vendor/github.com/containers/storage/pkg/archive/archive.go b/vendor/github.com/containers/storage/pkg/archive/archive.go index 48e846f7c..76544ff28 100644 --- a/vendor/github.com/containers/storage/pkg/archive/archive.go +++ b/vendor/github.com/containers/storage/pkg/archive/archive.go @@ -879,7 +879,7 @@ func TarWithOptions(srcPath string, options *TarOptions) (io.ReadCloser, error) if include != relFilePath { matches, err := pm.IsMatch(relFilePath) if err != nil { - logrus.Errorf("Error matching %s: %v", relFilePath, err) + logrus.Errorf("Matching %s: %v", relFilePath, err) return err } skip = matches diff --git a/vendor/github.com/containers/storage/pkg/chunked/storage_linux.go b/vendor/github.com/containers/storage/pkg/chunked/storage_linux.go index 7bd804c44..6efc6a4c8 100644 --- a/vendor/github.com/containers/storage/pkg/chunked/storage_linux.go +++ b/vendor/github.com/containers/storage/pkg/chunked/storage_linux.go @@ -345,6 +345,56 @@ func getFileDigest(f *os.File) (digest.Digest, error) { return digester.Digest(), nil } +// findFileInOSTreeRepos checks whether the requested file already exist in one of the OSTree repo and copies the file content from there if possible. +// file is the file to look for. +// ostreeRepos is a list of OSTree repos. +// dirfd is an open fd to the destination checkout. +// useHardLinks defines whether the deduplication can be performed using hard links. +func findFileInOSTreeRepos(file *internal.FileMetadata, ostreeRepos []string, dirfd int, useHardLinks bool) (bool, *os.File, int64, error) { + digest, err := digest.Parse(file.Digest) + if err != nil { + return false, nil, 0, nil + } + payloadLink := digest.Encoded() + ".payload-link" + if len(payloadLink) < 2 { + return false, nil, 0, nil + } + + for _, repo := range ostreeRepos { + sourceFile := filepath.Join(repo, "objects", payloadLink[:2], payloadLink[2:]) + st, err := os.Stat(sourceFile) + if err != nil || !st.Mode().IsRegular() { + continue + } + if st.Size() != file.Size { + continue + } + fd, err := unix.Open(sourceFile, unix.O_RDONLY|unix.O_NONBLOCK, 0) + if err != nil { + return false, nil, 0, nil + } + f := os.NewFile(uintptr(fd), "fd") + defer f.Close() + + // check if the open file can be deduplicated with hard links + if useHardLinks && !canDedupFileWithHardLink(file, fd, st) { + continue + } + + dstFile, written, err := copyFileContent(fd, file.Name, dirfd, 0, useHardLinks) + if err != nil { + return false, nil, 0, nil + } + return true, dstFile, written, nil + } + // If hard links deduplication was used and it has failed, try again without hard links. + if useHardLinks { + return findFileInOSTreeRepos(file, ostreeRepos, dirfd, false) + } + + return false, nil, 0, nil +} + // findFileOnTheHost checks whether the requested file already exist on the host and copies the file content from there if possible. // It is currently implemented to look only at the file with the same path. Ideally it can detect the same content also at different // paths. @@ -873,6 +923,9 @@ func (c *chunkedDiffer) ApplyDiff(dest string, options *archive.TarOptions) (gra // modifies the source file as well. useHardLinks := parseBooleanPullOption(&storeOpts, "use_hard_links", false) + // List of OSTree repositories to use for deduplication + ostreeRepos := strings.Split(storeOpts.PullOptions["ostree_repos"], ":") + // Generate the manifest var toc internal.TOC if err := json.Unmarshal(c.manifest, &toc); err != nil { @@ -1009,18 +1062,35 @@ func (c *chunkedDiffer) ApplyDiff(dest string, options *archive.TarOptions) (gra totalChunksSize += r.Size + finalizeFile := func(dstFile *os.File) error { + if dstFile != nil { + defer dstFile.Close() + if err := setFileAttrs(dstFile, mode, &r, options); err != nil { + return err + } + } + return nil + } + found, dstFile, _, err := findFileInOtherLayers(&r, dirfd, otherLayersCache, c.layersTarget, useHardLinks) if err != nil { return output, err } - if dstFile != nil { - if err := setFileAttrs(dstFile, mode, &r, options); err != nil { - dstFile.Close() + if found { + if err := finalizeFile(dstFile); err != nil { return output, err } - dstFile.Close() + continue + } + + found, dstFile, _, err = findFileInOSTreeRepos(&r, ostreeRepos, dirfd, useHardLinks) + if err != nil { + return output, err } if found { + if err := finalizeFile(dstFile); err != nil { + return output, err + } continue } @@ -1029,14 +1099,10 @@ func (c *chunkedDiffer) ApplyDiff(dest string, options *archive.TarOptions) (gra if err != nil { return output, err } - if dstFile != nil { - if err := setFileAttrs(dstFile, mode, &r, options); err != nil { - dstFile.Close() + if found { + if err := finalizeFile(dstFile); err != nil { return output, err } - dstFile.Close() - } - if found { continue } } diff --git a/vendor/github.com/containers/storage/pkg/fileutils/fileutils_unix.go b/vendor/github.com/containers/storage/pkg/fileutils/fileutils_unix.go index 9e0e97bd6..92056c1d5 100644 --- a/vendor/github.com/containers/storage/pkg/fileutils/fileutils_unix.go +++ b/vendor/github.com/containers/storage/pkg/fileutils/fileutils_unix.go @@ -14,7 +14,7 @@ import ( // reading it via /proc filesystem. func GetTotalUsedFds() int { if fds, err := ioutil.ReadDir(fmt.Sprintf("/proc/%d/fd", os.Getpid())); err != nil { - logrus.Errorf("Error opening /proc/%d/fd: %s", os.Getpid(), err) + logrus.Errorf("%v", err) } else { return len(fds) } diff --git a/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go b/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go index b224e7b5c..fc080acbe 100644 --- a/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go +++ b/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go @@ -36,7 +36,7 @@ type lockfile struct { // necessary. func openLock(path string, ro bool) (fd int, err error) { if ro { - fd, err = unix.Open(path, os.O_RDONLY|unix.O_CLOEXEC, 0) + fd, err = unix.Open(path, os.O_RDONLY|unix.O_CLOEXEC|os.O_CREATE, 0) } else { fd, err = unix.Open(path, os.O_RDWR|unix.O_CLOEXEC|os.O_CREATE, diff --git a/vendor/github.com/containers/storage/pkg/loopback/attach_loopback.go b/vendor/github.com/containers/storage/pkg/loopback/attach_loopback.go index e2cf30b41..6f0726505 100644 --- a/vendor/github.com/containers/storage/pkg/loopback/attach_loopback.go +++ b/vendor/github.com/containers/storage/pkg/loopback/attach_loopback.go @@ -43,7 +43,7 @@ func openNextAvailableLoopback(index int, sparseName string, sparseFile *os.File var st syscall.Stat_t err = syscall.Fstat(int(sparseFile.Fd()), &st) if err != nil { - logrus.Errorf("Error reading information about loopback file %s: %v", sparseName, err) + logrus.Errorf("Reading information about loopback file %s: %v", sparseName, err) return nil, ErrAttachLoopbackDevice } @@ -68,7 +68,7 @@ func openNextAvailableLoopback(index int, sparseName string, sparseFile *os.File // OpenFile adds O_CLOEXEC loopFile, err = os.OpenFile(target, os.O_RDWR, 0644) if err != nil { - logrus.Errorf("Error opening loopback device: %s", err) + logrus.Errorf("Opening loopback device: %s", err) return nil, ErrAttachLoopbackDevice } @@ -90,7 +90,7 @@ func openNextAvailableLoopback(index int, sparseName string, sparseFile *os.File // device and inode numbers. dev, ino, err := getLoopbackBackingFile(loopFile) if err != nil { - logrus.Errorf("Error getting loopback backing file: %s", err) + logrus.Errorf("Getting loopback backing file: %s", err) return nil, ErrGetLoopbackBackingFile } if dev != uint64(st.Dev) || ino != st.Ino { @@ -125,7 +125,7 @@ func AttachLoopDevice(sparseName string) (loop *os.File, err error) { // OpenFile adds O_CLOEXEC sparseFile, err := os.OpenFile(sparseName, os.O_RDWR, 0644) if err != nil { - logrus.Errorf("Error opening sparse file %s: %s", sparseName, err) + logrus.Errorf("Opening sparse file: %v", err) return nil, ErrAttachLoopbackDevice } defer sparseFile.Close() @@ -147,7 +147,7 @@ func AttachLoopDevice(sparseName string) (loop *os.File, err error) { // If the call failed, then free the loopback device if err := ioctlLoopClrFd(loopFile.Fd()); err != nil { - logrus.Error("Error while cleaning up the loopback device") + logrus.Error("While cleaning up the loopback device") } loopFile.Close() return nil, ErrAttachLoopbackDevice diff --git a/vendor/github.com/containers/storage/pkg/loopback/loopback.go b/vendor/github.com/containers/storage/pkg/loopback/loopback.go index f4cf2826e..c9be05776 100644 --- a/vendor/github.com/containers/storage/pkg/loopback/loopback.go +++ b/vendor/github.com/containers/storage/pkg/loopback/loopback.go @@ -13,7 +13,7 @@ import ( func getLoopbackBackingFile(file *os.File) (uint64, uint64, error) { loopInfo, err := ioctlLoopGetStatus64(file.Fd()) if err != nil { - logrus.Errorf("Error get loopback backing file: %s", err) + logrus.Errorf("Get loopback backing file: %v", err) return 0, 0, ErrGetLoopbackBackingFile } return loopInfo.loDevice, loopInfo.loInode, nil @@ -22,7 +22,7 @@ func getLoopbackBackingFile(file *os.File) (uint64, uint64, error) { // SetCapacity reloads the size for the loopback device. func SetCapacity(file *os.File) error { if err := ioctlLoopSetCapacity(file.Fd(), 0); err != nil { - logrus.Errorf("Error loopbackSetCapacity: %s", err) + logrus.Errorf("loopbackSetCapacity: %s", err) return ErrSetCapacity } return nil diff --git a/vendor/github.com/containers/storage/pkg/parsers/kernel/kernel_unix.go b/vendor/github.com/containers/storage/pkg/parsers/kernel/kernel_unix.go index 76e1e499f..7a68bc39b 100644 --- a/vendor/github.com/containers/storage/pkg/parsers/kernel/kernel_unix.go +++ b/vendor/github.com/containers/storage/pkg/parsers/kernel/kernel_unix.go @@ -35,7 +35,7 @@ func GetKernelVersion() (*VersionInfo, error) { // the given version. func CheckKernelVersion(k, major, minor int) bool { if v, err := GetKernelVersion(); err != nil { - logrus.Warnf("error getting kernel version: %s", err) + logrus.Warnf("Error getting kernel version: %s", err) } else { if CompareKernelVersion(*v, VersionInfo{Kernel: k, Major: major, Minor: minor}) < 0 { return false diff --git a/vendor/github.com/containers/storage/pkg/tarlog/tarlogger.go b/vendor/github.com/containers/storage/pkg/tarlog/tarlogger.go index 26cd8504c..674e0a0ba 100644 --- a/vendor/github.com/containers/storage/pkg/tarlog/tarlogger.go +++ b/vendor/github.com/containers/storage/pkg/tarlog/tarlogger.go @@ -34,7 +34,7 @@ func NewLogger(logger func(*tar.Header)) (io.WriteCloser, error) { } // Make sure to avoid writes after the reader has been closed. if err := reader.Close(); err != nil { - logrus.Errorf("error closing tarlogger reader: %v", err) + logrus.Errorf("Closing tarlogger reader: %v", err) } // Unblock the Close(). t.closeMutex.Unlock() diff --git a/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go b/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go index 96b857543..6d351ce80 100644 --- a/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go +++ b/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go @@ -154,7 +154,7 @@ func (c *Cmd) Start() error { pidString := "" b := new(bytes.Buffer) if _, err := io.Copy(b, pidRead); err != nil { - return errors.Wrapf(err, "error reading child PID") + return errors.Wrapf(err, "Reading child PID") } pidString = b.String() pid, err := strconv.Atoi(pidString) @@ -188,8 +188,8 @@ func (c *Cmd) Start() error { if len(c.UidMappings) == 0 || len(c.GidMappings) == 0 { uidmap, gidmap, err := GetHostIDMappings("") if err != nil { - fmt.Fprintf(continueWrite, "error reading ID mappings in parent: %v", err) - return errors.Wrapf(err, "error reading ID mappings in parent") + fmt.Fprintf(continueWrite, "Reading ID mappings in parent: %v", err) + return errors.Wrapf(err, "Reading ID mappings in parent") } if len(c.UidMappings) == 0 { c.UidMappings = uidmap @@ -222,8 +222,8 @@ func (c *Cmd) Start() error { if err == nil { gidmapSet = true } else { - logrus.Warnf("error running newgidmap: %v: %s", err, g.String()) - logrus.Warnf("falling back to single mapping") + logrus.Warnf("Error running newgidmap: %v: %s", err, g.String()) + logrus.Warnf("Falling back to single mapping") g.Reset() g.Write([]byte(fmt.Sprintf("0 %d 1\n", os.Getegid()))) } @@ -271,8 +271,8 @@ func (c *Cmd) Start() error { if err == nil { uidmapSet = true } else { - logrus.Warnf("error running newuidmap: %v: %s", err, u.String()) - logrus.Warnf("falling back to single mapping") + logrus.Warnf("Error running newuidmap: %v: %s", err, u.String()) + logrus.Warnf("Falling back to single mapping") u.Reset() u.Write([]byte(fmt.Sprintf("0 %d 1\n", os.Geteuid()))) } @@ -407,7 +407,7 @@ func MaybeReexecUsingUserNamespace(evenForRoot bool) { // ID and a range size. uidmap, gidmap, err = GetSubIDMappings(me.Username, me.Username) if err != nil { - logrus.Warnf("error reading allowed ID mappings: %v", err) + logrus.Warnf("Reading allowed ID mappings: %v", err) } if len(uidmap) == 0 { logrus.Warnf("Found no UID ranges set aside for user %q in /etc/subuid.", me.Username) @@ -434,13 +434,13 @@ func MaybeReexecUsingUserNamespace(evenForRoot bool) { // If we have CAP_SYS_ADMIN, then we don't need to create a new namespace in order to be able // to use unshare(), so don't bother creating a new user namespace at this point. capabilities, err := capability.NewPid(0) - bailOnError(err, "error reading the current capabilities sets") + bailOnError(err, "Reading the current capabilities sets") if capabilities.Get(capability.EFFECTIVE, capability.CAP_SYS_ADMIN) { return } // Read the set of ID mappings that we're currently using. uidmap, gidmap, err = GetHostIDMappings("") - bailOnError(err, "error reading current ID mappings") + bailOnError(err, "Reading current ID mappings") // Just reuse them. for i := range uidmap { uidmap[i].HostID = uidmap[i].ContainerID @@ -463,7 +463,7 @@ func MaybeReexecUsingUserNamespace(evenForRoot bool) { if _, present := os.LookupEnv("BUILDAH_ISOLATION"); !present { if err = os.Setenv("BUILDAH_ISOLATION", "rootless"); err != nil { if err := os.Setenv("BUILDAH_ISOLATION", "rootless"); err != nil { - logrus.Errorf("error setting BUILDAH_ISOLATION=rootless in environment: %v", err) + logrus.Errorf("Setting BUILDAH_ISOLATION=rootless in environment: %v", err) os.Exit(1) } } @@ -483,7 +483,7 @@ func MaybeReexecUsingUserNamespace(evenForRoot bool) { cmd.GidMappingsEnableSetgroups = true // Finish up. - logrus.Debugf("running %+v with environment %+v, UID map %+v, and GID map %+v", cmd.Cmd.Args, os.Environ(), cmd.UidMappings, cmd.GidMappings) + logrus.Debugf("Running %+v with environment %+v, UID map %+v, and GID map %+v", cmd.Cmd.Args, os.Environ(), cmd.UidMappings, cmd.GidMappings) ExecRunnable(cmd, nil) } @@ -512,7 +512,7 @@ func ExecRunnable(cmd Runnable, cleanup func()) { } } logrus.Errorf("%v", err) - logrus.Errorf("(unable to determine exit status)") + logrus.Errorf("(Unable to determine exit status)") exit(1) } exit(0) @@ -523,7 +523,7 @@ func getHostIDMappings(path string) ([]specs.LinuxIDMapping, error) { var mappings []specs.LinuxIDMapping f, err := os.Open(path) if err != nil { - return nil, errors.Wrapf(err, "error reading ID mappings from %q", path) + return nil, errors.Wrapf(err, "Reading ID mappings from %q", path) } defer f.Close() scanner := bufio.NewScanner(f) @@ -571,7 +571,7 @@ func GetHostIDMappings(pid string) ([]specs.LinuxIDMapping, []specs.LinuxIDMappi func GetSubIDMappings(user, group string) ([]specs.LinuxIDMapping, []specs.LinuxIDMapping, error) { mappings, err := idtools.NewIDMappings(user, group) if err != nil { - return nil, nil, errors.Wrapf(err, "error reading subuid mappings for user %q and subgid mappings for group %q", user, group) + return nil, nil, errors.Wrapf(err, "Reading subuid mappings for user %q and subgid mappings for group %q", user, group) } var uidmap, gidmap []specs.LinuxIDMapping for _, m := range mappings.UIDs() { diff --git a/vendor/github.com/containers/storage/store.go b/vendor/github.com/containers/storage/store.go index 8d6f2c4d7..6f6f69807 100644 --- a/vendor/github.com/containers/storage/store.go +++ b/vendor/github.com/containers/storage/store.go @@ -1131,10 +1131,6 @@ func (s *store) imageTopLayerForMapping(image *Image, ristore ROImageStore, crea if options.HostGIDMapping && len(layer.GIDMap) != 0 { return false } - // If we don't care about the mapping, it's fine. - if len(options.UIDMap) == 0 && len(options.GIDMap) == 0 { - return true - } // Compare the maps. return reflect.DeepEqual(layer.UIDMap, options.UIDMap) && reflect.DeepEqual(layer.GIDMap, options.GIDMap) } diff --git a/vendor/github.com/containers/storage/types/utils.go b/vendor/github.com/containers/storage/types/utils.go index b7ab07342..4dd1a786e 100644 --- a/vendor/github.com/containers/storage/types/utils.go +++ b/vendor/github.com/containers/storage/types/utils.go @@ -87,7 +87,7 @@ func getRootlessRuntimeDirIsolated(env rootlessRuntimeDirEnvironment) (string, e if tmpPerUserDir != "" { if _, err := env.systemLstat(tmpPerUserDir); os.IsNotExist(err) { if err := os.Mkdir(tmpPerUserDir, 0700); err != nil { - logrus.Errorf("failed to create temp directory for user: %v", err) + logrus.Errorf("Failed to create temp directory for user: %v", err) } else { return tmpPerUserDir, nil } diff --git a/vendor/github.com/containers/storage/userns.go b/vendor/github.com/containers/storage/userns.go index 3ada41f73..523c92dc8 100644 --- a/vendor/github.com/containers/storage/userns.go +++ b/vendor/github.com/containers/storage/userns.go @@ -43,7 +43,7 @@ func getAdditionalSubIDs(username string) (*idSet, *idSet, error) { } mappings, err := idtools.NewIDMappings(username, username) if err != nil { - logrus.Errorf("cannot find mappings for user %q: %v", username, err) + logrus.Errorf("Cannot find mappings for user %q: %v", username, err) } else { uids = getHostIDs(mappings.UIDs()) gids = getHostIDs(mappings.GIDs()) diff --git a/vendor/github.com/klauspost/compress/LICENSE b/vendor/github.com/klauspost/compress/LICENSE index 6cd1e9627..87d557477 100644 --- a/vendor/github.com/klauspost/compress/LICENSE +++ b/vendor/github.com/klauspost/compress/LICENSE @@ -290,3 +290,15 @@ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +----------------- + +Files: s2/cmd/internal/filepathx/* + +Copyright 2016 The filepathx Authors + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/github.com/klauspost/compress/README.md b/vendor/github.com/klauspost/compress/README.md index 235dc7cc6..3429879eb 100644 --- a/vendor/github.com/klauspost/compress/README.md +++ b/vendor/github.com/klauspost/compress/README.md @@ -17,12 +17,17 @@ This package provides various compression algorithms. # changelog
+* Aug 30, 2021 (v1.13.5)
+ * gz/zlib/flate: Alias stdlib errors [#425](https://github.com/klauspost/compress/pull/425)
+ * s2: Add block support to commandline tools [#413](https://github.com/klauspost/compress/pull/413)
+ * zstd: pooledZipWriter should return Writers to the same pool [#426](https://github.com/klauspost/compress/pull/426)
+ * Removed golang/snappy as external dependency for tests [#421](https://github.com/klauspost/compress/pull/421)
+
* Aug 12, 2021 (v1.13.4)
* Add [snappy replacement package](https://github.com/klauspost/compress/tree/master/snappy).
* zstd: Fix incorrect encoding in "best" mode [#415](https://github.com/klauspost/compress/pull/415)
* Aug 3, 2021 (v1.13.3)
-
* zstd: Improve Best compression [#404](https://github.com/klauspost/compress/pull/404)
* zstd: Fix WriteTo error forwarding [#411](https://github.com/klauspost/compress/pull/411)
* gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. [#406](https://github.com/klauspost/compress/pull/406)
@@ -31,7 +36,6 @@ This package provides various compression algorithms. * zstd: use SpeedBestCompression for level >= 10 [#410](https://github.com/klauspost/compress/pull/410)
* Jun 14, 2021 (v1.13.1)
-
* s2: Add full Snappy output support [#396](https://github.com/klauspost/compress/pull/396)
* zstd: Add configurable [Decoder window](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithDecoderMaxWindow) size [#394](https://github.com/klauspost/compress/pull/394)
* gzhttp: Add header to skip compression [#389](https://github.com/klauspost/compress/pull/389)
@@ -64,6 +68,9 @@ This package provides various compression algorithms. * zstd: Reduce memory usage further when using [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) [#346](https://github.com/klauspost/compress/pull/346)
* s2: Fix potential problem with amd64 assembly and profilers [#349](https://github.com/klauspost/compress/pull/349)
+<details>
+ <summary>See changes prior to v1.12.1</summary>
+
* Mar 26, 2021 (v1.11.13)
* zstd: Big speedup on small dictionary encodes [#344](https://github.com/klauspost/compress/pull/344) [#345](https://github.com/klauspost/compress/pull/345)
* zstd: Add [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) encoder option [#336](https://github.com/klauspost/compress/pull/336)
@@ -118,6 +125,7 @@ This package provides various compression algorithms. * zstd: Add experimental compression [dictionaries](https://github.com/klauspost/compress/tree/master/zstd#dictionaries) [#281](https://github.com/klauspost/compress/pull/281)
* zstd: Fix mixed Write and ReadFrom calls [#282](https://github.com/klauspost/compress/pull/282)
* inflate/gz: Limit variable shifts, ~5% faster decompression [#274](https://github.com/klauspost/compress/pull/274)
+</details>
<details>
<summary>See changes prior to v1.11.0</summary>
diff --git a/vendor/github.com/klauspost/compress/flate/fast_encoder.go b/vendor/github.com/klauspost/compress/flate/fast_encoder.go index 347ac2c90..a746eb733 100644 --- a/vendor/github.com/klauspost/compress/flate/fast_encoder.go +++ b/vendor/github.com/klauspost/compress/flate/fast_encoder.go @@ -215,24 +215,15 @@ func (e *fastGen) Reset() { func matchLen(a, b []byte) int { b = b[:len(a)] var checked int - if len(a) >= 4 { - // Try 4 bytes first - if diff := binary.LittleEndian.Uint32(a) ^ binary.LittleEndian.Uint32(b); diff != 0 { - return bits.TrailingZeros32(diff) >> 3 - } - // Switch to 8 byte matching. - checked = 4 - a = a[4:] - b = b[4:] - for len(a) >= 8 { - b = b[:len(a)] - if diff := binary.LittleEndian.Uint64(a) ^ binary.LittleEndian.Uint64(b); diff != 0 { - return checked + (bits.TrailingZeros64(diff) >> 3) - } - checked += 8 - a = a[8:] - b = b[8:] + + for len(a) >= 8 { + b = b[:len(a)] + if diff := binary.LittleEndian.Uint64(a) ^ binary.LittleEndian.Uint64(b); diff != 0 { + return checked + (bits.TrailingZeros64(diff) >> 3) } + checked += 8 + a = a[8:] + b = b[8:] } b = b[:len(a)] for i := range a { diff --git a/vendor/github.com/klauspost/compress/huff0/decompress.go b/vendor/github.com/klauspost/compress/huff0/decompress.go index 41703bba4..9b7cc8e97 100644 --- a/vendor/github.com/klauspost/compress/huff0/decompress.go +++ b/vendor/github.com/klauspost/compress/huff0/decompress.go @@ -344,35 +344,241 @@ func (d *Decoder) decompress1X8Bit(dst, src []byte) ([]byte, error) { var buf [256]byte var off uint8 - shift := (8 - d.actualTableLog) & 7 - - //fmt.Printf("mask: %b, tl:%d\n", mask, d.actualTableLog) - for br.off >= 4 { - br.fillFast() - v := dt[br.peekByteFast()>>shift] - br.advance(uint8(v.entry)) - buf[off+0] = uint8(v.entry >> 8) - - v = dt[br.peekByteFast()>>shift] - br.advance(uint8(v.entry)) - buf[off+1] = uint8(v.entry >> 8) - - v = dt[br.peekByteFast()>>shift] - br.advance(uint8(v.entry)) - buf[off+2] = uint8(v.entry >> 8) - - v = dt[br.peekByteFast()>>shift] - br.advance(uint8(v.entry)) - buf[off+3] = uint8(v.entry >> 8) - - off += 4 - if off == 0 { - if len(dst)+256 > maxDecodedSize { - br.close() - return nil, ErrMaxDecodedSizeExceeded + switch d.actualTableLog { + case 8: + const shift = 8 - 8 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 7: + const shift = 8 - 7 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 6: + const shift = 8 - 6 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 5: + const shift = 8 - 5 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 4: + const shift = 8 - 4 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 3: + const shift = 8 - 3 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 2: + const shift = 8 - 2 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) + } + } + case 1: + const shift = 8 - 1 + for br.off >= 4 { + br.fillFast() + v := dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+0] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+1] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+2] = uint8(v.entry >> 8) + + v = dt[uint8(br.value>>(56+shift))] + br.advance(uint8(v.entry)) + buf[off+3] = uint8(v.entry >> 8) + + off += 4 + if off == 0 { + if len(dst)+256 > maxDecodedSize { + br.close() + return nil, ErrMaxDecodedSizeExceeded + } + dst = append(dst, buf[:]...) } - dst = append(dst, buf[:]...) } + default: + return nil, fmt.Errorf("invalid tablelog: %d", d.actualTableLog) } if len(dst)+int(off) > maxDecodedSize { @@ -383,6 +589,8 @@ func (d *Decoder) decompress1X8Bit(dst, src []byte) ([]byte, error) { // br < 4, so uint8 is fine bitsLeft := int8(uint8(br.off)*8 + (64 - br.bitsRead)) + shift := (8 - d.actualTableLog) & 7 + for bitsLeft > 0 { if br.bitsRead >= 64-8 { for br.off > 0 { @@ -423,24 +631,24 @@ func (d *Decoder) decompress1X8BitExactly(dst, src []byte) ([]byte, error) { var buf [256]byte var off uint8 - const shift = 0 + const shift = 56 //fmt.Printf("mask: %b, tl:%d\n", mask, d.actualTableLog) for br.off >= 4 { br.fillFast() - v := dt[br.peekByteFast()>>shift] + v := dt[uint8(br.value>>shift)] br.advance(uint8(v.entry)) buf[off+0] = uint8(v.entry >> 8) - v = dt[br.peekByteFast()>>shift] + v = dt[uint8(br.value>>shift)] br.advance(uint8(v.entry)) buf[off+1] = uint8(v.entry >> 8) - v = dt[br.peekByteFast()>>shift] + v = dt[uint8(br.value>>shift)] br.advance(uint8(v.entry)) buf[off+2] = uint8(v.entry >> 8) - v = dt[br.peekByteFast()>>shift] + v = dt[uint8(br.value>>shift)] br.advance(uint8(v.entry)) buf[off+3] = uint8(v.entry >> 8) @@ -474,7 +682,7 @@ func (d *Decoder) decompress1X8BitExactly(dst, src []byte) ([]byte, error) { br.close() return nil, ErrMaxDecodedSizeExceeded } - v := dt[br.peekByteFast()>>shift] + v := dt[br.peekByteFast()] nBits := uint8(v.entry) br.advance(nBits) bitsLeft -= int8(nBits) @@ -709,7 +917,6 @@ func (d *Decoder) decompress4X8bit(dst, src []byte) ([]byte, error) { shift := (8 - d.actualTableLog) & 7 const tlSize = 1 << 8 - const tlMask = tlSize - 1 single := d.dt.single[:tlSize] // Use temp table to avoid bound checks/append penalty. diff --git a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash.go b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash.go index 426b9cac7..2c112a0ab 100644 --- a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash.go +++ b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash.go @@ -195,7 +195,6 @@ func (d *Digest) UnmarshalBinary(b []byte) error { b, d.v4 = consumeUint64(b) b, d.total = consumeUint64(b) copy(d.mem[:], b) - b = b[len(d.mem):] d.n = int(d.total % uint64(len(d.mem))) return nil } diff --git a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.go b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.go index 3ddbd5c0b..0ae847f75 100644 --- a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.go +++ b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.go @@ -9,4 +9,4 @@ package xxhash func Sum64(b []byte) uint64 //go:noescape -func writeBlocks(*Digest, []byte) int +func writeBlocks(d *Digest, b []byte) int diff --git a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s index 2c9c5357a..be8db5bf7 100644 --- a/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s +++ b/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s @@ -6,7 +6,7 @@ // Register allocation: // AX h -// CX pointer to advance through b +// SI pointer to advance through b // DX n // BX loop end // R8 v1, k1 @@ -16,39 +16,39 @@ // R12 tmp // R13 prime1v // R14 prime2v -// R15 prime4v +// DI prime4v -// round reads from and advances the buffer pointer in CX. +// round reads from and advances the buffer pointer in SI. // It assumes that R13 has prime1v and R14 has prime2v. #define round(r) \ - MOVQ (CX), R12 \ - ADDQ $8, CX \ + MOVQ (SI), R12 \ + ADDQ $8, SI \ IMULQ R14, R12 \ ADDQ R12, r \ ROLQ $31, r \ IMULQ R13, r // mergeRound applies a merge round on the two registers acc and val. -// It assumes that R13 has prime1v, R14 has prime2v, and R15 has prime4v. +// It assumes that R13 has prime1v, R14 has prime2v, and DI has prime4v. #define mergeRound(acc, val) \ IMULQ R14, val \ ROLQ $31, val \ IMULQ R13, val \ XORQ val, acc \ IMULQ R13, acc \ - ADDQ R15, acc + ADDQ DI, acc // func Sum64(b []byte) uint64 TEXT ·Sum64(SB), NOSPLIT, $0-32 // Load fixed primes. MOVQ ·prime1v(SB), R13 MOVQ ·prime2v(SB), R14 - MOVQ ·prime4v(SB), R15 + MOVQ ·prime4v(SB), DI // Load slice. - MOVQ b_base+0(FP), CX + MOVQ b_base+0(FP), SI MOVQ b_len+8(FP), DX - LEAQ (CX)(DX*1), BX + LEAQ (SI)(DX*1), BX // The first loop limit will be len(b)-32. SUBQ $32, BX @@ -65,14 +65,14 @@ TEXT ·Sum64(SB), NOSPLIT, $0-32 XORQ R11, R11 SUBQ R13, R11 - // Loop until CX > BX. + // Loop until SI > BX. blockLoop: round(R8) round(R9) round(R10) round(R11) - CMPQ CX, BX + CMPQ SI, BX JLE blockLoop MOVQ R8, AX @@ -100,16 +100,16 @@ noBlocks: afterBlocks: ADDQ DX, AX - // Right now BX has len(b)-32, and we want to loop until CX > len(b)-8. + // Right now BX has len(b)-32, and we want to loop until SI > len(b)-8. ADDQ $24, BX - CMPQ CX, BX + CMPQ SI, BX JG fourByte wordLoop: // Calculate k1. - MOVQ (CX), R8 - ADDQ $8, CX + MOVQ (SI), R8 + ADDQ $8, SI IMULQ R14, R8 ROLQ $31, R8 IMULQ R13, R8 @@ -117,18 +117,18 @@ wordLoop: XORQ R8, AX ROLQ $27, AX IMULQ R13, AX - ADDQ R15, AX + ADDQ DI, AX - CMPQ CX, BX + CMPQ SI, BX JLE wordLoop fourByte: ADDQ $4, BX - CMPQ CX, BX + CMPQ SI, BX JG singles - MOVL (CX), R8 - ADDQ $4, CX + MOVL (SI), R8 + ADDQ $4, SI IMULQ R13, R8 XORQ R8, AX @@ -138,19 +138,19 @@ fourByte: singles: ADDQ $4, BX - CMPQ CX, BX + CMPQ SI, BX JGE finalize singlesLoop: - MOVBQZX (CX), R12 - ADDQ $1, CX + MOVBQZX (SI), R12 + ADDQ $1, SI IMULQ ·prime5v(SB), R12 XORQ R12, AX ROLQ $11, AX IMULQ R13, AX - CMPQ CX, BX + CMPQ SI, BX JL singlesLoop finalize: @@ -179,13 +179,13 @@ TEXT ·writeBlocks(SB), NOSPLIT, $0-40 MOVQ ·prime2v(SB), R14 // Load slice. - MOVQ arg1_base+8(FP), CX - MOVQ arg1_len+16(FP), DX - LEAQ (CX)(DX*1), BX + MOVQ b_base+8(FP), SI + MOVQ b_len+16(FP), DX + LEAQ (SI)(DX*1), BX SUBQ $32, BX // Load vN from d. - MOVQ arg+0(FP), AX + MOVQ d+0(FP), AX MOVQ 0(AX), R8 // v1 MOVQ 8(AX), R9 // v2 MOVQ 16(AX), R10 // v3 @@ -199,7 +199,7 @@ blockLoop: round(R10) round(R11) - CMPQ CX, BX + CMPQ SI, BX JLE blockLoop // Copy vN back to d. @@ -208,8 +208,8 @@ blockLoop: MOVQ R10, 16(AX) MOVQ R11, 24(AX) - // The number of bytes written is CX minus the old base pointer. - SUBQ arg1_base+8(FP), CX - MOVQ CX, ret+32(FP) + // The number of bytes written is SI minus the old base pointer. + SUBQ b_base+8(FP), SI + MOVQ SI, ret+32(FP) RET diff --git a/vendor/github.com/mattn/go-isatty/.travis.yml b/vendor/github.com/mattn/go-isatty/.travis.yml deleted file mode 100644 index 604314dd4..000000000 --- a/vendor/github.com/mattn/go-isatty/.travis.yml +++ /dev/null @@ -1,14 +0,0 @@ -language: go -sudo: false -go: - - 1.13.x - - tip - -before_install: - - go get -t -v ./... - -script: - - ./go.test.sh - -after_success: - - bash <(curl -s https://codecov.io/bash) diff --git a/vendor/github.com/mattn/go-isatty/go.mod b/vendor/github.com/mattn/go-isatty/go.mod index 605c4c221..c9a20b7f3 100644 --- a/vendor/github.com/mattn/go-isatty/go.mod +++ b/vendor/github.com/mattn/go-isatty/go.mod @@ -2,4 +2,4 @@ module github.com/mattn/go-isatty go 1.12 -require golang.org/x/sys v0.0.0-20200116001909-b77594299b42 +require golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c diff --git a/vendor/github.com/mattn/go-isatty/isatty_bsd.go b/vendor/github.com/mattn/go-isatty/isatty_bsd.go index 711f28808..39bbcf00f 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_bsd.go +++ b/vendor/github.com/mattn/go-isatty/isatty_bsd.go @@ -1,3 +1,4 @@ +//go:build (darwin || freebsd || openbsd || netbsd || dragonfly) && !appengine // +build darwin freebsd openbsd netbsd dragonfly // +build !appengine diff --git a/vendor/github.com/mattn/go-isatty/isatty_others.go b/vendor/github.com/mattn/go-isatty/isatty_others.go index ff714a376..31503226f 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_others.go +++ b/vendor/github.com/mattn/go-isatty/isatty_others.go @@ -1,4 +1,5 @@ -// +build appengine js nacl +//go:build appengine || js || nacl || wasm +// +build appengine js nacl wasm package isatty diff --git a/vendor/github.com/mattn/go-isatty/isatty_plan9.go b/vendor/github.com/mattn/go-isatty/isatty_plan9.go index c5b6e0c08..bae7f9bb3 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_plan9.go +++ b/vendor/github.com/mattn/go-isatty/isatty_plan9.go @@ -1,3 +1,4 @@ +//go:build plan9 // +build plan9 package isatty diff --git a/vendor/github.com/mattn/go-isatty/isatty_solaris.go b/vendor/github.com/mattn/go-isatty/isatty_solaris.go index bdd5c79a0..0c3acf2dc 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_solaris.go +++ b/vendor/github.com/mattn/go-isatty/isatty_solaris.go @@ -1,5 +1,5 @@ -// +build solaris -// +build !appengine +//go:build solaris && !appengine +// +build solaris,!appengine package isatty @@ -8,10 +8,9 @@ import ( ) // IsTerminal returns true if the given file descriptor is a terminal. -// see: http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libbc/libc/gen/common/isatty.c +// see: https://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/gen/isatty.c func IsTerminal(fd uintptr) bool { - var termio unix.Termio - err := unix.IoctlSetTermio(int(fd), unix.TCGETA, &termio) + _, err := unix.IoctlGetTermio(int(fd), unix.TCGETA) return err == nil } diff --git a/vendor/github.com/mattn/go-isatty/isatty_tcgets.go b/vendor/github.com/mattn/go-isatty/isatty_tcgets.go index 31a1ca973..67787657f 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_tcgets.go +++ b/vendor/github.com/mattn/go-isatty/isatty_tcgets.go @@ -1,4 +1,5 @@ -// +build linux aix +//go:build (linux || aix || zos) && !appengine +// +build linux aix zos // +build !appengine package isatty diff --git a/vendor/github.com/mattn/go-isatty/isatty_windows.go b/vendor/github.com/mattn/go-isatty/isatty_windows.go index 1fa869154..8e3c99171 100644 --- a/vendor/github.com/mattn/go-isatty/isatty_windows.go +++ b/vendor/github.com/mattn/go-isatty/isatty_windows.go @@ -1,5 +1,5 @@ -// +build windows -// +build !appengine +//go:build windows && !appengine +// +build windows,!appengine package isatty @@ -76,7 +76,7 @@ func isCygwinPipeName(name string) bool { } // getFileNameByHandle use the undocomented ntdll NtQueryObject to get file full name from file handler -// since GetFileInformationByHandleEx is not avilable under windows Vista and still some old fashion +// since GetFileInformationByHandleEx is not available under windows Vista and still some old fashion // guys are using Windows XP, this is a workaround for those guys, it will also work on system from // Windows vista to 10 // see https://stackoverflow.com/a/18792477 for details diff --git a/vendor/github.com/mattn/go-isatty/renovate.json b/vendor/github.com/mattn/go-isatty/renovate.json deleted file mode 100644 index 5ae9d96b7..000000000 --- a/vendor/github.com/mattn/go-isatty/renovate.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "extends": [ - "config:base" - ], - "postUpdateOptions": [ - "gomodTidy" - ] -} diff --git a/vendor/modules.txt b/vendor/modules.txt index 1fb03d302..1927f8285 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -202,7 +202,7 @@ github.com/containers/psgo/internal/dev github.com/containers/psgo/internal/host github.com/containers/psgo/internal/proc github.com/containers/psgo/internal/process -# github.com/containers/storage v1.36.0 +# github.com/containers/storage v1.36.1-0.20210929132900-162a0bf730ce github.com/containers/storage github.com/containers/storage/drivers github.com/containers/storage/drivers/aufs @@ -405,7 +405,7 @@ github.com/json-iterator/go # github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a github.com/juju/ansiterm github.com/juju/ansiterm/tabwriter -# github.com/klauspost/compress v1.13.5 +# github.com/klauspost/compress v1.13.6 github.com/klauspost/compress github.com/klauspost/compress/flate github.com/klauspost/compress/fse @@ -423,7 +423,7 @@ github.com/manifoldco/promptui/list github.com/manifoldco/promptui/screenbuf # github.com/mattn/go-colorable v0.1.8 github.com/mattn/go-colorable -# github.com/mattn/go-isatty v0.0.12 +# github.com/mattn/go-isatty v0.0.14 github.com/mattn/go-isatty # github.com/mattn/go-runewidth v0.0.13 github.com/mattn/go-runewidth |