diff options
42 files changed, 523 insertions, 67 deletions
@@ -5,7 +5,7 @@ Podman (the POD MANager) is a tool for managing containers and images, volumes mounted into those containers, and pods made from groups of containers. Podman is based on libpod, a library for container lifecycle management that is also contained in this repository. The libpod library provides APIs for managing containers, pods, container images, and volumes. -* [Latest Version: 3.1.0](https://github.com/containers/podman/releases/latest) +* [Latest Version: 3.2.0](https://github.com/containers/podman/releases/latest) * Latest Remote client for Windows * Latest Remote client for MacOs * Latest Static Remote client for Linux diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 5ba5e251b..c6efff5dd 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,5 +1,130 @@ # Release Notes +## 3.2.0 +### Features +- Docker Compose is now supported with rootless Podman ([#9169](https://github.com/containers/podman/issues/9169)). +- The `podman network connect`, `podman network disconnect`, and `podman network reload` commands have been enabled for rootless Podman. +- An experimental new set of commands, `podman machine`, was added to assist in managing virtual machines containing a Podman server. These are intended for easing the use of Podman on OS X by handling the creation of a Linux VM for running Podman. +- The `podman generate kube` command can now be run on Podman named volumes (generating `PersistentVolumeClaim` YAML), in addition to pods and containers. +- The `podman play kube` command now supports two new options, `--ip` and `--mac`, to set static IPs and MAC addresses for created pods ([#8442](https://github.com/containers/podman/issues/8442) and [#9731](https://github.com/containers/podman/issues/9731)). +- The `podman play kube` command's support for `PersistentVolumeClaim` YAML has been greatly improved. +- The `podman generate kube` command now preserves the label used by `podman auto-update` to identify containers to update as a Kubernetes annotation, and the `podman play kube` command will convert this annotation back into a label. This allows `podman auto-update` to be used with containers created by `podman play kube`. +- The `podman play kube` command now supports Kubernetes `secretRef` YAML (using the secrets support from `podman secret`) for environment variables. +- Secrets can now be added to containers as environment variables using the `type=env` option to the `--secret` flag to `podman create` and `podman run`. +- The `podman start` command now supports the `--all` option, allowing all containers to be started simultaneously with a single command. The `--filter` option has also been added to filter which containers to start when `--all` is used. +- Filtering containers with the `--filter` option to `podman ps` and `podman start` now supports a new filter, `restart-policy`, to filter containers based on their restart policy. +- The `--group-add` option to rootless `podman run` and `podman create` now accepts a new value, `keep-groups`, which instructs Podman to retain the supplemental groups of the user running Podman in the created container. This is only supported with the `crun` OCI runtime. +- The `podman run` and `podman create` commands now support a new option, `--timeout`. This sets a maximum time the container is allowed to run, after which it is killed ([#6412](https://github.com/containers/podman/issues/6412)). +- The `podman run` and `podman create` commands now support a new option, `--pidfile`. This will create a file when the container is started containing the PID of the first process in the container. +- The `podman run` and `podman create` commands now support a new option, `--requires`. The `--requires` option adds dependency containers - containers that must be running before the current container. Commands like `podman start` will automatically start the requirements of a container before starting the container itself. +- Auto-updating containers can now be done with locally-built images, not just images hosted on a registry, by creating containers with the `io.containers.autoupdate` label set to `local`. +- Podman now supports the [Container Device Interface](https://github.com/container-orchestrated-devices/container-device-interface) (CDI) standard. +- Podman now adds an entry to `/etc/hosts`, `host.containers.internal`, pointing to the current gateway (which, for root containers, is usually a bridge interface on the host system) ([#5651](https://github.com/containers/podman/issues/5651)). +- The `podman ps`, `podman pod ps`, `podman network list`, `podman secret list`, and `podman volume list` commands now support a `--noheading` option, which will cause Podman to omit the heading line including column names. +- The `podman unshare` command now supports a new flag, `--rootless-cni`, to join the rootless network namespace. This allows commands to be run in the same network environment as rootless containers with CNI networking. +- The `--security-opt unmask=` option to `podman run` and `podman create` now supports glob operations to unmask a group of paths at once (e.g. `podman run --security-opt unmask=/proc/* ...` will unmask all paths in `/proc` in the container). +- The `podman network prune` command now supports a `--filter` option to filter which networks will be pruned. + +### Changes +- The change in Podman 3.1.2 where the `:z` and `:Z` mount options for volumes were ignored for privileged containers has been reverted after discussion in [#10209](https://github.com/containers/podman/issues/10209). +- Podman's rootless CNI functionality no longer requires a sidecar container! The removal of the requirement for the `rootless-cni-infra` container means that rootless CNI is now usable on all architectures, not just AMD64, and no longer requires pulling an image ([#8709](https://github.com/containers/podman/issues/8709)). +- The Image handling code used by Podman has seen a major rewrite to improve code sharing with our other projects, Buildah and CRI-O. This should result in fewer bugs and performance gains in the long term. Work on this is still ongoing. +- The `podman auto-update` command now prunes previous versions of images after updating if they are unused, to prevent disk exhaustion after repeated updates ([#10190](https://github.com/containers/podman/issues/10190)). +- The `podman play kube` now treats environment variables configured as references to a `ConfigMap` as mandatory unless the `optional` parameter was set; this better matches the behavior of Kubernetes. +- Podman now supports the `--context=default` flag from Docker as a no-op for compatibility purposes. +- When Podman is run as root, but without `CAP_SYS_ADMIN` being available, it will run in a user namespace using the same code as rootless Podman (instead of failing outright). +- The `podman info` command now includes the path of the Seccomp profile Podman is using, available cgroup controllers, and whether Podman is connected to a remote service or running containers locally. +- Containers created with the `--rm` option now automatically use the `volatile` storage flag when available for their root filesystems, causing them not to write changes to disk as often as they will be removed at completion anyways. This should result in improved performance. +- The `podman generate systemd --new` command will now include environment variables referenced by the container in generated unit files if the value would be looked up from the system environment. +- Podman now requires that Conmon v2.0.24 be available. + +### Bugfixes +- Fixed a bug where the remote Podman client's `podman build` command did not support the `--arch`, `--platform`, and `--os`, options. +- Fixed a bug where the remote Podman client's `podman build` command ignored the `--rm=false` option ([#9869](https://github.com/containers/podman/issues/9869)). +- Fixed a bug where the remote Podman client's `podman build --iidfile` command could include extra output (in addition to just the image ID) in the image ID file written ([#10233](https://github.com/containers/podman/issues/10233)). +- Fixed a bug where the remote Podman client's `podman build` command did not preserve hardlinks when moving files into the container via `COPY` instructions ([#9893](https://github.com/containers/podman/issues/9893)). +- Fixed a bug where the `podman generate systemd --new` command could generate extra `--iidfile` arguments if the container was already created with one. +- Fixed a bug where the `podman generate systemd --new` command would generate unit files that did not include `RequiresMountsFor` lines ([#10493](https://github.com/containers/podman/issues/10493)). +- Fixed a bug where the `podman generate kube` command produced incorrect YAML for containers which bind-mounted both `/` and `/root` from the host system into the container ([#9764](https://github.com/containers/podman/issues/9764)). +- Fixed a bug where pods created by `podman play kube` from YAML that specified `ShareProcessNamespace` would only share the PID namespace (and not also the UTS, Network, and IPC namespaces) ([#9128](https://github.com/containers/podman/issues/9128)). +- Fixed a bug where the `podman network reload` command could generate spurious error messages when `iptables-nft` was in use. +- Fixed a bug where rootless Podman could fail to attach to containers when the user running Podman had a large UID. +- Fixed a bug where the `podman ps` command could fail with a `no such container` error due to a race condition with container removal ([#10120](https://github.com/containers/podman/issues/10120)). +- Fixed a bug where containers using the `slirp4netns` network mode and setting a custom `slirp4netns` subnet while using the `rootlesskit` port forwarder would not be able to forward ports ([#9828](https://github.com/containers/podman/issues/9828)). +- Fixed a bug where the `--filter ancestor=` option to `podman ps` did not require an exact match of the image name/ID to include a container in its results. +- Fixed a bug where the `--filter until=` option to `podman image prune` would prune images created after the specified time (instead of before). +- Fixed a bug where setting a custom Seccomp profile via the `seccomp_profile` option in `containers.conf` had no effect, and the default profile was used instead. +- Fixed a bug where the `--cgroup-parent` option to `podman create` and `podman run` was ignored in rootless Podman on cgroups v2 systems with the `cgroupfs` cgroup manager ([#10173](https://github.com/containers/podman/issues/10173)). +- Fixed a bug where the `IMAGE` and `NAME` variables in `podman container runlabel` were not being correctly substituted ([#10192](https://github.com/containers/podman/issues/10192)). +- Fixed a bug where Podman could freeze when creating containers with a specific combination of volumes and working directory ([#10216](https://github.com/containers/podman/issues/10216)). +- Fixed a bug where rootless Podman containers restarted by restart policy (e.g. containers created with `--restart=always`) would lose networking after being restarted ([#8047](https://github.com/containers/podman/issues/8047)). +- Fixed a bug where the `podman cp` command could not copy files into containers created with the `--pid=host` flag ([#9985](https://github.com/containers/podman/issues/9985)). +- Fixed a bug where filters to the `podman events` command could not be specified twice (if a filter is specified more than once, it will match if any of the given values match - logical or) ([#10507](https://github.com/containers/podman/issues/10507)). +- Fixed a bug where Podman would include IPv6 nameservers in `resolv.conf` in containers without IPv6 connectivity ([#10158](https://github.com/containers/podman/issues/10158)). +- Fixed a bug where containers could not be created with static IP addresses when connecting to a network using the `macvlan` driver ([#10283](https://github.com/containers/podman/issues/10283)). + +### API +- Fixed a bug where the Compat Create endpoint for Containers did not allow advanced network options to be set ([#10110](https://github.com/containers/podman/issues/10110)). +- Fixed a bug where the Compat Create endpoint for Containers ignored static IP information provided in the `IPAMConfig` block ([#10245](https://github.com/containers/podman/issues/10245)). +- Fixed a bug where the Compat Inspect endpoint for Containers returned null (instead of an empty list) for Networks when the container was not joined to a CNI network ([#9837](https://github.com/containers/podman/issues/9837)). +- Fixed a bug where the Compat Wait endpoint for Containers could miss containers exiting if they were immediately restarted. +- Fixed a bug where the Compat Create endpoint for Volumes required that the user provide a name for the new volume ([#9803](https://github.com/containers/podman/issues/9803)). +- Fixed a bug where the Libpod Info handler would sometimes not return the correct path to the Podman API socket. +- Fixed a bug where the Compat Events handler used the wrong name for container exited events (`died` instead of `die`) ([#10168](https://github.com/containers/podman/issues/10168)). +- Fixed a bug where the Compat Push endpoint for Images could leak goroutines if the remote end closed the connection prematurely. + +### Misc +- Updated Buildah to v1.21.0 +- Updated the containers/common library to v0.38.5 +- Updated the containers/storage library to v1.31.3 + +## 3.1.2 +### Bugfixes +- The Compat Export endpoint for Images now supports exporting multiple images at the same time to a single archive. +- Fixed a bug where images with empty layers were stored incorrectly, causing them to be unable to be pushed or saved. +- Fixed a bug where the `podman rmi` command could fail to remove corrupt images from storage. +- Fixed a bug where the remote Podman client's `podman save` command did not support the `oci-dir` and `docker-dir` formats ([#9742](https://github.com/containers/podman/issues/9742)). +- Fixed a bug where volume mounts from `podman play kube` created with a trailing `/` in the container path were were not properly superceding named volumes from the image ([#9618](https://github.com/containers/podman/issues/9618)). +- Fixed a bug where Podman could fail to build on 32-bit architectures. + +### Misc +- Updated the containers/image library to v5.11.1 + +## 3.1.1 +### Changes +- Podman now recognizes `trace` as a valid argument to the `--log-level` command. Trace logging is now the most verbose level of logging available. +- The `:z` and `:Z` options for volume mounts are now ignored when the container is privileged or is run with SELinux isolation disabled (`--security-opt label=disable`). This matches better matches Docker's behavior in this case. + +### Bugfixes +- Fixed a bug where pruning images with the `podman image prune` or `podman system prune` commands could cause Podman to panic. +- Fixed a bug where the `podman save` command did not properly error when the `--compress` flag was used with incompatible format types. +- Fixed a bug where the `--security-opt` and `--ulimit` options to the remote Podman client's `podman build` command were nonfunctional. +- Fixed a bug where the `--log-rusage` option to the remote Podman client's `podman build` command was nonfunctional ([#9489](https://github.com/containers/podman/issues/9889)). +- Fixed a bug where the `podman build` command could, in some circumstances, use the wrong OCI runtime ([#9459](https://github.com/containers/podman/issues/9459)). +- Fixed a bug where the remote Podman client's `podman build` command could return 0 despite failing ([#10029](https://github.com/containers/podman/issues/10029)). +- Fixed a bug where the `podman container runlabel` command did not properly expand the `IMAGE` and `NAME` variables in the label ([#9405](https://github.com/containers/podman/issues/9405)). +- Fixed a bug where poststop OCI hooks would be executed twice on containers started with the `--rm` argument ([#9983](https://github.com/containers/podman/issues/9983)). +- Fixed a bug where rootless Podman could fail to launch containers on cgroups v2 systems when the `cgroupfs` cgroup manager was in use. +- Fixed a bug where the `podman stats` command could error when statistics tracked exceeded the maximum size of a 32-bit signed integer ([#9979](https://github.com/containers/podman/issues/9979)). +- Fixed a bug where rootless Podman containers run with `--userns=keepid` (without a `--user` flag in addition) would grant exec sessions run in them too many capabilities ([#9919](https://github.com/containers/podman/issues/9919)). +- Fixed a bug where the `--authfile` option to `podman build` did not validate that the path given existed ([#9572](https://github.com/containers/podman/issues/9572)). +- Fixed a bug where the `--storage-opt` option to Podman was appending to, instead of overriding (as is documented), the default storage options. +- Fixed a bug where the `podman system service` connection did not function properly when run in a socket-activated systemd unit file as a non-root user. +- Fixed a bug where the `--network` option to the `podman play kube` command of the remote Podman client was being ignored ([#9698](https://github.com/containers/podman/issues/9698)). +- Fixed a bug where the `--log-driver` option to the `podman play kube` command was nonfunctional ([#10015](https://github.com/containers/podman/issues/10015)). + +### API +- Fixed a bug where the Libpod Create endpoint for Manifests did not properly validate the image the manifest was being created with. +- Fixed a bug where the Libpod DF endpoint could, in error cases, append an extra null to the JSON response, causing decode errors. +- Fixed a bug where the Libpod and Compat Top endpoint for Containers would return process names that included extra whitespace. +- Fixed a bug where the Compat Prune endpoint for Containers accepted too many types of filter. + +### Misc +- Updated Buildah to v1.20.1 +- Updated the containers/storage library to v1.29.0 +- Updated the containers/image library to v5.11.0 +- Updated the containers/common library to v0.36.0 + ## 3.1.0 ### Features - A set of new commands has been added to manage secrets! The `podman secret create`, `podman secret inspect`, `podman secret ls` and `podman secret rm` commands have been added to handle secrets, along with the `--secret` option to `podman run` and `podman create` to add secrets to containers. The initial driver for secrets does not support encryption - this will be added in a future release. diff --git a/docs/source/machine.rst b/docs/source/machine.rst index 16c042173..3962fca99 100644 --- a/docs/source/machine.rst +++ b/docs/source/machine.rst @@ -3,8 +3,13 @@ Machine :doc:`init <markdown/podman-machine-init.1>` Initialize a new virtual machine + :doc:`list <markdown/podman-machine-list.1>` List virtual machines + :doc:`rm <markdown/podman-machine-rm.1>` Remove a virtual machine + :doc:`ssh <markdown/podman-machine-ssh.1>` SSH into a virtual machine + :doc:`start <markdown/podman-machine-start.1>` Start a virtual machine + :doc:`stop <markdown/podman-machine-stop.1>` Stop a virtual machine @@ -25,7 +25,7 @@ require ( github.com/davecgh/go-spew v1.1.1 github.com/digitalocean/go-qemu v0.0.0-20210209191958-152a1535e49f github.com/docker/distribution v2.7.1+incompatible - github.com/docker/docker v20.10.6+incompatible + github.com/docker/docker v20.10.7+incompatible github.com/docker/go-connections v0.4.0 github.com/docker/go-plugins-helpers v0.0.0-20200102110956-c9a8a2d92ccc github.com/docker/go-units v0.4.0 @@ -49,7 +49,7 @@ require ( github.com/opencontainers/runc v1.0.0-rc95 github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 github.com/opencontainers/runtime-tools v0.9.0 - github.com/opencontainers/selinux v1.8.1 + github.com/opencontainers/selinux v1.8.2 github.com/pkg/errors v0.9.1 github.com/pmezard/go-difflib v1.0.0 github.com/rootless-containers/rootlesskit v0.14.2 @@ -61,7 +61,7 @@ require ( github.com/uber/jaeger-client-go v2.29.1+incompatible github.com/vbauerster/mpb/v6 v6.0.4 github.com/vishvananda/netlink v1.1.1-0.20201029203352-d40f9887b852 - go.etcd.io/bbolt v1.3.5 + go.etcd.io/bbolt v1.3.6 golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 @@ -90,6 +90,8 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= github.com/bitly/go-simplejson v0.5.0/go.mod h1:cXHtHw4XUPsvGaxgjIAn8PhEWG9NfngEKAMDJEczWVA= +github.com/bits-and-blooms/bitset v1.2.0 h1:Kn4yilvwNtMACtf1eYDlG8H77R07mZSPbMjLyS07ChA= +github.com/bits-and-blooms/bitset v1.2.0/go.mod h1:gIdJ4wp64HaoK2YrL1Q5/N7Y16edYb8uY+O0FJTyyDA= github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84= github.com/blang/semver v3.1.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= @@ -295,8 +297,9 @@ github.com/docker/distribution v2.7.1+incompatible h1:a5mlkVzth6W5A4fOsS3D2EO5BU github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= github.com/docker/docker v1.4.2-0.20191219165747-a9416c67da9f/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/docker v20.10.3-0.20210216175712-646072ed6524+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= -github.com/docker/docker v20.10.6+incompatible h1:oXI3Vas8TI8Eu/EjH4srKHJBVqraSzJybhxY7Om9faQ= github.com/docker/docker v20.10.6+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v20.10.7+incompatible h1:Z6O9Nhsjv+ayUEeI1IojKbYcsGdgYSNqxe1s2MYzUhQ= +github.com/docker/docker v20.10.7+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/docker-credential-helpers v0.6.3 h1:zI2p9+1NQYdnG6sMU26EX4aVGlqbInSQxQXLvzJ4RPQ= github.com/docker/docker-credential-helpers v0.6.3/go.mod h1:WRaJzqw3CTB9bk10avuGsjVBZsD05qeibJ1/TYlvc0Y= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= @@ -696,8 +699,9 @@ github.com/opencontainers/runtime-tools v0.9.0/go.mod h1:r3f7wjNzSs2extwzU3Y+6pK github.com/opencontainers/selinux v1.5.1/go.mod h1:yTcKuYAh6R95iDpefGLQaPaRwJFwyzAJufJyiTt7s0g= github.com/opencontainers/selinux v1.6.0/go.mod h1:VVGKuOLlE7v4PJyT6h7mNWvq1rzqiriPsEqVhc+svHE= github.com/opencontainers/selinux v1.8.0/go.mod h1:RScLhm78qiWa2gbVCcGkC7tCGdgk3ogry1nUQF8Evvo= -github.com/opencontainers/selinux v1.8.1 h1:yvEZh7CsfnJNwKzG9ZeXwbvR05RAZsu5RS/3vA6qFTA= github.com/opencontainers/selinux v1.8.1/go.mod h1:RScLhm78qiWa2gbVCcGkC7tCGdgk3ogry1nUQF8Evvo= +github.com/opencontainers/selinux v1.8.2 h1:c4ca10UMgRcvZ6h0K4HtS15UaVSBEaE+iln2LVpAuGc= +github.com/opencontainers/selinux v1.8.2/go.mod h1:MUIHuUEvKB1wtJjQdOyYRgOnLD2xAPP8dBsCoU0KuF8= github.com/openshift/imagebuilder v1.2.2-0.20210415181909-87f3e48c2656 h1:WaxyNFpmIDu4i6so9r6LVFIbSaXqsj8oitMitt86ae4= github.com/openshift/imagebuilder v1.2.2-0.20210415181909-87f3e48c2656/go.mod h1:9aJRczxCH0mvT6XQ+5STAQaPWz7OsWcU5/mRkt8IWeo= github.com/ostreedev/ostree-go v0.0.0-20190702140239-759a8c1ac913 h1:TnbXhKzrTOyuvWrjI8W6pcoI9XPbLHFXCdN2dtUw7Rw= @@ -852,7 +856,6 @@ github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17 github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae h1:4hwBBUfQCFe3Cym0ZtKyq7L16eZUtYKs+BaHDN6mAns= github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= github.com/willf/bitset v1.1.11-0.20200630133818-d5bec3311243/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4= -github.com/willf/bitset v1.1.11 h1:N7Z7E9UvjW+sGsEl7k/SJrvY2reP1A07MrGuCjIOjRE= github.com/willf/bitset v1.1.11/go.mod h1:83CECat5yLh5zVOf4P1ErAgKA5UDvKtgyUABdr3+MjI= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= github.com/xeipuuv/gojsonpointer v0.0.0-20190809123943-df4f5c81cb3b h1:6cLsL+2FW6dRAdl5iMtHgRogVCff0QpRi9653YmdcJA= @@ -872,8 +875,9 @@ github.com/yvasiyarov/gorelic v0.0.0-20141212073537-a9bba5b9ab50/go.mod h1:NUSPS github.com/yvasiyarov/newrelic_platform_go v0.0.0-20140908184405-b21fdbd4370f/go.mod h1:GlGEuHIJweS1mbCqG+7vt2nvWLzLLnRHbXz5JKd/Qbg= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= -go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0= go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ= +go.etcd.io/bbolt v1.3.6 h1:/ecaJf0sk1l4l6V4awd65v2C3ILy7MSj+s/x1ADCIMU= +go.etcd.io/bbolt v1.3.6/go.mod h1:qXsaaIqmgQH0T+OPdb99Bf+PKfBBQVAdyD6TY9G8XM4= go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489/go.mod h1:yVHk9ub3CSBatqGNg7GRmsnfLWtoW60w4eDYfh7vHDg= go.mozilla.org/pkcs7 v0.0.0-20200128120323-432b2356ecb1 h1:A/5uWzF44DlIgdm/PQFwfMkW0JX+cIcQi/SwLAmZP5M= go.mozilla.org/pkcs7 v0.0.0-20200128120323-432b2356ecb1/go.mod h1:SNgMg+EgDFwmvSmLRTNKC5fegJjB7v23qTQ0XLGUNHk= @@ -1054,6 +1058,7 @@ golang.org/x/sys v0.0.0-20200831180312-196b9ba8737a/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200909081042-eff7692f9009/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200916030750-2334cc1a136f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200922070232-aee5d888a860/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201101102859-da207088b7d1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= diff --git a/libpod/container_internal_linux.go b/libpod/container_internal_linux.go index 1b2f5a496..74a3fec32 100644 --- a/libpod/container_internal_linux.go +++ b/libpod/container_internal_linux.go @@ -1668,17 +1668,16 @@ func (c *Container) generateResolvConf() (string, error) { return "", err } - // Ensure that the container's /etc/resolv.conf is compatible with its - // network configuration. - // TODO: set ipv6 enable bool more sanely - resolv, err := resolvconf.FilterResolvDNS(contents, true, c.config.CreateNetNS) - if err != nil { - return "", errors.Wrapf(err, "error parsing host resolv.conf") - } - + ipv6 := false // Check if CNI gave back and DNS servers for us to add in cniResponse := c.state.NetworkStatus for _, i := range cniResponse { + for _, ip := range i.IPs { + // Note: only using To16() does not work since it also returns a vaild ip for ipv4 + if ip.Address.IP.To4() == nil && ip.Address.IP.To16() != nil { + ipv6 = true + } + } if i.DNS.Nameservers != nil { cniNameServers = append(cniNameServers, i.DNS.Nameservers...) logrus.Debugf("adding nameserver(s) from cni response of '%q'", i.DNS.Nameservers) @@ -1689,6 +1688,25 @@ func (c *Container) generateResolvConf() (string, error) { } } + if c.config.NetMode.IsSlirp4netns() { + ctrNetworkSlipOpts := []string{} + if c.config.NetworkOptions != nil { + ctrNetworkSlipOpts = append(ctrNetworkSlipOpts, c.config.NetworkOptions["slirp4netns"]...) + } + slirpOpts, err := parseSlirp4netnsNetworkOptions(c.runtime, ctrNetworkSlipOpts) + if err != nil { + return "", err + } + ipv6 = slirpOpts.enableIPv6 + } + + // Ensure that the container's /etc/resolv.conf is compatible with its + // network configuration. + resolv, err := resolvconf.FilterResolvDNS(contents, ipv6, c.config.CreateNetNS) + if err != nil { + return "", errors.Wrapf(err, "error parsing host resolv.conf") + } + dns := make([]net.IP, 0, len(c.runtime.config.Containers.DNSServers)) for _, i := range c.runtime.config.Containers.DNSServers { result := net.ParseIP(i) diff --git a/pkg/api/handlers/compat/events.go b/pkg/api/handlers/compat/events.go index 405e616c5..9fbac91e0 100644 --- a/pkg/api/handlers/compat/events.go +++ b/pkg/api/handlers/compat/events.go @@ -75,7 +75,7 @@ func GetEvents(w http.ResponseWriter, r *http.Request) { coder := json.NewEncoder(w) coder.SetEscapeHTML(true) - for stream := true; stream; stream = query.Stream { + for { select { case err := <-errorChannel: if err != nil { diff --git a/pkg/api/handlers/libpod/swagger.go b/pkg/api/handlers/libpod/swagger.go index 9450a70d9..19eced986 100644 --- a/pkg/api/handlers/libpod/swagger.go +++ b/pkg/api/handlers/libpod/swagger.go @@ -95,7 +95,7 @@ type swagInfoResponse struct { // swagger:response NetworkRmReport type swagNetworkRmReport struct { // in:body - Body entities.NetworkRmReport + Body []entities.NetworkRmReport } // Network inspect diff --git a/test/system/090-events.bats b/test/system/090-events.bats index 09c2d0c10..52936d7a0 100644 --- a/test/system/090-events.bats +++ b/test/system/090-events.bats @@ -27,7 +27,7 @@ load helpers } @test "image events" { - skip_if_remote "FIXME: remove events on podman-remote seem to be broken" + skip_if_remote "remote does not support --events-backend" pushedDir=$PODMAN_TMPDIR/dir mkdir -p $pushedDir @@ -86,7 +86,5 @@ function _events_disjunctive_filters() { } @test "events with disjunctive filters - default" { - # NOTE: the last event for bar doesn't show up reliably. - skip_if_remote "FIXME #10529: remote events lose data" _events_disjunctive_filters "" } diff --git a/test/system/500-networking.bats b/test/system/500-networking.bats index 63b9a7c14..55ec80bb2 100644 --- a/test/system/500-networking.bats +++ b/test/system/500-networking.bats @@ -329,4 +329,62 @@ load helpers run_podman network rm -f $mynetname } +@test "podman ipv6 in /etc/resolv.conf" { + ipv6_regex='([0-9A-Fa-f]{0,4}:){2,7}([0-9A-Fa-f]{0,4})(%\w+)?' + + # Make sure to read the correct /etc/resolv.conf file in case of systemd-resolved. + resolve_file=$(readlink -f /etc/resolv.conf) + if [[ "$resolve_file" == "/run/systemd/resolve/stub-resolv.conf" ]]; then + resolve_file="/run/systemd/resolve/resolv.conf" + fi + + # If the host doesn't have an ipv6 in resolv.conf skip this test. + # We should never modify resolv.conf on the host. + if ! grep -E "$ipv6_regex" "$resolve_file"; then + skip "This test needs an ipv6 nameserver in $resolve_file" + fi + + # ipv4 slirp + run_podman run --rm --network slirp4netns:enable_ipv6=false $IMAGE cat /etc/resolv.conf + if grep -E "$ipv6_regex" <<< $output; then + die "resolv.conf contains a ipv6 nameserver" + fi + + # ipv6 slirp + run_podman run --rm --network slirp4netns:enable_ipv6=true $IMAGE cat /etc/resolv.conf + # "is" does not like the ipv6 regex + if ! grep -E "$ipv6_regex" <<< $output; then + die "resolv.conf does not contain a ipv6 nameserver" + fi + + # ipv4 cni + local mysubnet=$(random_rfc1918_subnet) + local netname=testnet-$(random_string 10) + + run_podman network create --subnet $mysubnet.0/24 $netname + is "$output" ".*/cni/net.d/$netname.conflist" "output of 'network create'" + + run_podman run --rm --network $netname $IMAGE cat /etc/resolv.conf + if grep -E "$ipv6_regex" <<< $output; then + die "resolv.conf contains a ipv6 nameserver" + fi + + run_podman network rm -f $netname + + # ipv6 cni + mysubnet=fd00:4:4:4:4::/64 + netname=testnet-$(random_string 10) + + run_podman network create --subnet $mysubnet $netname + is "$output" ".*/cni/net.d/$netname.conflist" "output of 'network create'" + + run_podman run --rm --network $netname $IMAGE cat /etc/resolv.conf + # "is" does not like the ipv6 regex + if ! grep -E "$ipv6_regex" <<< $output; then + die "resolv.conf does not contain a ipv6 nameserver" + fi + + run_podman network rm -f $netname +} + # vim: filetype=sh diff --git a/vendor/github.com/willf/bitset/.gitignore b/vendor/github.com/bits-and-blooms/bitset/.gitignore index 5c204d28b..5c204d28b 100644 --- a/vendor/github.com/willf/bitset/.gitignore +++ b/vendor/github.com/bits-and-blooms/bitset/.gitignore diff --git a/vendor/github.com/willf/bitset/.travis.yml b/vendor/github.com/bits-and-blooms/bitset/.travis.yml index 094aa5ce0..094aa5ce0 100644 --- a/vendor/github.com/willf/bitset/.travis.yml +++ b/vendor/github.com/bits-and-blooms/bitset/.travis.yml diff --git a/vendor/github.com/willf/bitset/LICENSE b/vendor/github.com/bits-and-blooms/bitset/LICENSE index 59cab8a93..59cab8a93 100644 --- a/vendor/github.com/willf/bitset/LICENSE +++ b/vendor/github.com/bits-and-blooms/bitset/LICENSE diff --git a/vendor/github.com/willf/bitset/README.md b/vendor/github.com/bits-and-blooms/bitset/README.md index 50338e71d..97e83071e 100644 --- a/vendor/github.com/willf/bitset/README.md +++ b/vendor/github.com/bits-and-blooms/bitset/README.md @@ -2,10 +2,9 @@ *Go language library to map between non-negative integers and boolean values* -[](https://github.com/willf/bitset/actions?query=workflow%3ATest) -[](https://coveralls.io/github/willf/bitset?branch=master) +[](https://github.com/willf/bitset/actions?query=workflow%3ATest) [](https://goreportcard.com/report/github.com/willf/bitset) -[](https://pkg.go.dev/github.com/willf/bitset?tab=doc) +[](https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc) ## Description @@ -30,7 +29,7 @@ import ( "fmt" "math/rand" - "github.com/willf/bitset" + "github.com/bits-and-blooms/bitset" ) func main() { @@ -63,7 +62,7 @@ func main() { As an alternative to BitSets, one should check out the 'big' package, which provides a (less set-theoretical) view of bitsets. -Package documentation is at: https://pkg.go.dev/github.com/willf/bitset?tab=doc +Package documentation is at: https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc ## Memory Usage @@ -78,7 +77,7 @@ It is possible that a later version will match the `math/bits` return signature ## Installation ```bash -go get github.com/willf/bitset +go get github.com/bits-and-blooms/bitset ``` ## Contributing diff --git a/vendor/github.com/willf/bitset/azure-pipelines.yml b/vendor/github.com/bits-and-blooms/bitset/azure-pipelines.yml index f9b295918..f9b295918 100644 --- a/vendor/github.com/willf/bitset/azure-pipelines.yml +++ b/vendor/github.com/bits-and-blooms/bitset/azure-pipelines.yml diff --git a/vendor/github.com/willf/bitset/bitset.go b/vendor/github.com/bits-and-blooms/bitset/bitset.go index 21e889da2..d688806a5 100644 --- a/vendor/github.com/willf/bitset/bitset.go +++ b/vendor/github.com/bits-and-blooms/bitset/bitset.go @@ -209,6 +209,27 @@ func (b *BitSet) Flip(i uint) *BitSet { return b } +// FlipRange bit in [start, end). +// If end>= Cap(), this function will panic. +// Warning: using a very large value for 'end' +// may lead to a memory shortage and a panic: the caller is responsible +// for providing sensible parameters in line with their memory capacity. +func (b *BitSet) FlipRange(start, end uint) *BitSet { + if start >= end { + return b + } + + b.extendSetMaybe(end - 1) + var startWord uint = start >> log2WordSize + var endWord uint = end >> log2WordSize + b.set[startWord] ^= ^(^uint64(0) << (start & (wordSize - 1))) + for i := startWord; i < endWord; i++ { + b.set[i] = ^b.set[i] + } + b.set[endWord] ^= ^uint64(0) >> (-end & (wordSize - 1)) + return b +} + // Shrink shrinks BitSet so that the provided value is the last possible // set value. It clears all bits > the provided index and reduces the size // and length of the set. @@ -519,7 +540,7 @@ func (b *BitSet) Copy(c *BitSet) (count uint) { } // Count (number of set bits). -// Also known as "popcount" or "popularity count". +// Also known as "popcount" or "population count". func (b *BitSet) Count() uint { if b != nil && b.set != nil { return uint(popcntSlice(b.set)) diff --git a/vendor/github.com/bits-and-blooms/bitset/go.mod b/vendor/github.com/bits-and-blooms/bitset/go.mod new file mode 100644 index 000000000..c43e4522b --- /dev/null +++ b/vendor/github.com/bits-and-blooms/bitset/go.mod @@ -0,0 +1,3 @@ +module github.com/bits-and-blooms/bitset + +go 1.14 diff --git a/vendor/github.com/willf/bitset/go.sum b/vendor/github.com/bits-and-blooms/bitset/go.sum index e69de29bb..e69de29bb 100644 --- a/vendor/github.com/willf/bitset/go.sum +++ b/vendor/github.com/bits-and-blooms/bitset/go.sum diff --git a/vendor/github.com/willf/bitset/popcnt.go b/vendor/github.com/bits-and-blooms/bitset/popcnt.go index 76577a838..76577a838 100644 --- a/vendor/github.com/willf/bitset/popcnt.go +++ b/vendor/github.com/bits-and-blooms/bitset/popcnt.go diff --git a/vendor/github.com/willf/bitset/popcnt_19.go b/vendor/github.com/bits-and-blooms/bitset/popcnt_19.go index fc8ff4f36..fc8ff4f36 100644 --- a/vendor/github.com/willf/bitset/popcnt_19.go +++ b/vendor/github.com/bits-and-blooms/bitset/popcnt_19.go diff --git a/vendor/github.com/willf/bitset/popcnt_amd64.go b/vendor/github.com/bits-and-blooms/bitset/popcnt_amd64.go index 4cf64f24a..4cf64f24a 100644 --- a/vendor/github.com/willf/bitset/popcnt_amd64.go +++ b/vendor/github.com/bits-and-blooms/bitset/popcnt_amd64.go diff --git a/vendor/github.com/willf/bitset/popcnt_amd64.s b/vendor/github.com/bits-and-blooms/bitset/popcnt_amd64.s index 666c0dcc1..666c0dcc1 100644 --- a/vendor/github.com/willf/bitset/popcnt_amd64.s +++ b/vendor/github.com/bits-and-blooms/bitset/popcnt_amd64.s diff --git a/vendor/github.com/willf/bitset/popcnt_generic.go b/vendor/github.com/bits-and-blooms/bitset/popcnt_generic.go index 21e0ff7b4..21e0ff7b4 100644 --- a/vendor/github.com/willf/bitset/popcnt_generic.go +++ b/vendor/github.com/bits-and-blooms/bitset/popcnt_generic.go diff --git a/vendor/github.com/willf/bitset/trailing_zeros_18.go b/vendor/github.com/bits-and-blooms/bitset/trailing_zeros_18.go index c52b61be9..c52b61be9 100644 --- a/vendor/github.com/willf/bitset/trailing_zeros_18.go +++ b/vendor/github.com/bits-and-blooms/bitset/trailing_zeros_18.go diff --git a/vendor/github.com/willf/bitset/trailing_zeros_19.go b/vendor/github.com/bits-and-blooms/bitset/trailing_zeros_19.go index 36a988e71..36a988e71 100644 --- a/vendor/github.com/willf/bitset/trailing_zeros_19.go +++ b/vendor/github.com/bits-and-blooms/bitset/trailing_zeros_19.go diff --git a/vendor/github.com/docker/docker/pkg/archive/archive.go b/vendor/github.com/docker/docker/pkg/archive/archive.go index 8d14b7869..50b83c62c 100644 --- a/vendor/github.com/docker/docker/pkg/archive/archive.go +++ b/vendor/github.com/docker/docker/pkg/archive/archive.go @@ -402,10 +402,24 @@ func fillGo18FileTypeBits(mode int64, fi os.FileInfo) int64 { // ReadSecurityXattrToTarHeader reads security.capability xattr from filesystem // to a tar header func ReadSecurityXattrToTarHeader(path string, hdr *tar.Header) error { + const ( + // Values based on linux/include/uapi/linux/capability.h + xattrCapsSz2 = 20 + versionOffset = 3 + vfsCapRevision2 = 2 + vfsCapRevision3 = 3 + ) capability, _ := system.Lgetxattr(path, "security.capability") if capability != nil { + length := len(capability) + if capability[versionOffset] == vfsCapRevision3 { + // Convert VFS_CAP_REVISION_3 to VFS_CAP_REVISION_2 as root UID makes no + // sense outside the user namespace the archive is built in. + capability[versionOffset] = vfsCapRevision2 + length = xattrCapsSz2 + } hdr.Xattrs = make(map[string]string) - hdr.Xattrs["security.capability"] = string(capability) + hdr.Xattrs["security.capability"] = string(capability[:length]) } return nil } diff --git a/vendor/github.com/opencontainers/selinux/go-selinux/selinux_linux.go b/vendor/github.com/opencontainers/selinux/go-selinux/selinux_linux.go index 54597398b..a91a116f8 100644 --- a/vendor/github.com/opencontainers/selinux/go-selinux/selinux_linux.go +++ b/vendor/github.com/opencontainers/selinux/go-selinux/selinux_linux.go @@ -16,9 +16,9 @@ import ( "strings" "sync" + "github.com/bits-and-blooms/bitset" "github.com/opencontainers/selinux/pkg/pwalk" "github.com/pkg/errors" - "github.com/willf/bitset" "golang.org/x/sys/unix" ) diff --git a/vendor/github.com/willf/bitset/go.mod b/vendor/github.com/willf/bitset/go.mod deleted file mode 100644 index 583ecab78..000000000 --- a/vendor/github.com/willf/bitset/go.mod +++ /dev/null @@ -1,3 +0,0 @@ -module github.com/willf/bitset - -go 1.14 diff --git a/vendor/go.etcd.io/bbolt/.gitignore b/vendor/go.etcd.io/bbolt/.gitignore index 3bcd8cbaf..18312f004 100644 --- a/vendor/go.etcd.io/bbolt/.gitignore +++ b/vendor/go.etcd.io/bbolt/.gitignore @@ -3,3 +3,5 @@ *.swp /bin/ cover.out +/.idea +*.iml diff --git a/vendor/go.etcd.io/bbolt/.travis.yml b/vendor/go.etcd.io/bbolt/.travis.yml index 257dfdfee..452601e49 100644 --- a/vendor/go.etcd.io/bbolt/.travis.yml +++ b/vendor/go.etcd.io/bbolt/.travis.yml @@ -4,9 +4,10 @@ go_import_path: go.etcd.io/bbolt sudo: false go: -- 1.12 +- 1.15 before_install: +- go get -v golang.org/x/sys/unix - go get -v honnef.co/go/tools/... - go get -v github.com/kisielk/errcheck diff --git a/vendor/go.etcd.io/bbolt/Makefile b/vendor/go.etcd.io/bbolt/Makefile index 2968aaa61..21ecf48f6 100644 --- a/vendor/go.etcd.io/bbolt/Makefile +++ b/vendor/go.etcd.io/bbolt/Makefile @@ -2,8 +2,6 @@ BRANCH=`git rev-parse --abbrev-ref HEAD` COMMIT=`git rev-parse --short HEAD` GOLDFLAGS="-X main.branch $(BRANCH) -X main.commit $(COMMIT)" -default: build - race: @TEST_FREELIST_TYPE=hashmap go test -v -race -test.run="TestSimulate_(100op|1000op)" @echo "array freelist test" diff --git a/vendor/go.etcd.io/bbolt/README.md b/vendor/go.etcd.io/bbolt/README.md index c9e64b1a6..f1b4a7b2b 100644 --- a/vendor/go.etcd.io/bbolt/README.md +++ b/vendor/go.etcd.io/bbolt/README.md @@ -908,12 +908,14 @@ Below is a list of public, open source projects that use Bolt: * [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt. * [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners. * [BoltDbWeb](https://github.com/evnix/boltdbweb) - A web based GUI for BoltDB files. +* [BoltDB Viewer](https://github.com/zc310/rich_boltdb) - A BoltDB Viewer Can run on Windows、Linux、Android system. * [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend. * [btcwallet](https://github.com/btcsuite/btcwallet) - A bitcoin wallet. * [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining simple tx and key scans. * [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend. * [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations. +* [🌰 Chestnut](https://github.com/jrapoport/chestnut) - Chestnut is encrypted storage for Go. * [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware. * [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb. * [dcrwallet](https://github.com/decred/dcrwallet) - A wallet for the Decred cryptocurrency. @@ -938,9 +940,8 @@ Below is a list of public, open source projects that use Bolt: * [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite. * [MuLiFS](https://github.com/dankomiocevic/mulifs) - Music Library Filesystem creates a filesystem to organise your music files. * [NATS](https://github.com/nats-io/nats-streaming-server) - NATS Streaming uses bbolt for message and metadata storage. -* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard. -* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site. * [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system. +* [Rain](https://github.com/cenkalti/rain) - BitTorrent client and library. * [reef-pi](https://github.com/reef-pi/reef-pi) - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi. * [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service * [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read. diff --git a/vendor/go.etcd.io/bbolt/bolt_unix.go b/vendor/go.etcd.io/bbolt/bolt_unix.go index 2938fed58..4e5f65ccc 100644 --- a/vendor/go.etcd.io/bbolt/bolt_unix.go +++ b/vendor/go.etcd.io/bbolt/bolt_unix.go @@ -7,6 +7,8 @@ import ( "syscall" "time" "unsafe" + + "golang.org/x/sys/unix" ) // flock acquires an advisory lock on a file descriptor. @@ -49,13 +51,13 @@ func funlock(db *DB) error { // mmap memory maps a DB's data file. func mmap(db *DB, sz int) error { // Map the data file to memory. - b, err := syscall.Mmap(int(db.file.Fd()), 0, sz, syscall.PROT_READ, syscall.MAP_SHARED|db.MmapFlags) + b, err := unix.Mmap(int(db.file.Fd()), 0, sz, syscall.PROT_READ, syscall.MAP_SHARED|db.MmapFlags) if err != nil { return err } // Advise the kernel that the mmap is accessed randomly. - err = madvise(b, syscall.MADV_RANDOM) + err = unix.Madvise(b, syscall.MADV_RANDOM) if err != nil && err != syscall.ENOSYS { // Ignore not implemented error in kernel because it still works. return fmt.Errorf("madvise: %s", err) @@ -76,18 +78,9 @@ func munmap(db *DB) error { } // Unmap using the original byte slice. - err := syscall.Munmap(db.dataref) + err := unix.Munmap(db.dataref) db.dataref = nil db.data = nil db.datasz = 0 return err } - -// NOTE: This function is copied from stdlib because it is not available on darwin. -func madvise(b []byte, advice int) (err error) { - _, _, e1 := syscall.Syscall(syscall.SYS_MADVISE, uintptr(unsafe.Pointer(&b[0])), uintptr(len(b)), uintptr(advice)) - if e1 != 0 { - err = e1 - } - return -} diff --git a/vendor/go.etcd.io/bbolt/compact.go b/vendor/go.etcd.io/bbolt/compact.go new file mode 100644 index 000000000..e4fe91b04 --- /dev/null +++ b/vendor/go.etcd.io/bbolt/compact.go @@ -0,0 +1,114 @@ +package bbolt + +// Compact will create a copy of the source DB and in the destination DB. This may +// reclaim space that the source database no longer has use for. txMaxSize can be +// used to limit the transactions size of this process and may trigger intermittent +// commits. A value of zero will ignore transaction sizes. +// TODO: merge with: https://github.com/etcd-io/etcd/blob/b7f0f52a16dbf83f18ca1d803f7892d750366a94/mvcc/backend/backend.go#L349 +func Compact(dst, src *DB, txMaxSize int64) error { + // commit regularly, or we'll run out of memory for large datasets if using one transaction. + var size int64 + tx, err := dst.Begin(true) + if err != nil { + return err + } + defer tx.Rollback() + + if err := walk(src, func(keys [][]byte, k, v []byte, seq uint64) error { + // On each key/value, check if we have exceeded tx size. + sz := int64(len(k) + len(v)) + if size+sz > txMaxSize && txMaxSize != 0 { + // Commit previous transaction. + if err := tx.Commit(); err != nil { + return err + } + + // Start new transaction. + tx, err = dst.Begin(true) + if err != nil { + return err + } + size = 0 + } + size += sz + + // Create bucket on the root transaction if this is the first level. + nk := len(keys) + if nk == 0 { + bkt, err := tx.CreateBucket(k) + if err != nil { + return err + } + if err := bkt.SetSequence(seq); err != nil { + return err + } + return nil + } + + // Create buckets on subsequent levels, if necessary. + b := tx.Bucket(keys[0]) + if nk > 1 { + for _, k := range keys[1:] { + b = b.Bucket(k) + } + } + + // Fill the entire page for best compaction. + b.FillPercent = 1.0 + + // If there is no value then this is a bucket call. + if v == nil { + bkt, err := b.CreateBucket(k) + if err != nil { + return err + } + if err := bkt.SetSequence(seq); err != nil { + return err + } + return nil + } + + // Otherwise treat it as a key/value pair. + return b.Put(k, v) + }); err != nil { + return err + } + + return tx.Commit() +} + +// walkFunc is the type of the function called for keys (buckets and "normal" +// values) discovered by Walk. keys is the list of keys to descend to the bucket +// owning the discovered key/value pair k/v. +type walkFunc func(keys [][]byte, k, v []byte, seq uint64) error + +// walk walks recursively the bolt database db, calling walkFn for each key it finds. +func walk(db *DB, walkFn walkFunc) error { + return db.View(func(tx *Tx) error { + return tx.ForEach(func(name []byte, b *Bucket) error { + return walkBucket(b, nil, name, nil, b.Sequence(), walkFn) + }) + }) +} + +func walkBucket(b *Bucket, keypath [][]byte, k, v []byte, seq uint64, fn walkFunc) error { + // Execute callback. + if err := fn(keypath, k, v, seq); err != nil { + return err + } + + // If this is not a bucket then stop. + if v != nil { + return nil + } + + // Iterate over each child key/value. + keypath = append(keypath, k) + return b.ForEach(func(k, v []byte) error { + if v == nil { + bkt := b.Bucket(k) + return walkBucket(bkt, keypath, k, nil, bkt.Sequence(), fn) + } + return walkBucket(b, keypath, k, v, b.Sequence(), fn) + }) +} diff --git a/vendor/go.etcd.io/bbolt/db.go b/vendor/go.etcd.io/bbolt/db.go index 80b0095cc..a798c390a 100644 --- a/vendor/go.etcd.io/bbolt/db.go +++ b/vendor/go.etcd.io/bbolt/db.go @@ -120,6 +120,12 @@ type DB struct { // of truncate() and fsync() when growing the data file. AllocSize int + // Mlock locks database file in memory when set to true. + // It prevents major page faults, however used memory can't be reclaimed. + // + // Supported only on Unix via mlock/munlock syscalls. + Mlock bool + path string openFile func(string, int, os.FileMode) (*os.File, error) file *os.File @@ -188,6 +194,7 @@ func Open(path string, mode os.FileMode, options *Options) (*DB, error) { db.MmapFlags = options.MmapFlags db.NoFreelistSync = options.NoFreelistSync db.FreelistType = options.FreelistType + db.Mlock = options.Mlock // Set default values for later DB operations. db.MaxBatchSize = DefaultMaxBatchSize @@ -337,7 +344,8 @@ func (db *DB) mmap(minsz int) error { } // Ensure the size is at least the minimum size. - var size = int(info.Size()) + fileSize := int(info.Size()) + var size = fileSize if size < minsz { size = minsz } @@ -346,6 +354,13 @@ func (db *DB) mmap(minsz int) error { return err } + if db.Mlock { + // Unlock db memory + if err := db.munlock(fileSize); err != nil { + return err + } + } + // Dereference all mmap references before unmapping. if db.rwtx != nil { db.rwtx.root.dereference() @@ -361,6 +376,13 @@ func (db *DB) mmap(minsz int) error { return err } + if db.Mlock { + // Don't allow swapping of data file + if err := db.mlock(fileSize); err != nil { + return err + } + } + // Save references to the meta pages. db.meta0 = db.page(0).meta() db.meta1 = db.page(1).meta() @@ -422,12 +444,36 @@ func (db *DB) mmapSize(size int) (int, error) { return int(sz), nil } +func (db *DB) munlock(fileSize int) error { + if err := munlock(db, fileSize); err != nil { + return fmt.Errorf("munlock error: " + err.Error()) + } + return nil +} + +func (db *DB) mlock(fileSize int) error { + if err := mlock(db, fileSize); err != nil { + return fmt.Errorf("mlock error: " + err.Error()) + } + return nil +} + +func (db *DB) mrelock(fileSizeFrom, fileSizeTo int) error { + if err := db.munlock(fileSizeFrom); err != nil { + return err + } + if err := db.mlock(fileSizeTo); err != nil { + return err + } + return nil +} + // init creates a new database file and initializes its meta pages. func (db *DB) init() error { // Create two meta pages on a buffer. buf := make([]byte, db.pageSize*4) for i := 0; i < 2; i++ { - p := db.pageInBuffer(buf[:], pgid(i)) + p := db.pageInBuffer(buf, pgid(i)) p.id = pgid(i) p.flags = metaPageFlag @@ -444,13 +490,13 @@ func (db *DB) init() error { } // Write an empty freelist at page 3. - p := db.pageInBuffer(buf[:], pgid(2)) + p := db.pageInBuffer(buf, pgid(2)) p.id = pgid(2) p.flags = freelistPageFlag p.count = 0 // Write an empty leaf page at page 4. - p = db.pageInBuffer(buf[:], pgid(3)) + p = db.pageInBuffer(buf, pgid(3)) p.id = pgid(3) p.flags = leafPageFlag p.count = 0 @@ -462,6 +508,7 @@ func (db *DB) init() error { if err := fdatasync(db); err != nil { return err } + db.filesz = len(buf) return nil } @@ -973,6 +1020,12 @@ func (db *DB) grow(sz int) error { if err := db.file.Sync(); err != nil { return fmt.Errorf("file sync error: %s", err) } + if db.Mlock { + // unlock old file and lock new one + if err := db.mrelock(db.filesz, sz); err != nil { + return fmt.Errorf("mlock/munlock error: %s", err) + } + } } db.filesz = sz @@ -1064,6 +1117,11 @@ type Options struct { // OpenFile is used to open files. It defaults to os.OpenFile. This option // is useful for writing hermetic tests. OpenFile func(string, int, os.FileMode) (*os.File, error) + + // Mlock locks database file in memory when set to true. + // It prevents potential page faults, however + // used memory can't be reclaimed. (UNIX only) + Mlock bool } // DefaultOptions represent the options used if nil options are passed into Open(). diff --git a/vendor/go.etcd.io/bbolt/freelist_hmap.go b/vendor/go.etcd.io/bbolt/freelist_hmap.go index 02ef2be04..dbd67a1e7 100644 --- a/vendor/go.etcd.io/bbolt/freelist_hmap.go +++ b/vendor/go.etcd.io/bbolt/freelist_hmap.go @@ -4,7 +4,7 @@ import "sort" // hashmapFreeCount returns count of free pages(hashmap version) func (f *freelist) hashmapFreeCount() int { - // use the forwardmap to get the total count + // use the forwardMap to get the total count count := 0 for _, size := range f.forwardMap { count += int(size) @@ -41,7 +41,7 @@ func (f *freelist) hashmapAllocate(txid txid, n int) pgid { for pid := range bm { // remove the initial - f.delSpan(pid, uint64(size)) + f.delSpan(pid, size) f.allocs[pid] = txid @@ -51,7 +51,7 @@ func (f *freelist) hashmapAllocate(txid txid, n int) pgid { f.addSpan(pid+pgid(n), remain) for i := pgid(0); i < pgid(n); i++ { - delete(f.cache, pid+pgid(i)) + delete(f.cache, pid+i) } return pid } diff --git a/vendor/go.etcd.io/bbolt/go.mod b/vendor/go.etcd.io/bbolt/go.mod index c2366daef..96355a69b 100644 --- a/vendor/go.etcd.io/bbolt/go.mod +++ b/vendor/go.etcd.io/bbolt/go.mod @@ -2,4 +2,4 @@ module go.etcd.io/bbolt go 1.12 -require golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 +require golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d diff --git a/vendor/go.etcd.io/bbolt/go.sum b/vendor/go.etcd.io/bbolt/go.sum index 4ad15a488..c13f8f470 100644 --- a/vendor/go.etcd.io/bbolt/go.sum +++ b/vendor/go.etcd.io/bbolt/go.sum @@ -1,2 +1,2 @@ -golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 h1:LfCXLvNmTYH9kEmVgqbnsWfruoXZIrh4YBgqVHtDvw0= -golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d h1:L/IKR6COd7ubZrs2oTnTi73IhgqJ71c9s80WsQnh0Es= +golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= diff --git a/vendor/go.etcd.io/bbolt/mlock_unix.go b/vendor/go.etcd.io/bbolt/mlock_unix.go new file mode 100644 index 000000000..6a6c7b353 --- /dev/null +++ b/vendor/go.etcd.io/bbolt/mlock_unix.go @@ -0,0 +1,36 @@ +// +build !windows + +package bbolt + +import "golang.org/x/sys/unix" + +// mlock locks memory of db file +func mlock(db *DB, fileSize int) error { + sizeToLock := fileSize + if sizeToLock > db.datasz { + // Can't lock more than mmaped slice + sizeToLock = db.datasz + } + if err := unix.Mlock(db.dataref[:sizeToLock]); err != nil { + return err + } + return nil +} + +//munlock unlocks memory of db file +func munlock(db *DB, fileSize int) error { + if db.dataref == nil { + return nil + } + + sizeToUnlock := fileSize + if sizeToUnlock > db.datasz { + // Can't unlock more than mmaped slice + sizeToUnlock = db.datasz + } + + if err := unix.Munlock(db.dataref[:sizeToUnlock]); err != nil { + return err + } + return nil +} diff --git a/vendor/go.etcd.io/bbolt/mlock_windows.go b/vendor/go.etcd.io/bbolt/mlock_windows.go new file mode 100644 index 000000000..b4a36a493 --- /dev/null +++ b/vendor/go.etcd.io/bbolt/mlock_windows.go @@ -0,0 +1,11 @@ +package bbolt + +// mlock locks memory of db file +func mlock(_ *DB, _ int) error { + panic("mlock is supported only on UNIX systems") +} + +//munlock unlocks memory of db file +func munlock(_ *DB, _ int) error { + panic("munlock is supported only on UNIX systems") +} diff --git a/vendor/go.etcd.io/bbolt/tx.go b/vendor/go.etcd.io/bbolt/tx.go index 4b1a64a8b..869d41200 100644 --- a/vendor/go.etcd.io/bbolt/tx.go +++ b/vendor/go.etcd.io/bbolt/tx.go @@ -188,7 +188,6 @@ func (tx *Tx) Commit() error { } // If strict mode is enabled then perform a consistency check. - // Only the first consistency error is reported in the panic. if tx.db.StrictMode { ch := tx.Check() var errs []string @@ -393,7 +392,7 @@ func (tx *Tx) CopyFile(path string, mode os.FileMode) error { return err } - err = tx.Copy(f) + _, err = tx.WriteTo(f) if err != nil { _ = f.Close() return err diff --git a/vendor/modules.txt b/vendor/modules.txt index d0b65cbf5..786096f45 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -36,6 +36,8 @@ github.com/VividCortex/ewma github.com/acarl005/stripansi # github.com/beorn7/perks v1.0.1 github.com/beorn7/perks/quantile +# github.com/bits-and-blooms/bitset v1.2.0 +github.com/bits-and-blooms/bitset # github.com/blang/semver v3.5.1+incompatible github.com/blang/semver # github.com/buger/goterm v0.0.0-20181115115552-c206103e1f37 @@ -278,7 +280,7 @@ github.com/docker/distribution/registry/client/auth/challenge github.com/docker/distribution/registry/client/transport github.com/docker/distribution/registry/storage/cache github.com/docker/distribution/registry/storage/cache/memory -# github.com/docker/docker v20.10.6+incompatible +# github.com/docker/docker v20.10.7+incompatible github.com/docker/docker/api github.com/docker/docker/api/types github.com/docker/docker/api/types/blkiodev @@ -516,7 +518,7 @@ github.com/opencontainers/runtime-tools/generate github.com/opencontainers/runtime-tools/generate/seccomp github.com/opencontainers/runtime-tools/specerror github.com/opencontainers/runtime-tools/validate -# github.com/opencontainers/selinux v1.8.1 +# github.com/opencontainers/selinux v1.8.2 github.com/opencontainers/selinux/go-selinux github.com/opencontainers/selinux/go-selinux/label github.com/opencontainers/selinux/pkg/pwalk @@ -608,15 +610,13 @@ github.com/vishvananda/netlink github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae github.com/vishvananda/netns -# github.com/willf/bitset v1.1.11 -github.com/willf/bitset # github.com/xeipuuv/gojsonpointer v0.0.0-20190809123943-df4f5c81cb3b github.com/xeipuuv/gojsonpointer # github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 github.com/xeipuuv/gojsonreference # github.com/xeipuuv/gojsonschema v1.2.0 github.com/xeipuuv/gojsonschema -# go.etcd.io/bbolt v1.3.5 +# go.etcd.io/bbolt v1.3.6 go.etcd.io/bbolt # go.mozilla.org/pkcs7 v0.0.0-20200128120323-432b2356ecb1 go.mozilla.org/pkcs7 |