diff options
84 files changed, 805 insertions, 396 deletions
diff --git a/CODE-OF-CONDUCT.md b/CODE-OF-CONDUCT.md index b23672b1a..e418693f7 100644 --- a/CODE-OF-CONDUCT.md +++ b/CODE-OF-CONDUCT.md @@ -1,3 +1,3 @@ ## The Podman Project Community Code of Conduct -The Podman project which includes Libpod, follows the [Containers Community Code of Conduct](https://github.com/containers/common/blob/master/CODE-OF-CONDUCT.md). +The Podman project which includes Libpod, follows the [Containers Community Code of Conduct](https://github.com/containers/common/blob/main/CODE-OF-CONDUCT.md). @@ -389,10 +389,10 @@ cross: local-cross .PHONY: nixpkgs nixpkgs: @nix run \ - -f channel:nixos-20.09 nix-prefetch-git \ + -f channel:nixos-21.05 nix-prefetch-git \ -c nix-prefetch-git \ --no-deepClone \ - https://github.com/nixos/nixpkgs refs/heads/nixos-20.09 > nix/nixpkgs.json + https://github.com/nixos/nixpkgs refs/heads/nixos-21.05 > nix/nixpkgs.json # Build statically linked binary .PHONY: static @@ -437,7 +437,7 @@ $(MANPAGES): %: %.md .install.md2man docdir -e 's/\[\(podman[^]]*\)\]/\1/g' \ -e 's/\[\([^]]*\)](http[^)]\+)/\1/g' \ -e 's;<\(/\)\?\(a\|a\s\+[^>]*\|sup\)>;;g' \ - -e 's/\\$// /g' $< | \ + -e 's/\\$$/ /g' $< | \ $(GOMD2MAN) -in /dev/stdin -out $(subst source/markdown,build/man,$@) .PHONY: docdir diff --git a/SECURITY.md b/SECURITY.md index 03a192044..1f6d5088d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,3 +1,3 @@ ## Security and Disclosure Information Policy for the Libpod Project -The Libpod Project follows the [Security and Disclosure Information Policy](https://github.com/containers/common/blob/master/SECURITY.md) for the Containers Projects. +The Libpod Project follows the [Security and Disclosure Information Policy](https://github.com/containers/common/blob/main/SECURITY.md) for the Containers Projects. diff --git a/docs/MANPAGE_SYNTAX.md b/docs/MANPAGE_SYNTAX.md index c9b677688..9794a0c3d 100644 --- a/docs/MANPAGE_SYNTAX.md +++ b/docs/MANPAGE_SYNTAX.md @@ -4,7 +4,7 @@ podman\-command - short description ## SYNOPSIS -(Shows the command structure.) +(Shows the command structure. If the command can be written in two different ways, both of them have to be shown.) **podman command** [*optional*] *mandatory value* @@ -16,7 +16,7 @@ podman\-command - short description **podman subcommand command** [*optional*] *value1* | *value2* -(If an optinal value follows a mandatory one.) +(If an optional value follows a mandatory one.) **podman command** [*optional*] *value1* | *value2* [*optional*] @@ -33,23 +33,23 @@ podman\-command - short description Example sentence: The command **podman command** is an example command. Commands or files that are quoted from other podman manpages or podman repositories have to be linked to those. Non-podman commands are not to be linked.\ -Example sentence: You can use **[podman-run](podman-run.1.md)** or **[containers.conf(5)](https://github.com/containers/common/blob/master/docs/containers.conf.5.md)** for the problem. +Example sentence: Use **[podman-run](podman-run.1.md)** or **[containers.conf(5)](https://github.com/containers/common/blob/master/docs/containers.conf.5.md)** for the problem. -It should also be specified if the command can only be run as root. In addition, it should be described when a command, OPTION, or other content cannot be executed with the remote client or in combination with other commands, OPTIONS, or content. In this case, the following sentence is put at the end of a command, OPTION, or content: *IMPORTANT: This option/command/other is not available with the command/OPTION/content/remote Podman client*. For a command, this should be done in the DESCRIPTION section. For the OPTIONS, it should be done in the DESCRIPTION of the specified OPTION. Do not use pronouns in the man pages, especially the word `you`. +It should also be specified if the command can only be run as root. In addition, it should be described when a command, OPTION, or other content cannot be executed with the remote client or in combination with other commands, OPTIONS, or content. In this case, the following sentence is put at the end of a command, OPTION, or content: *IMPORTANT: This OPTION/command/other is not available with the command/OPTION/content/remote Podman client*. For a command, this should be done in the DESCRIPTION section. For the OPTIONS, it should be done in the DESCRIPTION of the specified OPTION. Do not use pronouns in the man pages, especially the word `you`. ## OPTIONS All flags are referred to as OPTIONS. The term flags should not be used. All OPTIONS are listed in this section. OPTIONS that appear in descriptions of other OPTIONS and sections retain their appearance, for example: **--exit**. OPTIONS that are quoted from other podman manpages or podman repositories have to be linked to those.\ -Example sentence: You can use **[podman-generate-systemd --new](podman-generate-systemd.1.md#--new)** for the problem. +Example sentence: Use **[podman-generate-systemd --new](podman-generate-systemd.1.md#--new)** for the problem. Each OPTION should be explained to the fullest extent below the OPTION itself. Each OPTION is behind an H4-header (`####`). If the OPTION has a default argument, it has to be explained in the description of the OPTION. If the OPTION is also not available with the remote client, the sentence about the default argument should the second to last sentence. #### **--version**, **-v** -OPTIONS can be put after the command in two different ways. Eather the long version with **--option** or as the short version **-o**. If there are two ways to write an OPTION they are separated by a comma. If there are two versions of one command the long version is always shown in front.\ -Example: The default is **false**. *IMPORTANT: This option is not available with the remote Podman client*. +OPTIONS can be put after the command in two different ways. Either the long version with **--option** or as the short version **-o**. If there are two ways to write an OPTION they are separated by a comma. If there are two versions of one command the long version is always shown in front.\ +Example: The default is **false**. *IMPORTANT: This OPTION is not available with the remote Podman client*. #### **--exit** @@ -57,7 +57,7 @@ An example of an OPTION that has only one possible structure. Thus, it cannot be #### **--answer**=, **-a**=**active** | *disable* -The "answer" option above is an example of an OPTION that accepts two possible arguments as inputs. If there is a default argument that is selected when the OPTION is not used in the command, it is shown in **bold**. If the OPTION is used it must include an argument afterwards. It must always be ensured that the standard argument is in the first position after the OPTION. In this example, there are two different ways to execute the command. Both possible OPTIONS have to be shown with the arguments following them. The default value is shown as **active**. +The "answer" OPTION above is an example of an OPTION that accepts two possible arguments as inputs. If there is a default argument that is selected when the OPTION is not used in the command, it is shown in **bold**. If the OPTION is used it must include an argument afterwards. It must always be ensured that the standard argument is in the first position after the OPTION. In this example, there are two different ways to execute the command. Both possible OPTIONS have to be shown with the arguments following them. The default value is shown as **active**. #### **--status**=**good** | *better* | *best* @@ -65,7 +65,7 @@ This is an example of three arguments following an OPTION. If the number of argu #### **--test**=**test** -OPTIONS that are followed by an equal sign include an argument after the equal sign in **bold**. If there is a default argument, that is used if the OPTION is not specified in the command, the argument after the eqaul sign is displayed in **bold**. All arguments must be listed and explained in the text below the OPTION. +OPTIONS that are followed by an equal sign include an argument after the equal sign in **bold**. If there is a default argument, that is used if the OPTION is not specified in the command, the argument after the equal sign is displayed in **bold**. All arguments must be listed and explained in the text below the OPTION. | Argument | Description | | ------------------ | --------------------------------------------------------------------------- | @@ -102,6 +102,7 @@ All EXAMPLES are listed in this section. This section should be at the end of ea Description of the EXAMPLE ``` +### Example comment $ podman command $ podman command -o @@ -111,9 +112,16 @@ $ cat $HOME/Dockerfile | podman command --option Description of the EXAMPLE two ``` -$ podman command --redhat +# podman command --status=better +``` +## SEE ALSO +All commands, including commands with OPTIONS, and config-files mentioned in the manpage have to be listed here. Podman commands, including commands with OPTIONS, and config-files have to be linked. If a command is mentioned several times with different OPTIONS it just have to be linked once. All other commands, including commands with OPTIONS, and config-files just have to be mentioned. If a command is mentioned several times with different OPTIONS it just has to be linked once. -$ podman command --redhat better +Example: +**[podman(1)](podman.1.md)**, **[podman-run(1)](podman-run.1.md)**, **[podman-create(1)](podman-create.1.md)** -$ podman command --redhat=better -``` +## HISTORY +Normally, the dates of changes, the content of the changes and the person who provided them is listed here. Most manpages don't keep this record. + +Example:\ +December 2021, Originally compiled by Alexander Richter <example@redhat.com> diff --git a/docs/source/markdown/podman-attach.1.md b/docs/source/markdown/podman-attach.1.md index 092772916..0a5948b4e 100644 --- a/docs/source/markdown/podman-attach.1.md +++ b/docs/source/markdown/podman-attach.1.md @@ -10,7 +10,7 @@ podman\-attach - Attach to a running container ## DESCRIPTION **podman attach** attaches to a running *container* using the *container's name* or *ID*, to either view its ongoing output or to control it interactively.\ -The *container* can detached from (and leave it running) using a configurable key sequence. The default sequence is `ctrl-p,ctrl-q`. Configure the keys sequence using the **--detach-keys** option, or specifying it in the `containers.conf` file: see **[containers.conf(5)](https://github.com/containers/common/blob/master/docs/containers.conf.5.md)** for more information. +The *container* can detached from (and leave it running) using a configurable key sequence. The default sequence is `ctrl-p,ctrl-q`. Configure the keys sequence using the **--detach-keys** OPTION, or specifying it in the `containers.conf` file: see **[containers.conf(5)](https://github.com/containers/common/blob/master/docs/containers.conf.5.md)** for more information. ## OPTIONS #### **--detach-keys**=**sequence** @@ -18,9 +18,8 @@ The *container* can detached from (and leave it running) using a configurable ke Specify the key **sequence** for detaching a *container*. Format is a single character `[a-Z]` or one or more `ctrl-<value>` characters where `<value>` is one of: `a-z`, `@`, `^`, `[`, `,` or `_`. Specifying "" will disable this feature. The default is `ctrl-p,ctrl-q`. #### **--latest**, **-l** - -Instead of providing the *container name* or *ID*, use the last created *container*. If you use methods other than Podman to run containers such as CRI-O, the last started *container* could be from either of those methods. The default is **false**.\ -*IMPORTANT: This option is not available with the remote Podman client* +Instead of providing the *container name* or *ID*, use the last created *container*. If other methods are used than Podman to run containers such as `CRI-O`, the last started *container* could be from either of those methods. The default is **false**.\ +*IMPORTANT: This OPTION is not available with the remote Podman client.* #### **--no-stdin** diff --git a/docs/source/markdown/podman-commit.1.md b/docs/source/markdown/podman-commit.1.md index 92574cdc5..bb7d3ce70 100644 --- a/docs/source/markdown/podman-commit.1.md +++ b/docs/source/markdown/podman-commit.1.md @@ -23,7 +23,17 @@ Set the author for the committed image. #### **--change**, **-c**=*instruction* Apply the following possible instructions to the created image: -**CMD** | **ENTRYPOINT** | **ENV** | **EXPOSE** | **LABEL** | **ONBUILD** | **STOPSIGNAL** | **USER** | **VOLUME** | **WORKDIR** + +- *CMD* +- *ENTRYPOINT* +- *ENV* +- *EXPOSE* +- *LABEL* +- *ONBUILD* +- *STOPSIGNAL* +- *USER* +- *VOLUME* +- *WORKDIR* Can be set multiple times. diff --git a/docs/source/markdown/podman-completion.1.md b/docs/source/markdown/podman-completion.1.md index 4ebe4e2e1..f8589ce68 100644 --- a/docs/source/markdown/podman-completion.1.md +++ b/docs/source/markdown/podman-completion.1.md @@ -4,60 +4,60 @@ podman\-completion - Generate shell completion scripts ## SYNOPSIS -**podman completion** [*options*] *bash*|*zsh*|*fish*|*powershell* +**podman completion** [*options*] *bash* | *zsh* | *fish* | *powershell* ## DESCRIPTION -The completion command generates shell completion scripts for a variety of shells. Supported shells are **bash**, **zsh**, **fish** and **powershell**. +**podman completion** generates shell completion scripts for a variety of shells. Supported shells are *bash*, *zsh*, *fish* and *powershell*. -These script are used by the shell to provide suggestions and complete commands when you are typing the command and press [TAB]. +These script are used by the shell to provide suggestions and complete commands when the command is typed and `[TAB]` is pressed. Usually these scripts are automatically installed via the package manager. ## OPTIONS -#### **--file**, **-f** +#### **--file**, **-f**=*file* -Write the generated output to file. +Write the generated output to a file. #### **--no-desc** -Do not provide description in the completions. +Do not provide description in the completions. The default is **false**. ## Installation ### BASH -Make sure you have `bash-completion` installed on the system. +`bash-completion` has to be installed on the system. -To load the completion script into the current session run: -`source <(podman completion bash)` +To load the completion script into the current session run:\ +**source <(podman completion bash)**. -To make it available for all bash sessions run: -`podman completion bash -f /etc/bash_completion.d/podman` +To make it available for all bash sessions run:\ +**podman completion -f /etc/bash_completion.d/podman bash**. ### ZSH -If shell completion is not already enabled in the environment you will need to enable it. You can execute the following once: -`echo "autoload -U compinit; compinit" >> ~/.zshrc` +Shell completion needs to be already enabled in the environment. The following can be executed:\ +**echo "autoload -U compinit; compinit" >> ~/.zshrc** -To make it available for all zsh sessions run: -`podman completion zsh -f "${fpath[1]}/_podman"` +To make it available for all zsh sessions run:\ +**podman completion -f "${fpath[1]}/_podman zsh"** -Once you reload the shell the auto-completion should be working. +Once the shell is reloaded the auto-completion should be working. ### FISH To load the completion script into the current session run: -`podman completion fish | source` +**podman completion fish | source** To make it available for all fish sessions run: -`podman completion fish -f ~/.config/fish/completions/podman.fish` +**podman completion -f ~/.config/fish/completions/podman.fish fish** ### POWERSHELL To load the completion script into the current session run: -`podman.exe completion powershell | Out-String | Invoke-Expression` +**podman.exe completion powershell | Out-String | Invoke-Expression** To make it available in all powershell sessions that a user has, write the completion output to a file and source that to the user's powershell profile. -More information about profiles is available with `Get-Help about_Profiles`. +More information about profiles is available with **Get-Help about_Profiles**. ## SEE ALSO -[podman(1)](podman.1.md) +**[podman(1)](podman.1.md)**, zsh(1), fish(1), powershell(1) diff --git a/docs/source/markdown/podman-container-checkpoint.1.md b/docs/source/markdown/podman-container-checkpoint.1.md index 271a44c9d..a86389f59 100644 --- a/docs/source/markdown/podman-container-checkpoint.1.md +++ b/docs/source/markdown/podman-container-checkpoint.1.md @@ -4,96 +4,102 @@ podman\-container\-checkpoint - Checkpoints one or more running containers ## SYNOPSIS -**podman container checkpoint** [*options*] *container* ... +**podman container checkpoint** [*options*] *container* [*container* ...] ## DESCRIPTION -Checkpoints all the processes in one or more containers. You may use container IDs or names as input. +**podman container checkpoint** checkpoints all the processes in one or more *containers*. A *container* can be restored from a checkpoint with **[podman-container-restore](podman-container-restore.1.md)**. The *container IDs* or *names* are used as input. ## OPTIONS #### **--all**, **-a** -Checkpoint all running containers. +Checkpoint all running *containers*. The default is **false**. -#### **--compress**, **-c** +#### **--compress**, **-c**=**zstd** | *none* | *gzip* Specify the compression algorithm used for the checkpoint archive created -with the **--export, -e** option. Possible algorithms are *gzip*, *none* -and *zstd*. If no compression algorithm is specified Podman will use -*zstd*. +with the **--export, -e** OPTION. Possible algorithms are **zstd**, *none* +and *gzip*. The default is **zstd**. One possible reason to use *none* is to enable faster creation of checkpoint archives. Not compressing the checkpoint archive can result in faster checkpoint archive creation. -``` -# podman container checkpoint -l --compress=none --export=dump.tar -# podman container checkpoint -l --compress=gzip --export=dump.tar.gz -``` - -#### **--export**, **-e** +#### **--export**, **-e**=*archive* Export the checkpoint to a tar.gz file. The exported checkpoint can be used -to import the container on another system and thus enabling container live -migration. This checkpoint archive also includes all changes to the container's -root file-system, if not explicitly disabled using **--ignore-rootfs** +to import the *container* on another system and thus enabling container live +migration. This checkpoint archive also includes all changes to the *container's* +root file-system, if not explicitly disabled using **--ignore-rootfs**. #### **--ignore-rootfs** This only works in combination with **--export, -e**. If a checkpoint is exported to a tar.gz file it is possible with the help of **--ignore-rootfs** to explicitly disable including changes to the root file-system into -the checkpoint archive file. +the checkpoint archive file. The default is **false**. #### **--ignore-volumes** -This option must be used in combination with the **--export, -e** option. -When this option is specified, the content of volumes associated with -the container will not be included into the checkpoint tar.gz file. +This OPTION must be used in combination with the **--export, -e** OPTION. +When this OPTION is specified, the content of volumes associated with +the *container* will not be included into the checkpoint tar.gz file. The default is **false**. #### **--keep**, **-k** -Keep all temporary log and statistics files created by CRIU during checkpointing. These files -are not deleted if checkpointing fails for further debugging. If checkpointing succeeds these -files are theoretically not needed, but if these files are needed Podman can keep the files -for further analysis. +Keep all temporary log and statistics files created by CRIU during checkpointing. These files are not deleted if checkpointing fails for further debugging. If checkpointing succeeds these files are theoretically not needed, but if these files are needed Podman can keep the files for further analysis. The default is **false**. #### **--latest**, **-l** -Instead of providing the container name or ID, checkpoint the last created container. (This option is not available with the remote Podman client) +Instead of providing the *container ID* or *name*, use the last created *container*. If you use methods other than Podman to run *containers* such as `CRI-O`, the last started *container* could be from either of those methods. The default is **false**.\ +*IMPORTANT: This OPTION is not available with the remote Podman client.* #### **--leave-running**, **-R** -Leave the container running after checkpointing instead of stopping it. +Leave the *container* running after checkpointing instead of stopping it. The default is **false**. #### **--pre-checkpoint**, **-P** -Dump the container's memory information only, leaving the container running. Later -operations will supersede prior dumps. It only works on runc 1.0-rc3 or higher. +Dump the *container's* memory information only, leaving the *container* running. Later +operations will supersede prior dumps. It only works on `runc 1.0-rc3` or `higher`. The default is **false**. #### **--tcp-established** -Checkpoint a container with established TCP connections. If the checkpoint -image contains established TCP connections, this options is required during -restore. Defaults to not checkpointing containers with established TCP -connections. +Checkpoint a *container* with established TCP connections. If the checkpoint +image contains established TCP connections, this OPTION is required during +restore. Defaults to not checkpointing *containers* with established TCP +connections. The default is **false**. #### **--with-previous** -Check out the container with previous criu image files in pre-dump. It only works -without **--pre-checkpoint** or **-P**. It only works on runc 1.0-rc3 or higher. +Check out the *container* with previous criu image files in pre-dump. It only works on `runc 1.0-rc3` or `higher`. The default is **false**.\ +*IMPORTANT: This OPTION is not available with **--pre-checkpoint***. + -## EXAMPLE +## EXAMPLES -podman container checkpoint mywebserver +Make a checkpoint for the container "mywebserver". +``` +# podman container checkpoint mywebserver +``` -podman container checkpoint 860a4b23 +Dumps the container's memory information of the latest container into an archive. +``` +# podman container checkpoint -P -e pre-checkpoint.tar.gz -l +``` -podman container checkpoint -P -e pre-checkpoint.tar.gz -l +Keep the container's memory information from an older dump and add the new container's memory information. +``` +# podman container checkpoint --with-previous -e checkpoint.tar.gz -l +``` -podman container checkpoint --with-previous -e checkpoint.tar.gz -l +Dump the container's memory information of the latest container into an archive with the specified compress method. +``` +# podman container checkpoint -l --compress=none --export=dump.tar +# podman container checkpoint -l --compress=gzip --export=dump.tar.gz +``` ## SEE ALSO -podman(1), podman-container-restore(1) +**[podman(1)](podman.1.md)**, **[podman-container-restore(1)](podman-container-restore.1.md)** ## HISTORY September 2018, Originally compiled by Adrian Reber <areber@redhat.com> diff --git a/docs/source/markdown/podman-container-cleanup.1.md b/docs/source/markdown/podman-container-cleanup.1.md index 19d0b7818..f33b68a1c 100644 --- a/docs/source/markdown/podman-container-cleanup.1.md +++ b/docs/source/markdown/podman-container-cleanup.1.md @@ -4,51 +4,51 @@ podman\-container\-cleanup - Cleanup the container's network and mountpoints ## SYNOPSIS -**podman container cleanup** [*options*] *container* +**podman container cleanup** [*options*] *container* [*container* ...] ## DESCRIPTION -**podman container cleanup** cleans up exited containers by removing all mountpoints and network configuration from the host. The container name or ID can be used. The cleanup command does not remove the containers. Running containers will not be cleaned up. -Sometimes container's mount points and network stacks can remain if the podman command was killed or the container ran in daemon mode. This command is automatically executed when you run containers in daemon mode by the conmon process when the container exits. +**podman container cleanup** cleans up exited *containers* by removing all mountpoints and network configuration from the host. The *container name* or *ID* can be used. The cleanup command does not remove the *containers*. Running *containers* will not be cleaned up.\ +Sometimes container mount points and network stacks can remain if the podman command was killed or the *container* ran in daemon mode. This command is automatically executed when you run *containers* in daemon mode by the `conmon process` when the *container* exits. ## OPTIONS #### **--all**, **-a** -Cleanup all containers. +Cleanup all *containers*. The default is **false**. -#### **--exec**=_session_ +#### **--exec**=*session* -Clean up an exec session for a single container. -Can only be specified if a single container is being cleaned up (conflicts with **--all** as such). -If **--rm** is not specified, temporary files for the exec session will be cleaned up; if it is, the exec session will be removed from the container. -Conflicts with **--rmi** as the container is not being cleaned up so the image cannot be removed. +Clean up an exec session for a single *container*. +Can only be specified if a single *container* is being cleaned up (conflicts with **--all** as such). If **--rm** is not specified, temporary files for the exec session will be cleaned up; if it is, the exec session will be removed from the *container*.\ +*IMPORTANT: Conflicts with **--rmi** as the container is not being cleaned up so the image cannot be removed.* #### **--latest**, **-l** -Instead of providing the container name or ID, use the last created container. If you use methods other than Podman -to run containers such as CRI-O, the last started container could be from either of those methods. (This option is not available with the remote Podman client) + +Instead of providing the *container ID* or *name*, use the last created *container*. If you use methods other than Podman to run *containers* such as `CRI-O`, the last started *container* could be from either of those methods. The default is **false**.\ +*IMPORTANT: This OPTION is not available with the remote Podman client.* #### **--rm** -After cleanup, remove the container entirely. +After cleanup, remove the *container* entirely. The default is **false**. #### **--rmi** -After cleanup, remove the image entirely. - -## EXAMPLE - -`podman container cleanup mywebserver` - -`podman container cleanup mywebserver myflaskserver 860a4b23` +After cleanup, remove the image entirely. The default is **false**. -`podman container cleanup 860a4b23` +## EXAMPLES -`podman container cleanup -a` +Cleanup the container "mywebserver". +``` +$ podman container cleanup mywebserver +``` -`podman container cleanup --latest` +Cleanup the containers with the names "mywebserver", "myflaskserver", "860a4b23". +``` +$ podman container cleanup mywebserver myflaskserver 860a4b23 +``` ## SEE ALSO -**podman**(1), **podman-container**(1), **conmon**(8). +**[podman(1)](podman.1.md)**, **[podman-container(1)](podman-container.1.md)**, conmon(8) ## HISTORY Jun 2018, Originally compiled by Dan Walsh <dwalsh@redhat.com> diff --git a/docs/source/markdown/podman-container-exists.1.md b/docs/source/markdown/podman-container-exists.1.md index 381d968ab..e42489d63 100644 --- a/docs/source/markdown/podman-container-exists.1.md +++ b/docs/source/markdown/podman-container-exists.1.md @@ -1,42 +1,40 @@ % podman-container-exists(1) ## NAME -podman-container-exists - Check if a container exists in local storage +podman\-container\-exists - Check if a container exists in local storage ## SYNOPSIS **podman container exists** [*options*] *container* ## DESCRIPTION -**podman container exists** checks if a container exists in local storage. The **ID** or **Name** -of the container may be used as input. Podman will return an exit code -of `0` when the container is found. A `1` will be returned otherwise. An exit code of `125` indicates there -was an issue accessing the local storage. +**podman container exists** checks if a container exists in local storage. The *container ID* or *name* is used as input. Podman will return an exit code +of `0` when the container is found. A `1` will be returned otherwise. An exit code of `125` indicates there was an issue accessing the local storage. ## OPTIONS -#### **--external**=*true|false* -Check for external containers as well as Podman containers. These external containers are generally created via other container technology such as Buildah or CRI-O. +#### **--external** +Check for external *containers* as well as Podman *containers*. These external *containers* are generally created via other container technology such as `Buildah` or `CRI-O`. The default is **false**. **-h**, **--help** -Print usage statement +Prints usage statement. ## EXAMPLES -Check if an container called `webclient` exists in local storage (the container does actually exist). +Check if an container called "webclient" exists in local storage. Here, the container does exist. ``` $ podman container exists webclient $ echo $? 0 ``` -Check if an container called `webbackend` exists in local storage (the container does not actually exist). +Check if an container called "webbackend" exists in local storage. Here, the container does not exist. ``` $ podman container exists webbackend $ echo $? 1 ``` -Check if an container called `ubi8-working-container` created via Buildah exists in local storage (the container does not actually exist). +Check if an container called "ubi8-working-container" created via Buildah exists in local storage. Here, the container does not exist. ``` $ podman container exists --external ubi8-working-container $ echo $? @@ -44,7 +42,7 @@ $ echo $? ``` ## SEE ALSO -podman(1) +**[podman(1)](podman.1.md)** ## HISTORY -November 2018, Originally compiled by Brent Baude (bbaude at redhat dot com) +November 2018, Originally compiled by Brent Baude <bbaude@redhat.com> diff --git a/docs/source/markdown/podman-manifest-rm.1.md b/docs/source/markdown/podman-manifest-rm.1.md index 396dd49c7..a65f32936 100644 --- a/docs/source/markdown/podman-manifest-rm.1.md +++ b/docs/source/markdown/podman-manifest-rm.1.md @@ -11,7 +11,7 @@ Removes one or more locally stored manifest lists. ## EXAMPLE -podman manifest rm <list> +podman manifest rm `<list>` podman manifest rm listid1 listid2 diff --git a/docs/source/markdown/podman-network-create.1.md b/docs/source/markdown/podman-network-create.1.md index 3d5d98055..d110c4ceb 100644 --- a/docs/source/markdown/podman-network-create.1.md +++ b/docs/source/markdown/podman-network-create.1.md @@ -9,7 +9,7 @@ podman\-network-create - Create a Podman CNI network ## DESCRIPTION Create a CNI-network configuration for use with Podman. By default, Podman creates a bridge connection. A *Macvlan* connection can be created with the *-d macvlan* option. A parent device for macvlan can -be designated with the *-o parent=\<device>* option. In the case of *Macvlan* connections, the +be designated with the *-o parent=`<device>`* option. In the case of *Macvlan* connections, the CNI *dhcp* plugin needs to be activated or the container image must have a DHCP client to interact with the host network's DHCP server. diff --git a/docs/source/markdown/podman-pod-create.1.md b/docs/source/markdown/podman-pod-create.1.md index 37eb098d1..4b890a7af 100644 --- a/docs/source/markdown/podman-pod-create.1.md +++ b/docs/source/markdown/podman-pod-create.1.md @@ -10,8 +10,8 @@ podman\-pod\-create - Create a new pod Creates an empty pod, or unit of multiple containers, and prepares it to have containers added to it. The pod id is printed to STDOUT. You can then use -**podman create --pod \<pod_id|pod_name\> ...** to add containers to the pod, and -**podman pod start \<pod_id|pod_name\>** to start the pod. +**podman create --pod `<pod_id|pod_name>` ...** to add containers to the pod, and +**podman pod start `<pod_id|pod_name>`** to start the pod. ## OPTIONS diff --git a/docs/source/markdown/podman.1.md b/docs/source/markdown/podman.1.md index 734d00971..2510eaa81 100644 --- a/docs/source/markdown/podman.1.md +++ b/docs/source/markdown/podman.1.md @@ -336,9 +336,11 @@ Images are pulled under `XDG_DATA_HOME` when specified, otherwise in the home di Currently the slirp4netns package is required to be installed to create a network device, otherwise rootless containers need to run in the network namespace of the host. +In certain environments like HPC (High Performance Computing), users cannot take advantage of the additional UIDs and GIDs from the /etc/subuid and /etc/subgid systems. However, in this environment, rootless Podman can operate with a single UID. To make this work, set the `ignore_chown_errors` option in the /etc/containers/storage.conf or in ~/.config/containers/storage.conf files. This option tells Podman when pulling an image to ignore chown errors when attempting to change a file in a container image to match the non-root UID in the image. This means all files get saved as the user's UID. Note this could cause issues when running the container. + ### **NOTE:** Unsupported file systems in rootless mode -The Overlay file system (OverlayFS) is not supported in rootless mode. The fuse-overlayfs package is a tool that provides the functionality of OverlayFS in user namespace that allows mounting file systems in rootless environments. It is recommended to install the fuse-overlayfs package. In rootless mode Podman will automatically use the fuse-overlafs program as the mount_program if installed, as long as the $HOME/.config/containers/storage.conf file was not previously created. If storage.conf exists in the homedir, add `mount_program = "/usr/bin/fuse-overlayfs"` under `[storage.options.overlay]` to enable this feature. +The Overlay file system (OverlayFS) is not supported with kernels prior to 5.12.9 in rootless mode. The fuse-overlayfs package is a tool that provides the functionality of OverlayFS in user namespace that allows mounting file systems in rootless environments. It is recommended to install the fuse-overlayfs package. In rootless mode, Podman will automatically use the fuse-overlayfs program as the mount_program if installed, as long as the $HOME/.config/containers/storage.conf file was not previously created. If storage.conf exists in the homedir, add `mount_program = "/usr/bin/fuse-overlayfs"` under `[storage.options.overlay]` to enable this feature. The Network File System (NFS) and other distributed file systems (for example: Lustre, Spectrum Scale, the General Parallel File System (GPFS)) are not supported when running in rootless mode as these file systems do not understand user namespace. However, rootless Podman can make use of an NFS Homedir by modifying the `$HOME/.config/containers/storage.conf` to have the `graphroot` option point to a directory stored on local (Non NFS) storage. @@ -17,7 +17,7 @@ require ( github.com/containers/image/v5 v5.12.0 github.com/containers/ocicrypt v1.1.1 github.com/containers/psgo v1.5.2 - github.com/containers/storage v1.32.1 + github.com/containers/storage v1.32.2 github.com/coreos/go-systemd/v22 v22.3.2 github.com/coreos/stream-metadata-go v0.0.0-20210225230131-70edb9eb47b3 github.com/cri-o/ocicni v0.2.1-0.20210301205850-541cf7c703cf @@ -239,8 +239,8 @@ github.com/containers/storage v1.23.5/go.mod h1:ha26Q6ngehFNhf3AWoXldvAvwI4jFe3E github.com/containers/storage v1.30.1/go.mod h1:NDJkiwxnSHD1Is+4DGcyR3SIEYSDOa0xnAW+uGQFx9E= github.com/containers/storage v1.31.1/go.mod h1:IFEf+yRTS0pvCGQt2tBv1Kzz2XUSPvED6uFBmWG7V/E= github.com/containers/storage v1.32.0/go.mod h1:J3q772EVbN9vgqoN/dkvInKnp4xK9ZXm7wHNfuiIDgE= -github.com/containers/storage v1.32.1 h1:JgvHY5dokiff+Ee4TdvPYO++Oq2BAave5DmyPetH2iU= -github.com/containers/storage v1.32.1/go.mod h1:do6oIF71kfkVS3CPUZr+6He94fIaj6pzF8ywevPuuOw= +github.com/containers/storage v1.32.2 h1:V1oKAKmH5e6OTgP7Uf8+T+ntVVYk2MNFnDDkPBI9kxU= +github.com/containers/storage v1.32.2/go.mod h1:YIBxxjfXZTi04Ah49sh1uSGfmT1V89+I5i3deRobzQo= github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= @@ -536,8 +536,8 @@ github.com/klauspost/compress v1.11.0/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYs github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.12.2/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= -github.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU= -github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= +github.com/klauspost/compress v1.13.0 h1:2T7tUoQrQT+fQWdaY5rjWztFGAFwbGD04iPJg90ZiOs= +github.com/klauspost/compress v1.13.0/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -576,8 +576,9 @@ github.com/mattn/go-runewidth v0.0.12 h1:Y41i/hVW3Pgwr8gV+J23B9YEY0zxjptBuCWEaxm github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk= github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o= github.com/mattn/go-shellwords v1.0.10/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y= -github.com/mattn/go-shellwords v1.0.11 h1:vCoR9VPpsk/TZFW2JwK5I9S0xdrtUq2bph6/YjEPnaw= github.com/mattn/go-shellwords v1.0.11/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y= +github.com/mattn/go-shellwords v1.0.12 h1:M2zGm7EW6UQJvDeQxo4T51eKPurbeFbe8WtebGE2xrk= +github.com/mattn/go-shellwords v1.0.12/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI= github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= diff --git a/libpod/container.go b/libpod/container.go index c6f0cd618..4b9bea5fc 100644 --- a/libpod/container.go +++ b/libpod/container.go @@ -957,6 +957,12 @@ func (c *Container) cGroupPath() (string, error) { // is the libpod-specific one we're looking for. // // See #8397 on the need for the longest-path look up. + // + // And another workaround for containers running systemd as the payload. + // containers running systemd moves themselves into a child subgroup of + // the named systemd cgroup hierarchy. Ignore any named cgroups during + // the lookup. + // See #10602 for more details. procPath := fmt.Sprintf("/proc/%d/cgroup", c.state.PID) lines, err := ioutil.ReadFile(procPath) if err != nil { @@ -972,6 +978,10 @@ func (c *Container) cGroupPath() (string, error) { logrus.Debugf("Error parsing cgroup: expected 3 fields but got %d: %s", len(fields), procPath) continue } + // Ignore named cgroups like name=systemd. + if bytes.Contains(fields[1], []byte("=")) { + continue + } path := string(fields[2]) if len(path) > len(cgroupPath) { cgroupPath = path diff --git a/libpod/container_exec.go b/libpod/container_exec.go index c359f1e5d..737bf74ad 100644 --- a/libpod/container_exec.go +++ b/libpod/container_exec.go @@ -1,6 +1,7 @@ package libpod import ( + "context" "io/ioutil" "net/http" "os" @@ -539,18 +540,7 @@ func (c *Container) ExecStop(sessionID string, timeout *uint) error { var cleanupErr error // Retrieve exit code and update status - exitCode, err := c.readExecExitCode(session.ID()) - if err != nil { - cleanupErr = err - } - session.ExitCode = exitCode - session.PID = 0 - session.State = define.ExecStateStopped - - if err := c.save(); err != nil { - if cleanupErr != nil { - logrus.Errorf("Error stopping container %s exec session %s: %v", c.ID(), session.ID(), cleanupErr) - } + if err := retrieveAndWriteExecExitCode(c, session.ID()); err != nil { cleanupErr = err } @@ -592,15 +582,7 @@ func (c *Container) ExecCleanup(sessionID string) error { return errors.Wrapf(define.ErrExecSessionStateInvalid, "cannot clean up container %s exec session %s as it is running", c.ID(), session.ID()) } - exitCode, err := c.readExecExitCode(session.ID()) - if err != nil { - return err - } - session.ExitCode = exitCode - session.PID = 0 - session.State = define.ExecStateStopped - - if err := c.save(); err != nil { + if err := retrieveAndWriteExecExitCode(c, session.ID()); err != nil { return err } } @@ -637,9 +619,9 @@ func (c *Container) ExecRemove(sessionID string, force bool) error { return err } if !running { - session.State = define.ExecStateStopped - // TODO: should we retrieve exit code here? - // TODO: Might be worth saving state here. + if err := retrieveAndWriteExecExitCode(c, session.ID()); err != nil { + return err + } } } @@ -653,6 +635,10 @@ func (c *Container) ExecRemove(sessionID string, force bool) error { return err } + if err := retrieveAndWriteExecExitCode(c, session.ID()); err != nil { + return err + } + if err := c.cleanupExecBundle(session.ID()); err != nil { return err } @@ -757,10 +743,25 @@ func (c *Container) Exec(config *ExecConfig, streams *define.AttachStreams, resi session, err := c.ExecSession(sessionID) if err != nil { + if errors.Cause(err) == define.ErrNoSuchExecSession { + // TODO: If a proper Context is ever plumbed in here, we + // should use it. + // As things stand, though, it's not worth it - this + // should always terminate quickly since it's not + // streaming. + diedEvent, err := c.runtime.GetExecDiedEvent(context.Background(), c.ID(), sessionID) + if err != nil { + return -1, errors.Wrapf(err, "error retrieving exec session %s exit code", sessionID) + } + return diedEvent.ContainerExitCode, nil + } return -1, err } exitCode := session.ExitCode if err := c.ExecRemove(sessionID, false); err != nil { + if errors.Cause(err) == define.ErrNoSuchExecSession { + return exitCode, nil + } return -1, err } @@ -927,6 +928,8 @@ func (c *Container) getActiveExecSessions() ([]string, error) { session.PID = 0 session.State = define.ExecStateStopped + c.newExecDiedEvent(session.ID(), exitCode) + needSave = true } if err := c.cleanupExecBundle(id); err != nil { @@ -1036,6 +1039,22 @@ func writeExecExitCode(c *Container, sessionID string, exitCode int) error { return errors.Wrapf(err, "error syncing container %s state to remove exec session %s", c.ID(), sessionID) } + return justWriteExecExitCode(c, sessionID, exitCode) +} + +func retrieveAndWriteExecExitCode(c *Container, sessionID string) error { + exitCode, err := c.readExecExitCode(sessionID) + if err != nil { + return err + } + + return justWriteExecExitCode(c, sessionID, exitCode) +} + +func justWriteExecExitCode(c *Container, sessionID string, exitCode int) error { + // Write an event first + c.newExecDiedEvent(sessionID, exitCode) + session, ok := c.state.ExecSessions[sessionID] if !ok { // Exec session already removed. diff --git a/libpod/container_internal.go b/libpod/container_internal.go index f77825efd..3e4eea003 100644 --- a/libpod/container_internal.go +++ b/libpod/container_internal.go @@ -42,6 +42,7 @@ const ( // name of the directory holding the artifacts artifactsDir = "artifacts" execDirPermission = 0755 + preCheckpointDir = "pre-checkpoint" ) // rootFsSize gets the size of the container's root filesystem @@ -141,7 +142,7 @@ func (c *Container) CheckpointPath() string { // PreCheckpointPath returns the path to the directory containing the pre-checkpoint-images func (c *Container) PreCheckPointPath() string { - return filepath.Join(c.bundlePath(), "pre-checkpoint") + return filepath.Join(c.bundlePath(), preCheckpointDir) } // AttachSocketPath retrieves the path of the container's attach socket @@ -427,7 +428,7 @@ func (c *Container) setupStorage(ctx context.Context) error { }, LabelOpts: c.config.LabelOpts, } - if c.restoreFromCheckpoint { + if c.restoreFromCheckpoint && !c.config.Privileged { // If restoring from a checkpoint, the root file-system // needs to be mounted with the same SELinux labels as // it was mounted previously. diff --git a/libpod/container_internal_linux.go b/libpod/container_internal_linux.go index 94bf7855b..ddfccb999 100644 --- a/libpod/container_internal_linux.go +++ b/libpod/container_internal_linux.go @@ -909,14 +909,15 @@ func (c *Container) exportCheckpoint(options ContainerCheckpointOptions) error { includeFiles := []string{ "artifacts", "ctr.log", - metadata.CheckpointDirectory, metadata.ConfigDumpFile, metadata.SpecDumpFile, metadata.NetworkStatusFile, } if options.PreCheckPoint { - includeFiles[0] = "pre-checkpoint" + includeFiles = append(includeFiles, preCheckpointDir) + } else { + includeFiles = append(includeFiles, metadata.CheckpointDirectory) } // Get root file-system changes included in the checkpoint archive var addToTarFiles []string diff --git a/libpod/events.go b/libpod/events.go index 839229674..22c51aeec 100644 --- a/libpod/events.go +++ b/libpod/events.go @@ -46,7 +46,22 @@ func (c *Container) newContainerExitedEvent(exitCode int32) { e.Type = events.Container e.ContainerExitCode = int(exitCode) if err := c.runtime.eventer.Write(e); err != nil { - logrus.Errorf("unable to write pod event: %q", err) + logrus.Errorf("unable to write container exited event: %q", err) + } +} + +// newExecDiedEvent creates a new event for an exec session's death +func (c *Container) newExecDiedEvent(sessionID string, exitCode int) { + e := events.NewEvent(events.ExecDied) + e.ID = c.ID() + e.Name = c.Name() + e.Image = c.config.RootfsImageName + e.Type = events.Container + e.ContainerExitCode = exitCode + e.Attributes = make(map[string]string) + e.Attributes["execID"] = sessionID + if err := c.runtime.eventer.Write(e); err != nil { + logrus.Errorf("unable to write exec died event: %q", err) } } @@ -154,3 +169,25 @@ func (r *Runtime) GetLastContainerEvent(ctx context.Context, nameOrID string, co // return the last element in the slice return containerEvents[len(containerEvents)-1], nil } + +// GetExecDiedEvent takes a container name or ID, exec session ID, and returns +// that exec session's Died event (if it has already occurred). +func (r *Runtime) GetExecDiedEvent(ctx context.Context, nameOrID, execSessionID string) (*events.Event, error) { + filters := []string{ + fmt.Sprintf("container=%s", nameOrID), + "event=exec_died", + "type=container", + fmt.Sprintf("label=execID=%s", execSessionID), + } + + containerEvents, err := r.GetEvents(ctx, filters) + if err != nil { + return nil, err + } + // There *should* only be one event maximum. + // But... just in case... let's not blow up if there's more than one. + if len(containerEvents) < 1 { + return nil, errors.Wrapf(events.ErrEventNotFound, "exec died event for session %s (container %s) not found", execSessionID, nameOrID) + } + return containerEvents[len(containerEvents)-1], nil +} diff --git a/libpod/events/config.go b/libpod/events/config.go index 085fa9d52..d88d7b6e3 100644 --- a/libpod/events/config.go +++ b/libpod/events/config.go @@ -127,6 +127,8 @@ const ( Create Status = "create" // Exec ... Exec Status = "exec" + // ExecDied indicates that an exec session in a container died. + ExecDied Status = "exec_died" // Exited indicates that a container's process died Exited Status = "died" // Export ... diff --git a/libpod/events/events.go b/libpod/events/events.go index 01ea6a386..e03215eff 100644 --- a/libpod/events/events.go +++ b/libpod/events/events.go @@ -149,6 +149,8 @@ func StringToStatus(name string) (Status, error) { return Create, nil case Exec.String(): return Exec, nil + case ExecDied.String(): + return ExecDied, nil case Exited.String(): return Exited, nil case Export.String(): diff --git a/libpod/networking_linux.go b/libpod/networking_linux.go index c928e02a6..5446841f6 100644 --- a/libpod/networking_linux.go +++ b/libpod/networking_linux.go @@ -1090,7 +1090,7 @@ func (c *Container) NetworkDisconnect(nameOrID, netName string, force bool) erro } c.newNetworkEvent(events.NetworkDisconnect, netName) - if c.state.State != define.ContainerStateRunning { + if !c.ensureState(define.ContainerStateRunning, define.ContainerStateCreated) { return nil } @@ -1145,7 +1145,7 @@ func (c *Container) NetworkConnect(nameOrID, netName string, aliases []string) e return err } c.newNetworkEvent(events.NetworkConnect, netName) - if c.state.State != define.ContainerStateRunning { + if !c.ensureState(define.ContainerStateRunning, define.ContainerStateCreated) { return nil } if c.state.NetNS == nil { diff --git a/libpod/oci_conmon_linux.go b/libpod/oci_conmon_linux.go index 3da49b85f..2914bd1a1 100644 --- a/libpod/oci_conmon_linux.go +++ b/libpod/oci_conmon_linux.go @@ -787,7 +787,11 @@ func (r *ConmonOCIRuntime) CheckpointContainer(ctr *Container, options Container args = append(args, "--pre-dump") } if !options.PreCheckPoint && options.WithPrevious { - args = append(args, "--parent-path", ctr.PreCheckPointPath()) + args = append( + args, + "--parent-path", + filepath.Join("..", preCheckpointDir), + ) } runtimeDir, err := util.GetRuntimeDir() if err != nil { diff --git a/libpod/options.go b/libpod/options.go index f942d264b..d3be46ad8 100644 --- a/libpod/options.go +++ b/libpod/options.go @@ -1641,6 +1641,19 @@ func WithVolumeGID(gid int) VolumeCreateOption { } } +// WithVolumeNoChown prevents the volume from being chowned to the process uid at first use. +func WithVolumeNoChown() VolumeCreateOption { + return func(volume *Volume) error { + if volume.valid { + return define.ErrVolumeFinalized + } + + volume.state.NeedsChown = false + + return nil + } +} + // withSetAnon sets a bool notifying libpod that this volume is anonymous and // should be removed when containers using it are removed and volumes are // specified for removal. diff --git a/libpod/volume_internal.go b/libpod/volume_internal.go index 694cdd149..19008a253 100644 --- a/libpod/volume_internal.go +++ b/libpod/volume_internal.go @@ -39,8 +39,23 @@ func (v *Volume) needsMount() bool { return true } - // Local driver with options needs mount - return len(v.config.Options) > 0 + // Commit 28138dafcc added the UID and GID options to this map + // However we should only mount when options other than uid and gid are set. + // see https://github.com/containers/podman/issues/10620 + index := 0 + if _, ok := v.config.Options["UID"]; ok { + index++ + } + if _, ok := v.config.Options["GID"]; ok { + index++ + } + // when uid or gid is set there is also the "o" option + // set so we have to ignore this one as well + if index > 0 { + index++ + } + // Local driver with options other than uid,gid needs mount + return len(v.config.Options) > index } // update() updates the volume state from the DB. diff --git a/nix/default-arm64.nix b/nix/default-arm64.nix index 3e374c1bd..d29f75520 100644 --- a/nix/default-arm64.nix +++ b/nix/default-arm64.nix @@ -25,6 +25,21 @@ let -i "$dev"/include/glib-2.0/gobject/gobjectnotifyqueue.c ''; }); + pcsclite = (static pkg.pcsclite).overrideAttrs (x: { + configureFlags = [ + "--enable-confdir=/etc" + "--enable-usbdropdir=/var/lib/pcsc/drivers" + "--disable-libsystemd" + ]; + buildInputs = [ pkgs.python3 pkgs.udev pkgs.dbus pkgs.systemd ]; + }); + systemd = (static pkg.systemd).overrideAttrs (x: { + outputs = [ "out" "dev" ]; + mesonFlags = x.mesonFlags ++ [ + "-Dglib=false" + "-Dstatic-libsystemd=true" + ]; + }); }; }; }); @@ -47,8 +62,8 @@ let doCheck = false; enableParallelBuilding = true; outputs = [ "out" ]; - nativeBuildInputs = [ bash gitMinimal go-md2man installShellFiles makeWrapper pkg-config which ]; - buildInputs = [ glibc glibc.static gpgme libassuan libgpgerror libseccomp libapparmor libselinux ]; + nativeBuildInputs = [ bash gitMinimal go-md2man pkg-config which ]; + buildInputs = [ glibc glibc.static glib gpgme libassuan libgpgerror libseccomp libapparmor libselinux ]; prePatch = '' export CFLAGS='-static -pthread' export LDFLAGS='-s -w -static-libgcc -static' diff --git a/nix/default.nix b/nix/default.nix index aae520c9e..f6194e3d9 100644 --- a/nix/default.nix +++ b/nix/default.nix @@ -23,6 +23,21 @@ let -i "$dev"/include/glib-2.0/gobject/gobjectnotifyqueue.c ''; }); + pcsclite = (static pkg.pcsclite).overrideAttrs (x: { + configureFlags = [ + "--enable-confdir=/etc" + "--enable-usbdropdir=/var/lib/pcsc/drivers" + "--with-systemdsystemunitdir=${placeholder "bin"}/lib/systemd/system" + ]; + buildInputs = [ pkgs.python3 pkgs.udev pkgs.dbus pkgs.systemd ]; + }); + systemd = (static pkg.systemd).overrideAttrs (x: { + outputs = [ "out" "dev" ]; + mesonFlags = x.mesonFlags ++ [ + "-Dglib=false" + "-Dstatic-libsystemd=true" + ]; + }); }; }; }); @@ -45,8 +60,8 @@ let doCheck = false; enableParallelBuilding = true; outputs = [ "out" ]; - nativeBuildInputs = [ bash gitMinimal go-md2man installShellFiles makeWrapper pkg-config which ]; - buildInputs = [ glibc glibc.static gpgme libassuan libgpgerror libseccomp libapparmor libselinux ]; + nativeBuildInputs = [ bash gitMinimal go-md2man pkg-config which ]; + buildInputs = [ glibc glibc.static glib gpgme libassuan libgpgerror libseccomp libapparmor libselinux ]; prePatch = '' export CFLAGS='-static -pthread' export LDFLAGS='-s -w -static-libgcc -static' diff --git a/nix/nixpkgs.json b/nix/nixpkgs.json index 9b8b8289e..a9771eade 100644 --- a/nix/nixpkgs.json +++ b/nix/nixpkgs.json @@ -1,9 +1,9 @@ { "url": "https://github.com/nixos/nixpkgs", - "rev": "eb7e1ef185f6c990cda5f71fdc4fb02e76ab06d5", - "date": "2021-05-05T23:16:00+02:00", - "path": "/nix/store/a98lkhjlsqh32ic2kkrv5kkik6jy25wh-nixpkgs", - "sha256": "1ibz204c41g7baqga2iaj11yz9l75cfdylkiqjnk5igm81ivivxg", + "rev": "60cce7e5e1fdf62421ef6d4184ee399b46209366", + "date": "2021-06-09T01:18:50-04:00", + "path": "/nix/store/fixgn194626rb7gf99l9jaqm0hbqn2ix-nixpkgs", + "sha256": "100xrb925cana1kfd0c7gwkjjalq891vfgr0rn1gl9j8gp3l3gx6", "fetchSubmodules": false, "deepClone": false, "leaveDotGit": false diff --git a/nix/nixpkgs.nix b/nix/nixpkgs.nix index 21e7f17a2..11c20380a 100644 --- a/nix/nixpkgs.nix +++ b/nix/nixpkgs.nix @@ -5,4 +5,5 @@ let url = "${json.url}/archive/${json.rev}.tar.gz"; inherit (json) sha256; }); -in nixpkgs +in +nixpkgs diff --git a/pkg/api/handlers/compat/images_build.go b/pkg/api/handlers/compat/images_build.go index 50423fb96..e933b9811 100644 --- a/pkg/api/handlers/compat/images_build.go +++ b/pkg/api/handlers/compat/images_build.go @@ -144,8 +144,8 @@ func BuildImage(w http.ResponseWriter, r *http.Request) { if _, found := r.URL.Query()["dockerfile"]; found { var m = []string{} if err := json.Unmarshal([]byte(query.Dockerfile), &m); err != nil { - utils.BadRequest(w, "dockerfile", query.Dockerfile, err) - return + // it's not json, assume just a string + m = append(m, query.Dockerfile) } containerFiles = m } else { @@ -189,8 +189,8 @@ func BuildImage(w http.ResponseWriter, r *http.Request) { var devices = []string{} if _, found := r.URL.Query()["devices"]; found { var m = []string{} - if err := json.Unmarshal([]byte(query.DropCapabilities), &m); err != nil { - utils.BadRequest(w, "devices", query.DropCapabilities, err) + if err := json.Unmarshal([]byte(query.Devices), &m); err != nil { + utils.BadRequest(w, "devices", query.Devices, err) return } devices = m diff --git a/pkg/bindings/images/build.go b/pkg/bindings/images/build.go index c7d432b16..937d05330 100644 --- a/pkg/bindings/images/build.go +++ b/pkg/bindings/images/build.go @@ -299,6 +299,22 @@ func Build(ctx context.Context, containerFiles []string, options entities.BuildO tarContent := []string{options.ContextDirectory} newContainerFiles := []string{} for _, c := range containerFiles { + if c == "/dev/stdin" { + content, err := ioutil.ReadAll(os.Stdin) + if err != nil { + return nil, err + } + tmpFile, err := ioutil.TempFile("", "build") + if err != nil { + return nil, err + } + defer os.Remove(tmpFile.Name()) // clean up + defer tmpFile.Close() + if _, err := tmpFile.Write(content); err != nil { + return nil, err + } + c = tmpFile.Name() + } containerfile, err := filepath.Abs(c) if err != nil { logrus.Errorf("cannot find absolute path of %v: %v", c, err) diff --git a/pkg/cgroups/cgroups.go b/pkg/cgroups/cgroups.go index 911edeb5b..9cb32a364 100644 --- a/pkg/cgroups/cgroups.go +++ b/pkg/cgroups/cgroups.go @@ -165,14 +165,13 @@ func getAvailableControllers(exclude map[string]controllerHandler, cgroup2 bool) if _, found := exclude[name]; found { continue } - isSymLink := false fileInfo, err := os.Stat(cgroupRoot + "/" + name) if err != nil { - isSymLink = !fileInfo.IsDir() + continue } c := controller{ name: name, - symlink: isSymLink, + symlink: !fileInfo.IsDir(), } controllers = append(controllers, c) } diff --git a/pkg/domain/infra/abi/containers.go b/pkg/domain/infra/abi/containers.go index 4908e72f6..5a953c047 100644 --- a/pkg/domain/infra/abi/containers.go +++ b/pkg/domain/infra/abi/containers.go @@ -595,7 +595,7 @@ func (ic *ContainerEngine) ContainerAttach(ctx context.Context, nameOrID string, return nil } -func makeExecConfig(options entities.ExecOptions) *libpod.ExecConfig { +func makeExecConfig(options entities.ExecOptions, rt *libpod.Runtime) (*libpod.ExecConfig, error) { execConfig := new(libpod.ExecConfig) execConfig.Command = options.Cmd execConfig.Terminal = options.Tty @@ -607,7 +607,20 @@ func makeExecConfig(options entities.ExecOptions) *libpod.ExecConfig { execConfig.PreserveFDs = options.PreserveFDs execConfig.AttachStdin = options.Interactive - return execConfig + // Make an exit command + storageConfig := rt.StorageConfig() + runtimeConfig, err := rt.GetConfig() + if err != nil { + return nil, errors.Wrapf(err, "error retrieving Libpod configuration to build exec exit command") + } + // TODO: Add some ability to toggle syslog + exitCommandArgs, err := generate.CreateExitCommandArgs(storageConfig, runtimeConfig, false, true, true) + if err != nil { + return nil, errors.Wrapf(err, "error constructing exit command for exec session") + } + execConfig.ExitCommand = exitCommandArgs + + return execConfig, nil } func checkExecPreserveFDs(options entities.ExecOptions) error { @@ -647,7 +660,10 @@ func (ic *ContainerEngine) ContainerExec(ctx context.Context, nameOrID string, o } ctr := ctrs[0] - execConfig := makeExecConfig(options) + execConfig, err := makeExecConfig(options, ic.Libpod) + if err != nil { + return ec, err + } ec, err = terminal.ExecAttachCtr(ctx, ctr, execConfig, &streams) return define.TranslateExecErrorToExitCode(ec, err), err @@ -664,20 +680,10 @@ func (ic *ContainerEngine) ContainerExecDetached(ctx context.Context, nameOrID s } ctr := ctrs[0] - execConfig := makeExecConfig(options) - - // Make an exit command - storageConfig := ic.Libpod.StorageConfig() - runtimeConfig, err := ic.Libpod.GetConfig() - if err != nil { - return "", errors.Wrapf(err, "error retrieving Libpod configuration to build exec exit command") - } - // TODO: Add some ability to toggle syslog - exitCommandArgs, err := generate.CreateExitCommandArgs(storageConfig, runtimeConfig, false, true, true) + execConfig, err := makeExecConfig(options, ic.Libpod) if err != nil { - return "", errors.Wrapf(err, "error constructing exit command for exec session") + return "", err } - execConfig.ExitCommand = exitCommandArgs // Create and start the exec session id, err := ctr.ExecCreate(execConfig) diff --git a/pkg/domain/infra/abi/parse/parse.go b/pkg/domain/infra/abi/parse/parse.go index 1c590d2d6..56c747711 100644 --- a/pkg/domain/infra/abi/parse/parse.go +++ b/pkg/domain/infra/abi/parse/parse.go @@ -37,7 +37,7 @@ func VolumeOptions(opts map[string]string) ([]libpod.VolumeCreateOption, error) return nil, errors.Wrapf(err, "cannot convert UID %s to integer", splitO[1]) } logrus.Debugf("Removing uid= from options and adding WithVolumeUID for UID %d", intUID) - libpodOptions = append(libpodOptions, libpod.WithVolumeUID(intUID)) + libpodOptions = append(libpodOptions, libpod.WithVolumeUID(intUID), libpod.WithVolumeNoChown()) finalVal = append(finalVal, o) // set option "UID": "$uid" volumeOptions["UID"] = splitO[1] @@ -50,7 +50,7 @@ func VolumeOptions(opts map[string]string) ([]libpod.VolumeCreateOption, error) return nil, errors.Wrapf(err, "cannot convert GID %s to integer", splitO[1]) } logrus.Debugf("Removing gid= from options and adding WithVolumeGID for GID %d", intGID) - libpodOptions = append(libpodOptions, libpod.WithVolumeGID(intGID)) + libpodOptions = append(libpodOptions, libpod.WithVolumeGID(intGID), libpod.WithVolumeNoChown()) finalVal = append(finalVal, o) // set option "GID": "$gid" volumeOptions["GID"] = splitO[1] diff --git a/pkg/machine/config.go b/pkg/machine/config.go index 58794ce42..db9bfa7de 100644 --- a/pkg/machine/config.go +++ b/pkg/machine/config.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/connection.go b/pkg/machine/connection.go index e3985d8ac..3edcbd10e 100644 --- a/pkg/machine/connection.go +++ b/pkg/machine/connection.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/fcos.go b/pkg/machine/fcos.go index 32f943c87..11936aee7 100644 --- a/pkg/machine/fcos.go +++ b/pkg/machine/fcos.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/ignition.go b/pkg/machine/ignition.go index a5c7210af..1d77083d0 100644 --- a/pkg/machine/ignition.go +++ b/pkg/machine/ignition.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/ignition_schema.go b/pkg/machine/ignition_schema.go index 9dbd90ba4..6ac8af826 100644 --- a/pkg/machine/ignition_schema.go +++ b/pkg/machine/ignition_schema.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine /* diff --git a/pkg/machine/keys.go b/pkg/machine/keys.go index 907e28f55..81ec44ea8 100644 --- a/pkg/machine/keys.go +++ b/pkg/machine/keys.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/libvirt/config.go b/pkg/machine/libvirt/config.go index 903f15fbc..1ce5ab154 100644 --- a/pkg/machine/libvirt/config.go +++ b/pkg/machine/libvirt/config.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package libvirt type MachineVM struct { diff --git a/pkg/machine/libvirt/machine.go b/pkg/machine/libvirt/machine.go index c38f63853..e1aa1569b 100644 --- a/pkg/machine/libvirt/machine.go +++ b/pkg/machine/libvirt/machine.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package libvirt import "github.com/containers/podman/v3/pkg/machine" diff --git a/pkg/machine/libvirt/machine_unsupported.go b/pkg/machine/libvirt/machine_unsupported.go new file mode 100644 index 000000000..8b54440fe --- /dev/null +++ b/pkg/machine/libvirt/machine_unsupported.go @@ -0,0 +1,3 @@ +// +build !amd64 amd64,windows + +package libvirt diff --git a/pkg/machine/machine_unsupported.go b/pkg/machine/machine_unsupported.go new file mode 100644 index 000000000..9309d16bc --- /dev/null +++ b/pkg/machine/machine_unsupported.go @@ -0,0 +1,3 @@ +// +build !amd64 amd64,windows + +package machine diff --git a/pkg/machine/pull.go b/pkg/machine/pull.go index 68bb551dc..662896de5 100644 --- a/pkg/machine/pull.go +++ b/pkg/machine/pull.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package machine import ( diff --git a/pkg/machine/qemu/config.go b/pkg/machine/qemu/config.go index e4687914d..013f28960 100644 --- a/pkg/machine/qemu/config.go +++ b/pkg/machine/qemu/config.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package qemu import "time" diff --git a/pkg/machine/qemu/machine.go b/pkg/machine/qemu/machine.go index 31c355d4a..22fb78a5c 100644 --- a/pkg/machine/qemu/machine.go +++ b/pkg/machine/qemu/machine.go @@ -1,3 +1,5 @@ +// +build amd64,linux arm64,linux amd64,darwin arm64,darwin + package qemu import ( diff --git a/pkg/machine/qemu/machine_unsupported.go b/pkg/machine/qemu/machine_unsupported.go new file mode 100644 index 000000000..da06ac324 --- /dev/null +++ b/pkg/machine/qemu/machine_unsupported.go @@ -0,0 +1,3 @@ +// +build !amd64 amd64,windows + +package qemu diff --git a/pkg/rootless/rootless_linux.c b/pkg/rootless/rootless_linux.c index 0d1d6e93e..e5f9e88d9 100644 --- a/pkg/rootless/rootless_linux.c +++ b/pkg/rootless/rootless_linux.c @@ -333,7 +333,7 @@ static void __attribute__((constructor)) init() uid_t uid; gid_t gid; char path[PATH_MAX]; - const char *const suffix = "/libpod/pause.pid"; + const char *const suffix = "/libpod/tmp/pause.pid"; char *cwd = getcwd (NULL, 0); char uid_fmt[16]; char gid_fmt[16]; diff --git a/test/apiv2/10-images.at b/test/apiv2/10-images.at index 037a4c01f..9e464dbc7 100644 --- a/test/apiv2/10-images.at +++ b/test/apiv2/10-images.at @@ -147,4 +147,39 @@ t GET "images/get?names=alpine&names=busybox" 200 '[POSIX tar archive]' img_cnt=$(tar xf "$WORKDIR/curl.result.out" manifest.json -O | jq "length") is "$img_cnt" 2 "number of images in tar archive" +# check build works when uploading container file as a tar, see issue #10660 +TMPD=$(mktemp -d podman-apiv2-test.build.XXXXXXXX) +function cleanBuildTest() { + podman rmi -a -f + rm -rf "${TMPD}" &> /dev/null +} +CONTAINERFILE_TAR="${TMPD}/containerfile.tar" +cat > $TMPD/containerfile << EOF +FROM quay.io/libpod/alpine_labels:latest +EOF +tar --format=posix -C $TMPD -cvf ${CONTAINERFILE_TAR} containerfile &> /dev/null + +curl -XPOST --data-binary @<(cat $CONTAINERFILE_TAR) \ + -H "content-type: application/x-tar" \ + --dump-header "${TMPD}/headers.txt" \ + -o "${TMPD}/response.txt" \ + "http://$HOST:$PORT/v1.40/libpod/build?dockerfile=containerfile" &> /dev/null + +BUILD_TEST_ERROR="" + +if ! grep -q '200 OK' "${TMPD}/headers.txt"; then + echo -e "${red}NOK: Image build from tar failed response was not 200 OK" + BUILD_TEST_ERROR="1" +fi + +if ! grep -q 'quay.io/libpod/alpine_labels' "${TMPD}/response.txt"; then + echo -e "${red}NOK: Image build from tar failed image name not in response" + BUILD_TEST_ERROR="1" +fi + +cleanBuildTest +if [[ "${BUILD_TEST_ERROR}" ]]; then + exit 1 +fi + # vim: filetype=sh diff --git a/test/compose/test-compose b/test/compose/test-compose index 981f78a79..70db6dd55 100755 --- a/test/compose/test-compose +++ b/test/compose/test-compose @@ -183,6 +183,8 @@ function test_port() { fi echo "# cat $WORKDIR/server.log:" cat $WORKDIR/server.log + echo "# cat $logfile:" + cat $logfile return fi diff --git a/test/e2e/build_test.go b/test/e2e/build_test.go index 6255690b1..abaacdd5e 100644 --- a/test/e2e/build_test.go +++ b/test/e2e/build_test.go @@ -604,4 +604,38 @@ RUN echo hello`, ALPINE) Expect(inspect.OutputToString()).To(Equal("windows")) }) + + It("podman build device test", func() { + if _, err := os.Lstat("/dev/fuse"); err != nil { + Skip(fmt.Sprintf("test requires stat /dev/fuse to work: %v", err)) + } + containerfile := fmt.Sprintf(`FROM %s +RUN ls /dev/fuse`, ALPINE) + containerfilePath := filepath.Join(podmanTest.TempDir, "Containerfile") + err := ioutil.WriteFile(containerfilePath, []byte(containerfile), 0755) + Expect(err).To(BeNil()) + session := podmanTest.Podman([]string{"build", "--pull-never", "-t", "test", "--file", containerfilePath, podmanTest.TempDir}) + session.WaitWithDefaultTimeout() + Expect(session.ExitCode()).To(Equal(125)) + + session = podmanTest.Podman([]string{"build", "--pull-never", "--device", "/dev/fuse", "-t", "test", "--file", containerfilePath, podmanTest.TempDir}) + session.WaitWithDefaultTimeout() + Expect(session.ExitCode()).To(Equal(0)) + }) + + It("podman build device rename test", func() { + SkipIfRootless("rootless builds do not currently support renaming devices") + containerfile := fmt.Sprintf(`FROM %s +RUN ls /dev/test1`, ALPINE) + containerfilePath := filepath.Join(podmanTest.TempDir, "Containerfile") + err := ioutil.WriteFile(containerfilePath, []byte(containerfile), 0755) + Expect(err).To(BeNil()) + session := podmanTest.Podman([]string{"build", "--pull-never", "-t", "test", "--file", containerfilePath, podmanTest.TempDir}) + session.WaitWithDefaultTimeout() + Expect(session.ExitCode()).To(Equal(125)) + + session = podmanTest.Podman([]string{"build", "--pull-never", "--device", "/dev/zero:/dev/test1", "-t", "test", "--file", containerfilePath, podmanTest.TempDir}) + session.WaitWithDefaultTimeout() + Expect(session.ExitCode()).To(Equal(0)) + }) }) diff --git a/test/e2e/run_volume_test.go b/test/e2e/run_volume_test.go index 9b77aaef8..4be1b2009 100644 --- a/test/e2e/run_volume_test.go +++ b/test/e2e/run_volume_test.go @@ -668,4 +668,36 @@ USER testuser`, fedoraMinimal) Expect(strings.Contains(test2.OutputToString(), testString)).To(BeTrue()) }) + + It("podman volume with uid and gid works", func() { + volName := "testVol" + volCreate := podmanTest.Podman([]string{"volume", "create", "--opt", "o=uid=1000", volName}) + volCreate.WaitWithDefaultTimeout() + Expect(volCreate.ExitCode()).To(Equal(0)) + + volMount := podmanTest.Podman([]string{"run", "--rm", "-v", fmt.Sprintf("%s:/test", volName), ALPINE, "stat", "-c", "%u", "/test"}) + volMount.WaitWithDefaultTimeout() + Expect(volMount.ExitCode()).To(Equal(0)) + Expect(volMount.OutputToString()).To(Equal("1000")) + + volName = "testVol2" + volCreate = podmanTest.Podman([]string{"volume", "create", "--opt", "o=gid=1000", volName}) + volCreate.WaitWithDefaultTimeout() + Expect(volCreate.ExitCode()).To(Equal(0)) + + volMount = podmanTest.Podman([]string{"run", "--rm", "-v", fmt.Sprintf("%s:/test", volName), ALPINE, "stat", "-c", "%g", "/test"}) + volMount.WaitWithDefaultTimeout() + Expect(volMount.ExitCode()).To(Equal(0)) + Expect(volMount.OutputToString()).To(Equal("1000")) + + volName = "testVol3" + volCreate = podmanTest.Podman([]string{"volume", "create", "--opt", "o=uid=1000,gid=1000", volName}) + volCreate.WaitWithDefaultTimeout() + Expect(volCreate.ExitCode()).To(Equal(0)) + + volMount = podmanTest.Podman([]string{"run", "--rm", "-v", fmt.Sprintf("%s:/test", volName), ALPINE, "stat", "-c", "%u:%g", "/test"}) + volMount.WaitWithDefaultTimeout() + Expect(volMount.ExitCode()).To(Equal(0)) + Expect(volMount.OutputToString()).To(Equal("1000:1000")) + }) }) diff --git a/test/e2e/systemd_test.go b/test/e2e/systemd_test.go index b132750b0..8dc14d5f7 100644 --- a/test/e2e/systemd_test.go +++ b/test/e2e/systemd_test.go @@ -6,6 +6,7 @@ import ( "strings" "time" + "github.com/containers/podman/v3/pkg/rootless" . "github.com/containers/podman/v3/test/utils" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" @@ -115,6 +116,12 @@ WantedBy=multi-user.target conData := result.InspectContainerToJSON() Expect(len(conData)).To(Equal(1)) Expect(conData[0].Config.SystemdMode).To(BeTrue()) + + if CGROUPSV2 || !rootless.IsRootless() { + stats := podmanTest.Podman([]string{"stats", "--no-stream", ctrName}) + stats.WaitWithDefaultTimeout() + Expect(stats.ExitCode()).To(Equal(0)) + } }) It("podman create container with systemd entrypoint triggers systemd mode", func() { diff --git a/test/system/070-build.bats b/test/system/070-build.bats index 40622d6cc..6843e28a5 100644 --- a/test/system/070-build.bats +++ b/test/system/070-build.bats @@ -29,6 +29,29 @@ EOF run_podman rmi -f build_test } +@test "podman build test -f -" { + rand_filename=$(random_string 20) + rand_content=$(random_string 50) + + tmpdir=$PODMAN_TMPDIR/build-test + mkdir -p $tmpdir + containerfile=$PODMAN_TMPDIR/Containerfile + cat >$containerfile <<EOF +FROM $IMAGE +RUN apk add nginx +RUN echo $rand_content > /$rand_filename +EOF + + # The 'apk' command can take a long time to fetch files; bump timeout + PODMAN_TIMEOUT=240 run_podman build -t build_test -f - --format=docker $tmpdir < $containerfile + is "$output" ".*STEP 4: COMMIT" "COMMIT seen in log" + + run_podman run --rm build_test cat /$rand_filename + is "$output" "$rand_content" "reading generated file in image" + + run_podman rmi -f build_test +} + @test "podman build - global runtime flags test" { skip_if_remote "--runtime-flag flag not supported for remote" diff --git a/test/system/410-selinux.bats b/test/system/410-selinux.bats index f8cee0e59..4ef9c8b30 100644 --- a/test/system/410-selinux.bats +++ b/test/system/410-selinux.bats @@ -183,7 +183,10 @@ function check_label() { # runc and crun emit different diagnostics runtime=$(podman_runtime) case "$runtime" in - crun) expect="\`/proc/thread-self/attr/exec\`: OCI runtime error: unable to assign security attribute" ;; + # crun 0.20.1 changes the error message + # from /proc/thread-self/attr/exec`: .* unable to assign + # to /proc/self/attr/keycreate`: .* unable to process + crun) expect="\`/proc/.*\`: OCI runtime error: unable to \(assign\|process\) security attribute" ;; runc) expect="OCI runtime error: .*: failed to set /proc/self/attr/keycreate on procfs" ;; *) skip "Unknown runtime '$runtime'";; esac diff --git a/test/system/build-testimage b/test/system/build-testimage index 3e5b982ce..eb5849b5e 100755 --- a/test/system/build-testimage +++ b/test/system/build-testimage @@ -61,8 +61,8 @@ chmod 755 pause # - check for updates @ https://hub.docker.com/_/alpine # busybox-extras provides httpd needed in 500-networking.bats cat >Containerfile <<EOF -ARG ARCH=please-override-arch -FROM docker.io/\${ARCH}/alpine:3.12.0 +ARG REPO=please-override-repo +FROM docker.io/\${REPO}/alpine:3.13.5 RUN apk add busybox-extras ADD testimage-id pause /home/podman/ LABEL created_by=$create_script @@ -74,17 +74,46 @@ EOF # --squash-all : needed by 'tree' test in 070-build.bats podman rmi -f testimage &> /dev/null || true +# There should always be a testimage tagged ':0000000<X>' (eight digits, +# zero-padded sequence ID) in the same location; this is used by tests +# which need to pull a non-locally-cached image. This image will rarely +# if ever need to change, nor in fact does it even have to be a copy of +# this testimage since all we use it for is 'true'. +# However, it does need to be multiarch :-( +zerotag_latest=$(skopeo list-tags docker://quay.io/libpod/testimage |\ + jq -r '.Tags[]' |\ + sort --version-sort |\ + grep '^000' |\ + tail -n 1) +zerotag_next=$(printf "%08d" $((zerotag_latest + 1))) + +# We don't always need to push the :00xx image, but build it anyway. +zeroimg=quay.io/libpod/testimage:${zerotag_next} +buildah manifest create $zeroimg + # We need to use buildah because (as of 2021-02-23) only buildah has --manifest # and because Dan says arch emulation is not currently working on podman # (no further details). # Arch emulation on Fedora requires the qemu-user-static package. -for arch in amd64 arm64v8 ppc64le s390x;do +for arch in amd64 arm64 ppc64le s390x;do + # docker.io repo is usually the same name as the desired arch; except + # for arm64, where podman needs to have the arch be 'arm64' but the + # image lives in 'arm64v8'. + repo=$arch + if [[ $repo = "arm64" ]]; then + repo="${repo}v8" + fi + ${BUILDAH} bud \ --arch=$arch \ - --build-arg ARCH=$arch \ + --build-arg REPO=$repo \ --manifest=testimage \ --squash \ . + + # The zero-tag image + ${BUILDAH} pull --arch $arch docker.io/$repo/busybox:1.33.1 + ${BUILDAH} manifest add $zeroimg docker.io/$repo/busybox:1.33.1 done # Clean up @@ -94,23 +123,13 @@ rm -rf $tmpdir # Tag image and push (all arches) to quay. remote_tag=quay.io/libpod/testimage:$YMD podman tag testimage ${remote_tag} -${BUILDAH} manifest push --all ${remote_tag} docker://${remote_tag} +cat <<EOF -# Side note: there should always be a testimage tagged ':0000000<X>' -# (eight digits, zero-padded sequence ID) in the same location; this is -# used by tests which need to pull a non-locally-cached image. This -# image will rarely if ever need to change, nor in fact does it even -# have to be a copy of this testimage since all we use it for is 'true'. -# However, it does need to be multiarch :-( -# -# As of 2021-02-24 it is simply busybox, because it is super small, -# but it's complicated because of multiarch: -# -# img=quay.io/libpod/testimage:0000000<current+1> -# buildah manifest create $img -# for arch in amd64 arm64v8 ppc64le s390x;do -# buildah pull --arch $arch docker.io/$arch/busybox:1.32.0 -# buildah manifest add $img docker.io/$arch/busybox:1.32.0 -# done -# buildah manifest push --all $img docker://$img -# +If you're happy with these images, run: + + ${BUILDAH} manifest push --all ${remote_tag} docker://${remote_tag} + ${BUILDAH} manifest push --all ${zeroimg} docker://${zeroimg} + +(You do not always need to push the :0000 image) + +EOF diff --git a/test/system/helpers.bash b/test/system/helpers.bash index e0c208f57..1859a2168 100644 --- a/test/system/helpers.bash +++ b/test/system/helpers.bash @@ -7,14 +7,15 @@ PODMAN=${PODMAN:-podman} PODMAN_TEST_IMAGE_REGISTRY=${PODMAN_TEST_IMAGE_REGISTRY:-"quay.io"} PODMAN_TEST_IMAGE_USER=${PODMAN_TEST_IMAGE_USER:-"libpod"} PODMAN_TEST_IMAGE_NAME=${PODMAN_TEST_IMAGE_NAME:-"testimage"} -PODMAN_TEST_IMAGE_TAG=${PODMAN_TEST_IMAGE_TAG:-"20210427"} +PODMAN_TEST_IMAGE_TAG=${PODMAN_TEST_IMAGE_TAG:-"20210610"} PODMAN_TEST_IMAGE_FQN="$PODMAN_TEST_IMAGE_REGISTRY/$PODMAN_TEST_IMAGE_USER/$PODMAN_TEST_IMAGE_NAME:$PODMAN_TEST_IMAGE_TAG" PODMAN_TEST_IMAGE_ID= # Remote image that we *DO NOT* fetch or keep by default; used for testing pull -# This changed from 0 to 1 on 2021-02-24 due to multiarch considerations; it -# should change only very rarely. -PODMAN_NONLOCAL_IMAGE_FQN="$PODMAN_TEST_IMAGE_REGISTRY/$PODMAN_TEST_IMAGE_USER/$PODMAN_TEST_IMAGE_NAME:00000002" +# This has changed in 2021, from 0 through 3, various iterations of getting +# multiarch to work. It should change only very rarely. +PODMAN_NONLOCAL_IMAGE_TAG=${PODMAN_NONLOCAL_IMAGE_TAG:-"00000003"} +PODMAN_NONLOCAL_IMAGE_FQN="$PODMAN_TEST_IMAGE_REGISTRY/$PODMAN_TEST_IMAGE_USER/$PODMAN_TEST_IMAGE_NAME:$PODMAN_NONLOCAL_IMAGE_TAG" # Because who wants to spell that out each time? IMAGE=$PODMAN_TEST_IMAGE_FQN diff --git a/vendor/github.com/containers/storage/VERSION b/vendor/github.com/containers/storage/VERSION index 96cd6ee1e..c78d39b8e 100644 --- a/vendor/github.com/containers/storage/VERSION +++ b/vendor/github.com/containers/storage/VERSION @@ -1 +1 @@ -1.32.1 +1.32.2 diff --git a/vendor/github.com/containers/storage/drivers/overlay/overlay.go b/vendor/github.com/containers/storage/drivers/overlay/overlay.go index d5d161bfd..87d608393 100644 --- a/vendor/github.com/containers/storage/drivers/overlay/overlay.go +++ b/vendor/github.com/containers/storage/drivers/overlay/overlay.go @@ -1189,7 +1189,11 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.MountO if d.usingMetacopy { optsList = append(optsList, "metacopy=on") } else { - logrus.Warnf("ignoring metacopy option from storage.conf, not supported with booted kernel") + logLevel := logrus.WarnLevel + if unshare.IsRootless() { + logLevel = logrus.DebugLevel + } + logrus.StandardLogger().Logf(logLevel, "ignoring metacopy option from storage.conf, not supported with booted kernel") } } } diff --git a/vendor/github.com/containers/storage/go.mod b/vendor/github.com/containers/storage/go.mod index e7ca56e64..2217ea739 100644 --- a/vendor/github.com/containers/storage/go.mod +++ b/vendor/github.com/containers/storage/go.mod @@ -10,15 +10,15 @@ require ( github.com/google/go-intervals v0.0.2 github.com/hashicorp/go-multierror v1.1.1 github.com/json-iterator/go v1.1.11 - github.com/klauspost/compress v1.12.3 + github.com/klauspost/compress v1.13.0 github.com/klauspost/pgzip v1.2.5 - github.com/mattn/go-shellwords v1.0.11 + github.com/mattn/go-shellwords v1.0.12 github.com/mistifyio/go-zfs v2.1.2-0.20190413222219-f784269be439+incompatible github.com/moby/sys/mountinfo v0.4.1 github.com/opencontainers/go-digest v1.0.0 github.com/opencontainers/runc v1.0.0-rc95 github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 - github.com/opencontainers/selinux v1.8.1 + github.com/opencontainers/selinux v1.8.2 github.com/pkg/errors v0.9.1 github.com/sirupsen/logrus v1.8.1 github.com/stretchr/testify v1.7.0 diff --git a/vendor/github.com/containers/storage/go.sum b/vendor/github.com/containers/storage/go.sum index 5373d0597..6cd6df17c 100644 --- a/vendor/github.com/containers/storage/go.sum +++ b/vendor/github.com/containers/storage/go.sum @@ -77,6 +77,8 @@ github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+Ce github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= github.com/bitly/go-simplejson v0.5.0/go.mod h1:cXHtHw4XUPsvGaxgjIAn8PhEWG9NfngEKAMDJEczWVA= +github.com/bits-and-blooms/bitset v1.2.0 h1:Kn4yilvwNtMACtf1eYDlG8H77R07mZSPbMjLyS07ChA= +github.com/bits-and-blooms/bitset v1.2.0/go.mod h1:gIdJ4wp64HaoK2YrL1Q5/N7Y16edYb8uY+O0FJTyyDA= github.com/blang/semver v3.1.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4= @@ -383,8 +385,8 @@ github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= -github.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU= -github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= +github.com/klauspost/compress v1.13.0 h1:2T7tUoQrQT+fQWdaY5rjWztFGAFwbGD04iPJg90ZiOs= +github.com/klauspost/compress v1.13.0/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -408,8 +410,8 @@ github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaO github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o= -github.com/mattn/go-shellwords v1.0.11 h1:vCoR9VPpsk/TZFW2JwK5I9S0xdrtUq2bph6/YjEPnaw= -github.com/mattn/go-shellwords v1.0.11/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y= +github.com/mattn/go-shellwords v1.0.12 h1:M2zGm7EW6UQJvDeQxo4T51eKPurbeFbe8WtebGE2xrk= +github.com/mattn/go-shellwords v1.0.12/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= github.com/miekg/pkcs11 v1.0.3/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs= @@ -476,8 +478,8 @@ github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417/go.m github.com/opencontainers/runtime-tools v0.0.0-20181011054405-1d69bd0f9c39/go.mod h1:r3f7wjNzSs2extwzU3Y+6pKfobzPh+kKFJ3ofN+3nfs= github.com/opencontainers/selinux v1.6.0/go.mod h1:VVGKuOLlE7v4PJyT6h7mNWvq1rzqiriPsEqVhc+svHE= github.com/opencontainers/selinux v1.8.0/go.mod h1:RScLhm78qiWa2gbVCcGkC7tCGdgk3ogry1nUQF8Evvo= -github.com/opencontainers/selinux v1.8.1 h1:yvEZh7CsfnJNwKzG9ZeXwbvR05RAZsu5RS/3vA6qFTA= -github.com/opencontainers/selinux v1.8.1/go.mod h1:RScLhm78qiWa2gbVCcGkC7tCGdgk3ogry1nUQF8Evvo= +github.com/opencontainers/selinux v1.8.2 h1:c4ca10UMgRcvZ6h0K4HtS15UaVSBEaE+iln2LVpAuGc= +github.com/opencontainers/selinux v1.8.2/go.mod h1:MUIHuUEvKB1wtJjQdOyYRgOnLD2xAPP8dBsCoU0KuF8= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= @@ -588,7 +590,6 @@ github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmF github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU= github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= github.com/willf/bitset v1.1.11-0.20200630133818-d5bec3311243/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4= -github.com/willf/bitset v1.1.11 h1:N7Z7E9UvjW+sGsEl7k/SJrvY2reP1A07MrGuCjIOjRE= github.com/willf/bitset v1.1.11/go.mod h1:83CECat5yLh5zVOf4P1ErAgKA5UDvKtgyUABdr3+MjI= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ= diff --git a/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go b/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go index 0a92da2c0..f0e46b6b6 100644 --- a/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go +++ b/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go @@ -194,11 +194,7 @@ func (l *lockfile) Touch() error { defer l.stateMutex.Unlock() l.lw = stringid.GenerateRandomID() id := []byte(l.lw) - _, err := unix.Seek(int(l.fd), 0, os.SEEK_SET) - if err != nil { - return err - } - n, err := unix.Write(int(l.fd), id) + n, err := unix.Pwrite(int(l.fd), id, 0) if err != nil { return err } @@ -217,11 +213,7 @@ func (l *lockfile) Modified() (bool, error) { panic("attempted to check last-writer in lockfile without locking it first") } defer l.stateMutex.Unlock() - _, err := unix.Seek(int(l.fd), 0, os.SEEK_SET) - if err != nil { - return true, err - } - n, err := unix.Read(int(l.fd), id) + n, err := unix.Pread(int(l.fd), id, 0) if err != nil { return true, err } diff --git a/vendor/github.com/containers/storage/store.go b/vendor/github.com/containers/storage/store.go index d6d547c64..dc6eaafa2 100644 --- a/vendor/github.com/containers/storage/store.go +++ b/vendor/github.com/containers/storage/store.go @@ -937,7 +937,7 @@ func (s *store) ContainerStore() (ContainerStore, error) { } func (s *store) canUseShifting(uidmap, gidmap []idtools.IDMap) bool { - if !s.graphDriver.SupportsShifting() { + if s.graphDriver == nil || !s.graphDriver.SupportsShifting() { return false } if uidmap != nil && !idtools.IsContiguous(uidmap) { @@ -2668,6 +2668,10 @@ func (s *store) mount(id string, options drivers.MountOpts) (string, error) { s.lastLoaded = time.Now() } + if options.UidMaps != nil || options.GidMaps != nil { + options.DisableShifting = !s.canUseShifting(options.UidMaps, options.GidMaps) + } + if rlstore.Exists(id) { return rlstore.Mount(id, options) } @@ -2708,7 +2712,6 @@ func (s *store) Mount(id, mountLabel string) (string, error) { options.Volatile = v.(bool) } } - options.DisableShifting = !s.canUseShifting(container.UIDMap, container.GIDMap) } return s.mount(id, options) } diff --git a/vendor/github.com/containers/storage/types/default_override_test.conf b/vendor/github.com/containers/storage/types/default_override_test.conf new file mode 100644 index 000000000..caa537ba9 --- /dev/null +++ b/vendor/github.com/containers/storage/types/default_override_test.conf @@ -0,0 +1,11 @@ +[storage] + +# Default Storage Driver +driver = "" + +# Primary Read/Write location of container storage +graphroot = "environment_override_graphroot" + +# Storage path for rootless users +# +rootless_storage_path = "environment_override_rootless_storage_path" diff --git a/vendor/github.com/containers/storage/types/utils.go b/vendor/github.com/containers/storage/types/utils.go index d2dca7b68..4d62b151a 100644 --- a/vendor/github.com/containers/storage/types/utils.go +++ b/vendor/github.com/containers/storage/types/utils.go @@ -160,7 +160,14 @@ func expandEnvPath(path string, rootlessUID int) (string, error) { } func DefaultConfigFile(rootless bool) (string, error) { - if defaultConfigFileSet || !rootless { + if defaultConfigFileSet { + return defaultConfigFile, nil + } + + if path, ok := os.LookupEnv("CONTAINERS_STORAGE_CONF"); ok { + return path, nil + } + if !rootless { return defaultConfigFile, nil } diff --git a/vendor/github.com/klauspost/compress/zstd/blockdec.go b/vendor/github.com/klauspost/compress/zstd/blockdec.go index 6cea054d2..e30af505c 100644 --- a/vendor/github.com/klauspost/compress/zstd/blockdec.go +++ b/vendor/github.com/klauspost/compress/zstd/blockdec.go @@ -144,7 +144,7 @@ func (b *blockDec) reset(br byteBuffer, windowSize uint64) error { } cSize = 1 case blockTypeCompressed: - if debug { + if debugDecoder { println("Data size on stream:", cSize) } b.RLESize = 0 @@ -153,7 +153,7 @@ func (b *blockDec) reset(br byteBuffer, windowSize uint64) error { maxSize = int(windowSize) } if cSize > maxCompressedBlockSize || uint64(cSize) > b.WindowSize { - if debug { + if debugDecoder { printf("compressed block too big: csize:%d block: %+v\n", uint64(cSize), b) } return ErrCompressedSizeTooBig @@ -179,7 +179,7 @@ func (b *blockDec) reset(br byteBuffer, windowSize uint64) error { } b.data, err = br.readBig(cSize, b.dataStorage) if err != nil { - if debug { + if debugDecoder { println("Reading block:", err, "(", cSize, ")", len(b.data)) printf("%T", br) } @@ -249,7 +249,7 @@ func (b *blockDec) startDecoder() { b: b.dst, err: err, } - if debug { + if debugDecoder { println("Decompressed to", len(b.dst), "bytes, error:", err) } b.result <- o @@ -264,7 +264,7 @@ func (b *blockDec) startDecoder() { default: panic("Invalid block type") } - if debug { + if debugDecoder { println("blockDec: Finished block") } } @@ -297,7 +297,7 @@ func (b *blockDec) decodeBuf(hist *history) error { b.dst = hist.b hist.b = nil err := b.decodeCompressed(hist) - if debug { + if debugDecoder { println("Decompressed to total", len(b.dst), "bytes, hash:", xxhash.Sum64(b.dst), "error:", err) } hist.b = b.dst @@ -390,7 +390,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { in = in[5:] } } - if debug { + if debugDecoder { println("literals type:", litType, "litRegenSize:", litRegenSize, "litCompSize:", litCompSize, "sizeFormat:", sizeFormat, "4X:", fourStreams) } var literals []byte @@ -428,7 +428,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { literals[i] = v } in = in[1:] - if debug { + if debugDecoder { printf("Found %d RLE compressed literals\n", litRegenSize) } case literalsBlockTreeless: @@ -439,7 +439,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { // Store compressed literals, so we defer decoding until we get history. literals = in[:litCompSize] in = in[litCompSize:] - if debug { + if debugDecoder { printf("Found %d compressed literals\n", litCompSize) } case literalsBlockCompressed: @@ -481,7 +481,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { if len(literals) != litRegenSize { return fmt.Errorf("literal output size mismatch want %d, got %d", litRegenSize, len(literals)) } - if debug { + if debugDecoder { printf("Decompressed %d literals into %d bytes\n", litCompSize, litRegenSize) } } @@ -532,12 +532,12 @@ func (b *blockDec) decodeCompressed(hist *history) error { br := byteReader{b: in, off: 0} compMode := br.Uint8() br.advance(1) - if debug { + if debugDecoder { printf("Compression modes: 0b%b", compMode) } for i := uint(0); i < 3; i++ { mode := seqCompMode((compMode >> (6 - i*2)) & 3) - if debug { + if debugDecoder { println("Table", tableIndex(i), "is", mode) } var seq *sequenceDec @@ -568,7 +568,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { } dec.setRLE(symb) seq.fse = dec - if debug { + if debugDecoder { printf("RLE set to %+v, code: %v", symb, v) } case compModeFSE: @@ -584,7 +584,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { println("Transform table error:", err) return err } - if debug { + if debugDecoder { println("Read table ok", "symbolLen:", dec.symbolLen) } seq.fse = dec @@ -652,7 +652,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { if huff != nil { hist.huffTree = huff } - if debug { + if debugDecoder { println("Final literals:", len(literals), "hash:", xxhash.Sum64(literals), "and", nSeqs, "sequences.") } @@ -669,7 +669,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { if err != nil { return err } - if debug { + if debugDecoder { println("History merged ok") } br := &bitReader{} @@ -728,7 +728,7 @@ func (b *blockDec) decodeCompressed(hist *history) error { } hist.append(b.dst) hist.recentOffsets = seqs.prevOffset - if debug { + if debugDecoder { println("Finished block with literals:", len(literals), "and", nSeqs, "sequences.") } diff --git a/vendor/github.com/klauspost/compress/zstd/blockenc.go b/vendor/github.com/klauspost/compress/zstd/blockenc.go index e1be092f3..3df185ee4 100644 --- a/vendor/github.com/klauspost/compress/zstd/blockenc.go +++ b/vendor/github.com/klauspost/compress/zstd/blockenc.go @@ -156,7 +156,7 @@ func (h *literalsHeader) setSize(regenLen int) { switch { case inBits < 5: lh |= (uint64(regenLen) << 3) | (1 << 60) - if debug { + if debugEncoder { got := int(lh>>3) & 0xff if got != regenLen { panic(fmt.Sprint("litRegenSize = ", regenLen, "(want) != ", got, "(got)")) @@ -184,7 +184,7 @@ func (h *literalsHeader) setSizes(compLen, inLen int, single bool) { lh |= 1 << 2 } lh |= (uint64(inLen) << 4) | (uint64(compLen) << (10 + 4)) | (3 << 60) - if debug { + if debugEncoder { const mmask = (1 << 24) - 1 n := (lh >> 4) & mmask if int(n&1023) != inLen { @@ -312,7 +312,7 @@ func (b *blockEnc) encodeRaw(a []byte) { bh.setType(blockTypeRaw) b.output = bh.appendTo(b.output[:0]) b.output = append(b.output, a...) - if debug { + if debugEncoder { println("Adding RAW block, length", len(a), "last:", b.last) } } @@ -325,7 +325,7 @@ func (b *blockEnc) encodeRawTo(dst, src []byte) []byte { bh.setType(blockTypeRaw) dst = bh.appendTo(dst) dst = append(dst, src...) - if debug { + if debugEncoder { println("Adding RAW block, length", len(src), "last:", b.last) } return dst @@ -339,7 +339,7 @@ func (b *blockEnc) encodeLits(lits []byte, raw bool) error { // Don't compress extremely small blocks if len(lits) < 8 || (len(lits) < 32 && b.dictLitEnc == nil) || raw { - if debug { + if debugEncoder { println("Adding RAW block, length", len(lits), "last:", b.last) } bh.setType(blockTypeRaw) @@ -371,7 +371,7 @@ func (b *blockEnc) encodeLits(lits []byte, raw bool) error { switch err { case huff0.ErrIncompressible: - if debug { + if debugEncoder { println("Adding RAW block, length", len(lits), "last:", b.last) } bh.setType(blockTypeRaw) @@ -379,7 +379,7 @@ func (b *blockEnc) encodeLits(lits []byte, raw bool) error { b.output = append(b.output, lits...) return nil case huff0.ErrUseRLE: - if debug { + if debugEncoder { println("Adding RLE block, length", len(lits)) } bh.setType(blockTypeRLE) @@ -396,12 +396,12 @@ func (b *blockEnc) encodeLits(lits []byte, raw bool) error { bh.setType(blockTypeCompressed) var lh literalsHeader if reUsed { - if debug { + if debugEncoder { println("Reused tree, compressed to", len(out)) } lh.setType(literalsBlockTreeless) } else { - if debug { + if debugEncoder { println("New tree, compressed to", len(out), "tree size:", len(b.litEnc.OutTable)) } lh.setType(literalsBlockCompressed) @@ -517,7 +517,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { lh.setSize(len(b.literals)) b.output = lh.appendTo(b.output) b.output = append(b.output, b.literals...) - if debug { + if debugEncoder { println("Adding literals RAW, length", len(b.literals)) } case huff0.ErrUseRLE: @@ -525,22 +525,22 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { lh.setSize(len(b.literals)) b.output = lh.appendTo(b.output) b.output = append(b.output, b.literals[0]) - if debug { + if debugEncoder { println("Adding literals RLE") } case nil: // Compressed litLen... if reUsed { - if debug { + if debugEncoder { println("reused tree") } lh.setType(literalsBlockTreeless) } else { - if debug { + if debugEncoder { println("new tree, size:", len(b.litEnc.OutTable)) } lh.setType(literalsBlockCompressed) - if debug { + if debugEncoder { _, _, err := huff0.ReadTable(out, nil) if err != nil { panic(err) @@ -548,18 +548,18 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { } } lh.setSizes(len(out), len(b.literals), single) - if debug { + if debugEncoder { printf("Compressed %d literals to %d bytes", len(b.literals), len(out)) println("Adding literal header:", lh) } b.output = lh.appendTo(b.output) b.output = append(b.output, out...) b.litEnc.Reuse = huff0.ReusePolicyAllow - if debug { + if debugEncoder { println("Adding literals compressed") } default: - if debug { + if debugEncoder { println("Adding literals ERROR:", err) } return err @@ -577,7 +577,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { n := len(b.sequences) - 0x7f00 b.output = append(b.output, 255, uint8(n), uint8(n>>8)) } - if debug { + if debugEncoder { println("Encoding", len(b.sequences), "sequences") } b.genCodes() @@ -611,17 +611,17 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { nSize = nSize + (nSize+2*8*16)>>4 switch { case predefSize <= prevSize && predefSize <= nSize || forcePreDef: - if debug { + if debugEncoder { println("Using predefined", predefSize>>3, "<=", nSize>>3) } return preDef, compModePredefined case prevSize <= nSize: - if debug { + if debugEncoder { println("Using previous", prevSize>>3, "<=", nSize>>3) } return prev, compModeRepeat default: - if debug { + if debugEncoder { println("Using new, predef", predefSize>>3, ". previous:", prevSize>>3, ">", nSize>>3, "header max:", cur.maxHeaderSize()>>3, "bytes") println("tl:", cur.actualTableLog, "symbolLen:", cur.symbolLen, "norm:", cur.norm[:cur.symbolLen], "hist", cur.count[:cur.symbolLen]) } @@ -634,7 +634,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { if llEnc.useRLE { mode |= uint8(compModeRLE) << 6 llEnc.setRLE(b.sequences[0].llCode) - if debug { + if debugEncoder { println("llEnc.useRLE") } } else { @@ -645,7 +645,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { if ofEnc.useRLE { mode |= uint8(compModeRLE) << 4 ofEnc.setRLE(b.sequences[0].ofCode) - if debug { + if debugEncoder { println("ofEnc.useRLE") } } else { @@ -657,7 +657,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { if mlEnc.useRLE { mode |= uint8(compModeRLE) << 2 mlEnc.setRLE(b.sequences[0].mlCode) - if debug { + if debugEncoder { println("mlEnc.useRLE, code: ", b.sequences[0].mlCode, "value", b.sequences[0].matchLen) } } else { @@ -666,7 +666,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { mode |= uint8(m) << 2 } b.output = append(b.output, mode) - if debug { + if debugEncoder { printf("Compression modes: 0b%b", mode) } b.output, err = llEnc.writeCount(b.output) @@ -786,7 +786,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error { // Size is output minus block header. bh.setSize(uint32(len(b.output)-bhOffset) - 3) - if debug { + if debugEncoder { println("Rewriting block header", bh) } _ = bh.appendTo(b.output[bhOffset:bhOffset]) diff --git a/vendor/github.com/klauspost/compress/zstd/bytebuf.go b/vendor/github.com/klauspost/compress/zstd/bytebuf.go index 17e820a6a..aab71c6cf 100644 --- a/vendor/github.com/klauspost/compress/zstd/bytebuf.go +++ b/vendor/github.com/klauspost/compress/zstd/bytebuf.go @@ -91,7 +91,7 @@ func (r *readerWrapper) readSmall(n int) ([]byte, error) { if err == io.EOF { return nil, io.ErrUnexpectedEOF } - if debug { + if debugDecoder { println("readSmall: got", n2, "want", n, "err", err) } return nil, err diff --git a/vendor/github.com/klauspost/compress/zstd/decoder.go b/vendor/github.com/klauspost/compress/zstd/decoder.go index f593e464b..4d984c3b2 100644 --- a/vendor/github.com/klauspost/compress/zstd/decoder.go +++ b/vendor/github.com/klauspost/compress/zstd/decoder.go @@ -113,9 +113,6 @@ func NewReader(r io.Reader, opts ...DOption) (*Decoder, error) { // Returns the number of bytes written and any error that occurred. // When the stream is done, io.EOF will be returned. func (d *Decoder) Read(p []byte) (int, error) { - if d.stream == nil { - return 0, ErrDecoderNilInput - } var n int for { if len(d.current.b) > 0 { @@ -138,7 +135,7 @@ func (d *Decoder) Read(p []byte) (int, error) { } } if len(d.current.b) > 0 { - if debug { + if debugDecoder { println("returning", n, "still bytes left:", len(d.current.b)) } // Only return error at end of block @@ -147,7 +144,7 @@ func (d *Decoder) Read(p []byte) (int, error) { if d.current.err != nil { d.drainOutput() } - if debug { + if debugDecoder { println("returning", n, d.current.err, len(d.decoders)) } return n, d.current.err @@ -167,20 +164,17 @@ func (d *Decoder) Reset(r io.Reader) error { if r == nil { d.current.err = ErrDecoderNilInput + if len(d.current.b) > 0 { + d.current.b = d.current.b[:0] + } d.current.flushed = true return nil } - if d.stream == nil { - d.stream = make(chan decodeStream, 1) - d.streamWg.Add(1) - go d.startStreamDecoder(d.stream) - } - - // If bytes buffer and < 1MB, do sync decoding anyway. - if bb, ok := r.(byter); ok && bb.Len() < 1<<20 { + // If bytes buffer and < 5MB, do sync decoding anyway. + if bb, ok := r.(byter); ok && bb.Len() < 5<<20 { bb2 := bb - if debug { + if debugDecoder { println("*bytes.Buffer detected, doing sync decode, len:", bb.Len()) } b := bb2.Bytes() @@ -196,12 +190,18 @@ func (d *Decoder) Reset(r io.Reader) error { d.current.b = dst d.current.err = err d.current.flushed = true - if debug { + if debugDecoder { println("sync decode to", len(dst), "bytes, err:", err) } return nil } + if d.stream == nil { + d.stream = make(chan decodeStream, 1) + d.streamWg.Add(1) + go d.startStreamDecoder(d.stream) + } + // Remove current block. d.current.decodeOutput = decodeOutput{} d.current.err = nil @@ -225,7 +225,7 @@ func (d *Decoder) drainOutput() { d.current.cancel = nil } if d.current.d != nil { - if debug { + if debugDecoder { printf("re-adding current decoder %p, decoders: %d", d.current.d, len(d.decoders)) } d.decoders <- d.current.d @@ -238,7 +238,7 @@ func (d *Decoder) drainOutput() { } for v := range d.current.output { if v.d != nil { - if debug { + if debugDecoder { printf("re-adding decoder %p", v.d) } d.decoders <- v.d @@ -255,9 +255,6 @@ func (d *Decoder) drainOutput() { // The return value n is the number of bytes written. // Any error encountered during the write is also returned. func (d *Decoder) WriteTo(w io.Writer) (int64, error) { - if d.stream == nil { - return 0, ErrDecoderNilInput - } var n int64 for { if len(d.current.b) > 0 { @@ -297,7 +294,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) { block := <-d.decoders frame := block.localFrame defer func() { - if debug { + if debugDecoder { printf("re-adding decoder: %p", block) } frame.rawInput = nil @@ -310,7 +307,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) { frame.history.reset() err := frame.reset(&frame.bBuf) if err == io.EOF { - if debug { + if debugDecoder { println("frame reset return EOF") } return dst, nil @@ -355,7 +352,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) { return dst, err } if len(frame.bBuf) == 0 { - if debug { + if debugDecoder { println("frame dbuf empty") } break @@ -371,7 +368,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) { // if no data was available without blocking. func (d *Decoder) nextBlock(blocking bool) (ok bool) { if d.current.d != nil { - if debug { + if debugDecoder { printf("re-adding current decoder %p", d.current.d) } d.decoders <- d.current.d @@ -391,7 +388,7 @@ func (d *Decoder) nextBlock(blocking bool) (ok bool) { return false } } - if debug { + if debugDecoder { println("got", len(d.current.b), "bytes, error:", d.current.err) } return true @@ -485,7 +482,7 @@ func (d *Decoder) startStreamDecoder(inStream chan decodeStream) { defer d.streamWg.Done() frame := newFrameDec(d.o) for stream := range inStream { - if debug { + if debugDecoder { println("got new stream") } br := readerWrapper{r: stream.r} @@ -493,7 +490,7 @@ func (d *Decoder) startStreamDecoder(inStream chan decodeStream) { for { frame.history.reset() err := frame.reset(&br) - if debug && err != nil { + if debugDecoder && err != nil { println("Frame decoder returned", err) } if err == nil && frame.DictionaryID != nil { @@ -510,7 +507,7 @@ func (d *Decoder) startStreamDecoder(inStream chan decodeStream) { } break } - if debug { + if debugDecoder { println("starting frame decoder") } diff --git a/vendor/github.com/klauspost/compress/zstd/dict.go b/vendor/github.com/klauspost/compress/zstd/dict.go index fa25a18d8..a36ae83ef 100644 --- a/vendor/github.com/klauspost/compress/zstd/dict.go +++ b/vendor/github.com/klauspost/compress/zstd/dict.go @@ -82,7 +82,7 @@ func loadDict(b []byte) (*dict, error) { println("Transform table error:", err) return err } - if debug { + if debugDecoder || debugEncoder { println("Read table ok", "symbolLen:", dec.symbolLen) } // Set decoders as predefined so they aren't reused. diff --git a/vendor/github.com/klauspost/compress/zstd/enc_best.go b/vendor/github.com/klauspost/compress/zstd/enc_best.go index dc1eed5f0..b7d4b9004 100644 --- a/vendor/github.com/klauspost/compress/zstd/enc_best.go +++ b/vendor/github.com/klauspost/compress/zstd/enc_best.go @@ -132,7 +132,7 @@ func (e *bestFastEncoder) Encode(blk *blockEnc, src []byte) { } _ = addLiterals - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -274,7 +274,7 @@ encodeLoop: nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, best.length) } @@ -412,7 +412,7 @@ encodeLoop: blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) blk.recentOffsets[2] = uint32(offset3) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } diff --git a/vendor/github.com/klauspost/compress/zstd/enc_better.go b/vendor/github.com/klauspost/compress/zstd/enc_better.go index 604954290..eab7b5083 100644 --- a/vendor/github.com/klauspost/compress/zstd/enc_better.go +++ b/vendor/github.com/klauspost/compress/zstd/enc_better.go @@ -138,7 +138,7 @@ func (e *betterFastEncoder) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -204,7 +204,7 @@ encodeLoop: nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -264,7 +264,7 @@ encodeLoop: s += lenght + repOff2 nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -553,7 +553,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } @@ -656,7 +656,7 @@ func (e *betterFastEncoderDict) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -724,7 +724,7 @@ encodeLoop: nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -787,7 +787,7 @@ encodeLoop: s += lenght + repOff2 nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -1084,7 +1084,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } diff --git a/vendor/github.com/klauspost/compress/zstd/enc_dfast.go b/vendor/github.com/klauspost/compress/zstd/enc_dfast.go index 8629d43d8..96b21b90e 100644 --- a/vendor/github.com/klauspost/compress/zstd/enc_dfast.go +++ b/vendor/github.com/klauspost/compress/zstd/enc_dfast.go @@ -109,7 +109,7 @@ func (e *doubleFastEncoder) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -170,7 +170,7 @@ encodeLoop: s += lenght + repOff nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -368,7 +368,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } @@ -427,7 +427,7 @@ func (e *doubleFastEncoder) EncodeNoHist(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -483,7 +483,7 @@ encodeLoop: s += length + repOff nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, length) } @@ -677,7 +677,7 @@ encodeLoop: blk.literals = append(blk.literals, src[nextEmit:]...) blk.extraLits = len(src) - int(nextEmit) } - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } @@ -767,7 +767,7 @@ func (e *doubleFastEncoderDict) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -830,7 +830,7 @@ encodeLoop: s += lenght + repOff nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, lenght) } @@ -1039,7 +1039,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } // If we encoded more than 64K mark all dirty. diff --git a/vendor/github.com/klauspost/compress/zstd/enc_fast.go b/vendor/github.com/klauspost/compress/zstd/enc_fast.go index ba4a17e10..2246d286d 100644 --- a/vendor/github.com/klauspost/compress/zstd/enc_fast.go +++ b/vendor/github.com/klauspost/compress/zstd/enc_fast.go @@ -103,7 +103,7 @@ func (e *fastEncoder) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -178,7 +178,7 @@ encodeLoop: s += length + 2 nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, length) } @@ -330,7 +330,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } @@ -343,7 +343,7 @@ func (e *fastEncoder) EncodeNoHist(blk *blockEnc, src []byte) { inputMargin = 8 minNonLiteralBlockSize = 1 + 1 + inputMargin ) - if debug { + if debugEncoder { if len(src) > maxBlockSize { panic("src too big") } @@ -391,7 +391,7 @@ func (e *fastEncoder) EncodeNoHist(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -462,7 +462,7 @@ encodeLoop: s += length + 2 nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, length) } @@ -616,7 +616,7 @@ encodeLoop: blk.literals = append(blk.literals, src[nextEmit:]...) blk.extraLits = len(src) - int(nextEmit) } - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } // We do not store history, so we must offset e.cur to avoid false matches for next user. @@ -696,7 +696,7 @@ func (e *fastEncoderDict) Encode(blk *blockEnc, src []byte) { blk.literals = append(blk.literals, src[nextEmit:until]...) s.litLen = uint32(until - nextEmit) } - if debug { + if debugEncoder { println("recent offsets:", blk.recentOffsets) } @@ -773,7 +773,7 @@ encodeLoop: s += length + 2 nextEmit = s if s >= sLimit { - if debug { + if debugEncoder { println("repeat ended", s, length) } @@ -926,7 +926,7 @@ encodeLoop: } blk.recentOffsets[0] = uint32(offset1) blk.recentOffsets[1] = uint32(offset2) - if debug { + if debugEncoder { println("returning, recent offsets:", blk.recentOffsets, "extra literals:", blk.extraLits) } } diff --git a/vendor/github.com/klauspost/compress/zstd/encoder.go b/vendor/github.com/klauspost/compress/zstd/encoder.go index 4871dd03a..ea85548fc 100644 --- a/vendor/github.com/klauspost/compress/zstd/encoder.go +++ b/vendor/github.com/klauspost/compress/zstd/encoder.go @@ -245,7 +245,7 @@ func (e *Encoder) nextBlock(final bool) error { s.filling, s.current, s.previous = s.previous[:0], s.filling, s.current s.wg.Add(1) go func(src []byte) { - if debug { + if debugEncoder { println("Adding block,", len(src), "bytes, final:", final) } defer func() { @@ -290,7 +290,7 @@ func (e *Encoder) nextBlock(final bool) error { } switch err { case errIncompressible: - if debug { + if debugEncoder { println("Storing incompressible block as raw") } blk.encodeRaw(src) @@ -313,7 +313,7 @@ func (e *Encoder) nextBlock(final bool) error { // // The Copy function uses ReaderFrom if available. func (e *Encoder) ReadFrom(r io.Reader) (n int64, err error) { - if debug { + if debugEncoder { println("Using ReadFrom") } @@ -336,20 +336,20 @@ func (e *Encoder) ReadFrom(r io.Reader) (n int64, err error) { switch err { case io.EOF: e.state.filling = e.state.filling[:len(e.state.filling)-len(src)] - if debug { + if debugEncoder { println("ReadFrom: got EOF final block:", len(e.state.filling)) } return n, nil case nil: default: - if debug { + if debugEncoder { println("ReadFrom: got error:", err) } e.state.err = err return n, err } if len(src) > 0 { - if debug { + if debugEncoder { println("ReadFrom: got space left in source:", len(src)) } continue @@ -512,7 +512,7 @@ func (e *Encoder) EncodeAll(src, dst []byte) []byte { switch err { case errIncompressible: - if debug { + if debugEncoder { println("Storing incompressible block as raw") } dst = blk.encodeRawTo(dst, src) @@ -548,7 +548,7 @@ func (e *Encoder) EncodeAll(src, dst []byte) []byte { switch err { case errIncompressible: - if debug { + if debugEncoder { println("Storing incompressible block as raw") } dst = blk.encodeRawTo(dst, todo) diff --git a/vendor/github.com/klauspost/compress/zstd/framedec.go b/vendor/github.com/klauspost/compress/zstd/framedec.go index 4dc151213..e8cc9a2c2 100644 --- a/vendor/github.com/klauspost/compress/zstd/framedec.go +++ b/vendor/github.com/klauspost/compress/zstd/framedec.go @@ -78,20 +78,33 @@ func newFrameDec(o decoderOptions) *frameDec { func (d *frameDec) reset(br byteBuffer) error { d.HasCheckSum = false d.WindowSize = 0 - var b []byte + var signature [4]byte for { var err error - b, err = br.readSmall(4) + // Check if we can read more... + b, err := br.readSmall(1) switch err { case io.EOF, io.ErrUnexpectedEOF: return io.EOF default: return err case nil: + signature[0] = b[0] + } + // Read the rest, don't allow io.ErrUnexpectedEOF + b, err = br.readSmall(3) + switch err { + case io.EOF: + return io.EOF + default: + return err + case nil: + copy(signature[1:], b) } - if !bytes.Equal(b[1:4], skippableFrameMagic) || b[0]&0xf0 != 0x50 { - if debug { - println("Not skippable", hex.EncodeToString(b), hex.EncodeToString(skippableFrameMagic)) + + if !bytes.Equal(signature[1:4], skippableFrameMagic) || signature[0]&0xf0 != 0x50 { + if debugDecoder { + println("Not skippable", hex.EncodeToString(signature[:]), hex.EncodeToString(skippableFrameMagic)) } // Break if not skippable frame. break @@ -99,28 +112,34 @@ func (d *frameDec) reset(br byteBuffer) error { // Read size to skip b, err = br.readSmall(4) if err != nil { - println("Reading Frame Size", err) + if debugDecoder { + println("Reading Frame Size", err) + } return err } n := uint32(b[0]) | (uint32(b[1]) << 8) | (uint32(b[2]) << 16) | (uint32(b[3]) << 24) println("Skipping frame with", n, "bytes.") err = br.skipN(int(n)) if err != nil { - if debug { + if debugDecoder { println("Reading discarded frame", err) } return err } } - if !bytes.Equal(b, frameMagic) { - println("Got magic numbers: ", b, "want:", frameMagic) + if !bytes.Equal(signature[:], frameMagic) { + if debugDecoder { + println("Got magic numbers: ", signature, "want:", frameMagic) + } return ErrMagicMismatch } // Read Frame_Header_Descriptor fhd, err := br.readByte() if err != nil { - println("Reading Frame_Header_Descriptor", err) + if debugDecoder { + println("Reading Frame_Header_Descriptor", err) + } return err } d.SingleSegment = fhd&(1<<5) != 0 @@ -135,7 +154,9 @@ func (d *frameDec) reset(br byteBuffer) error { if !d.SingleSegment { wd, err := br.readByte() if err != nil { - println("Reading Window_Descriptor", err) + if debugDecoder { + println("Reading Window_Descriptor", err) + } return err } printf("raw: %x, mantissa: %d, exponent: %d\n", wd, wd&7, wd>>3) @@ -153,7 +174,7 @@ func (d *frameDec) reset(br byteBuffer) error { size = 4 } - b, err = br.readSmall(int(size)) + b, err := br.readSmall(int(size)) if err != nil { println("Reading Dictionary_ID", err) return err @@ -167,7 +188,7 @@ func (d *frameDec) reset(br byteBuffer) error { case 4: id = uint32(b[0]) | (uint32(b[1]) << 8) | (uint32(b[2]) << 16) | (uint32(b[3]) << 24) } - if debug { + if debugDecoder { println("Dict size", size, "ID:", id) } if id > 0 { @@ -191,7 +212,7 @@ func (d *frameDec) reset(br byteBuffer) error { } d.FrameContentSize = 0 if fcsSize > 0 { - b, err = br.readSmall(fcsSize) + b, err := br.readSmall(fcsSize) if err != nil { println("Reading Frame content", err) return err @@ -209,7 +230,7 @@ func (d *frameDec) reset(br byteBuffer) error { d2 := uint32(b[4]) | (uint32(b[5]) << 8) | (uint32(b[6]) << 16) | (uint32(b[7]) << 24) d.FrameContentSize = uint64(d1) | (uint64(d2) << 32) } - if debug { + if debugDecoder { println("field size bits:", v, "fcsSize:", fcsSize, "FrameContentSize:", d.FrameContentSize, hex.EncodeToString(b[:fcsSize]), "singleseg:", d.SingleSegment, "window:", d.WindowSize) } } @@ -252,7 +273,7 @@ func (d *frameDec) reset(br byteBuffer) error { // next will start decoding the next block from stream. func (d *frameDec) next(block *blockDec) error { - if debug { + if debugDecoder { printf("decoding new block %p:%p", block, block.data) } err := block.reset(d.rawInput, d.WindowSize) @@ -263,7 +284,7 @@ func (d *frameDec) next(block *blockDec) error { return err } block.input <- struct{}{} - if debug { + if debugDecoder { println("next block:", block) } d.asyncRunningMu.Lock() @@ -318,12 +339,12 @@ func (d *frameDec) checkCRC() error { } if !bytes.Equal(tmp[:], want) { - if debug { + if debugDecoder { println("CRC Check Failed:", tmp[:], "!=", want) } return ErrCRCMismatch } - if debug { + if debugDecoder { println("CRC ok", tmp[:]) } return nil @@ -344,7 +365,7 @@ func (d *frameDec) initAsync() { if cap(d.decoding) < d.o.concurrent { d.decoding = make(chan *blockDec, d.o.concurrent) } - if debug { + if debugDecoder { h := d.history printf("history init. len: %d, cap: %d", len(h.b), cap(h.b)) } @@ -392,7 +413,7 @@ func (d *frameDec) startDecoder(output chan decodeOutput) { output <- r return } - if debug { + if debugDecoder { println("got result, from ", d.offset, "to", d.offset+int64(len(r.b))) d.offset += int64(len(r.b)) } @@ -400,7 +421,7 @@ func (d *frameDec) startDecoder(output chan decodeOutput) { // Send history to next block select { case next = <-d.decoding: - if debug { + if debugDecoder { println("Sending ", len(d.history.b), "bytes as history") } next.history <- &d.history @@ -438,7 +459,7 @@ func (d *frameDec) startDecoder(output chan decodeOutput) { output <- r if next == nil { // There was no decoder available, we wait for one now that we have sent to the writer. - if debug { + if debugDecoder { println("Sending ", len(d.history.b), " bytes as history") } next = <-d.decoding @@ -462,7 +483,7 @@ func (d *frameDec) runDecoder(dst []byte, dec *blockDec) ([]byte, error) { if err != nil { break } - if debug { + if debugDecoder { println("next block:", dec) } err = dec.decodeBuf(&d.history) diff --git a/vendor/github.com/klauspost/compress/zstd/fse_encoder.go b/vendor/github.com/klauspost/compress/zstd/fse_encoder.go index c74681b99..b4757ee3f 100644 --- a/vendor/github.com/klauspost/compress/zstd/fse_encoder.go +++ b/vendor/github.com/klauspost/compress/zstd/fse_encoder.go @@ -229,7 +229,7 @@ func (s *fseEncoder) setRLE(val byte) { deltaFindState: 0, deltaNbBits: 0, } - if debug { + if debugEncoder { println("setRLE: val", val, "symbolTT", s.ct.symbolTT[val]) } s.rleVal = val diff --git a/vendor/github.com/klauspost/compress/zstd/snappy.go b/vendor/github.com/klauspost/compress/zstd/snappy.go index 9d9d1d567..0372b1714 100644 --- a/vendor/github.com/klauspost/compress/zstd/snappy.go +++ b/vendor/github.com/klauspost/compress/zstd/snappy.go @@ -203,7 +203,7 @@ func (r *SnappyConverter) Convert(in io.Reader, w io.Writer) (int64, error) { written += int64(n) continue case chunkTypeUncompressedData: - if debug { + if debugEncoder { println("Uncompressed, chunklen", chunkLen) } // Section 4.3. Uncompressed data (chunk type 0x01). @@ -246,7 +246,7 @@ func (r *SnappyConverter) Convert(in io.Reader, w io.Writer) (int64, error) { continue case chunkTypeStreamIdentifier: - if debug { + if debugEncoder { println("stream id", chunkLen, len(snappyMagicBody)) } // Section 4.1. Stream identifier (chunk type 0xff). diff --git a/vendor/github.com/klauspost/compress/zstd/zstd.go b/vendor/github.com/klauspost/compress/zstd/zstd.go index 1ba308c8b..ef1d49a00 100644 --- a/vendor/github.com/klauspost/compress/zstd/zstd.go +++ b/vendor/github.com/klauspost/compress/zstd/zstd.go @@ -15,6 +15,12 @@ import ( // enable debug printing const debug = false +// enable encoding debug printing +const debugEncoder = debug + +// enable decoding debug printing +const debugDecoder = debug + // Enable extra assertions. const debugAsserts = debug || false @@ -82,13 +88,13 @@ var ( ) func println(a ...interface{}) { - if debug { + if debug || debugDecoder || debugEncoder { log.Println(a...) } } func printf(format string, a ...interface{}) { - if debug { + if debug || debugDecoder || debugEncoder { log.Printf(format, a...) } } diff --git a/vendor/github.com/mattn/go-shellwords/shellwords.go b/vendor/github.com/mattn/go-shellwords/shellwords.go index 01afd94d1..1b42a0017 100644 --- a/vendor/github.com/mattn/go-shellwords/shellwords.go +++ b/vendor/github.com/mattn/go-shellwords/shellwords.go @@ -232,7 +232,7 @@ loop: case '\'': if !doubleQuoted && !dollarQuote { if singleQuoted { - got = argSingle + got = argQuoted } singleQuoted = !singleQuoted continue diff --git a/vendor/modules.txt b/vendor/modules.txt index a00f1becc..f2a38caca 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -195,7 +195,7 @@ github.com/containers/psgo/internal/dev github.com/containers/psgo/internal/host github.com/containers/psgo/internal/proc github.com/containers/psgo/internal/process -# github.com/containers/storage v1.32.1 +# github.com/containers/storage v1.32.2 github.com/containers/storage github.com/containers/storage/drivers github.com/containers/storage/drivers/aufs @@ -397,7 +397,7 @@ github.com/json-iterator/go # github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a github.com/juju/ansiterm github.com/juju/ansiterm/tabwriter -# github.com/klauspost/compress v1.12.3 +# github.com/klauspost/compress v1.13.0 github.com/klauspost/compress/flate github.com/klauspost/compress/fse github.com/klauspost/compress/huff0 @@ -417,7 +417,7 @@ github.com/mattn/go-colorable github.com/mattn/go-isatty # github.com/mattn/go-runewidth v0.0.12 github.com/mattn/go-runewidth -# github.com/mattn/go-shellwords v1.0.11 +# github.com/mattn/go-shellwords v1.0.12 github.com/mattn/go-shellwords # github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 github.com/matttproud/golang_protobuf_extensions/pbutil |