1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
# How to sign and distribute container images using Podman
Signing container images originates from the motivation of trusting only
dedicated image providers to mitigate man-in-the-middle (MITM) attacks or
attacks on container registries. One way to sign images is to utilize a GNU
Privacy Guard ([GPG][0]) key. This technique is generally compatible with any
OCI compliant container registry like [Quay.io][1]. It is worth mentioning that
the OpenShift integrated container registry supports this signing mechanism out
of the box, which makes separate signature storage unnecessary.
[0]: https://gnupg.org
[1]: https://quay.io
From a technical perspective, we can utilize Podman to sign the image before
pushing it into a remote registry. After that, all systems running Podman have
to be configured to retrieve the signatures from a remote server, which can
be any simple web server. This means that every unsigned image will be rejected
during an image pull operation. But how does this work?
First of all, we have to create a GPG key pair or select an already locally
available one. To generate a new GPG key, just run `gpg --full-gen-key` and
follow the interactive dialog. Now we should be able to verify that the key
exists locally:
```bash
> gpg --list-keys sgrunert@suse.com
pub rsa2048 2018-11-26 [SC] [expires: 2020-11-25]
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uid [ultimate] Sascha Grunert <sgrunert@suse.com>
sub rsa2048 2018-11-26 [E] [expires: 2020-11-25]
```
Now let’s assume that we run a container registry. For example we could simply
start one on our local machine:
```bash
> sudo podman run -d -p 5000:5000 docker.io/registry
```
The registry does not know anything about image signing, it just provides the remote
storage for the container images. This means if we want to sign an image, we
have to take care of how to distribute the signatures.
Let’s choose a standard `alpine` image for our signing experiment:
```bash
> sudo podman pull docker://docker.io/alpine:latest
```
```bash
> sudo podman images alpine
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/alpine latest e7d92cdc71fe 6 weeks ago 5.86 MB
```
Now we can re-tag the image to point it to our local registry:
```bash
> sudo podman tag alpine localhost:5000/alpine
```
```bash
> sudo podman images alpine
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost:5000/alpine latest e7d92cdc71fe 6 weeks ago 5.86 MB
docker.io/library/alpine latest e7d92cdc71fe 6 weeks ago 5.86 MB
```
Podman would now be able to push the image and sign it in one command. But to
let this work, we have to modify our system-wide registries configuration at
`/etc/containers/registries.d/default.yaml`:
```yaml
default-docker:
sigstore: http://localhost:8000 # Added by us
sigstore-staging: file:///var/lib/containers/sigstore
```
We can see that we have two signature stores configured:
- `sigstore`: referencing a web server for signature reading
- `sigstore-staging`: referencing a file path for signature writing
Now, let’s push and sign the image:
```bash
> sudo -E GNUPGHOME=$HOME/.gnupg \
podman push \
--tls-verify=false \
--sign-by sgrunert@suse.com \
localhost:5000/alpine
…
Storing signatures
```
If we now take a look at the systems signature storage, then we see that there
is a new signature available, which was caused by the image push:
```bash
> sudo ls /var/lib/containers/sigstore
'alpine@sha256=e9b65ef660a3ff91d28cc50eba84f21798a6c5c39b4dd165047db49e84ae1fb9'
```
The default signature store in our edited version of
`/etc/containers/registries.d/default.yaml` references a web server listening at
`http://localhost:8000`. For our experiment, we simply start a new server inside
the local staging signature store:
```bash
> sudo bash -c 'cd /var/lib/containers/sigstore && python3 -m http.server'
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
```
Let’s remove the local images for our verification test:
```
> sudo podman rmi docker.io/alpine localhost:5000/alpine
```
We have to write a policy to enforce that the signature has to be valid. This
can be done by adding a new rule in `/etc/containers/policy.json`. From the
below example, copy the `"docker"` entry into the `"transports"` section of your
`policy.json`.
```json
{
"default": [{ "type": "insecureAcceptAnything" }],
"transports": {
"docker": {
"localhost:5000": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/tmp/key.gpg"
}
]
}
}
}
```
The `keyPath` does not exist yet, so we have to put the GPG key there:
```bash
> gpg --output /tmp/key.gpg --armor --export sgrunert@suse.com
```
If we now pull the image:
```bash
> sudo podman pull --tls-verify=false localhost:5000/alpine
…
Storing signatures
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a
```
Then we can see in the logs of the web server that the signature has been
accessed:
```
127.0.0.1 - - [04/Mar/2020 11:18:21] "GET /alpine@sha256=e9b65ef660a3ff91d28cc50eba84f21798a6c5c39b4dd165047db49e84ae1fb9/signature-1 HTTP/1.1" 200 -
```
As an counterpart example, if we specify the wrong key at `/tmp/key.gpg`:
```bash
> gpg --output /tmp/key.gpg --armor --export mail@saschagrunert.de
File '/tmp/key.gpg' exists. Overwrite? (y/N) y
```
Then a pull is not possible any more:
```bash
> sudo podman pull --tls-verify=false localhost:5000/alpine
Trying to pull localhost:5000/alpine...
Error: error pulling image "localhost:5000/alpine": unable to pull localhost:5000/alpine: unable to pull image: Source image rejected: Invalid GPG signature: …
```
So in general there are four main things to be taken into consideration when
signing container images with Podman and GPG:
1. We need a valid private GPG key on the signing machine and corresponding
public keys on every system which would pull the image
2. A web server has to run somewhere which has access to the signature storage
3. The web server has to be configured in any
`/etc/containers/registries.d/*.yaml` file
4. Every image pulling system has to be configured to contain the enforcing
policy configuration via `policy.conf`
That’s it for image signing and GPG. The cool thing is that this setup works out
of the box with [CRI-O][2] as well and can be used to sign container images in
Kubernetes environments.
[2]: https://cri-o.io
|
