| Commit message (Collapse) | Author | Age |
|\
| |
| |
| |
| | |
fgimenez/fix-docker-networksettings-type-discrepancy
Introduces Address type to be used in secondary IPv4 and IPv6 inspect data structure
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
structure.
Resolves a discrepancy between the types used in inspect for docker and podman.
This causes a panic when using the docker client against podman when the
secondary IP fields in the `NetworkSettings` inspect field are populated.
Fixes containers#12165
Signed-off-by: Federico Gimenez <fgimenez@redhat.com>
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The netns cleanup code is checking if there are running containers, this
can fail if you run several libpod instances with diffrent root/runroot.
To fix it we use one netns for each libpod instances. To prevent name
conflicts we use a hash from the static dir as part of the name.
Previously this worked because we would use the CNI files to check if
the netns was still in use. but this is no longer possible with netavark.
[NO NEW TESTS NEEDED]
Fixes #12306
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Some field names are confusing. Change them so that they make more sense
to the reader.
Since these fields are only in the main branch we can safely rename them
without worrying about backwards compatibility.
Note we have to change the field names in netavark too.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commits adds port forwarding logic directly into podman. The
podman-machine cni plugin is no longer needed.
The following new features are supported:
- works with cni, netavark and slirp4netns
- ports can use the hostIP to bind instead of hard coding 0.0.0.0
- gvproxy no longer listens on 0.0.0.0:7777 (requires a new gvproxy
version)
- support the udp protocol
With this we no longer need podman-machine-cni and should remove it from
the packaging. There is also a change to make sure we are backwards
compatible with old config which include this plugin.
Fixes #11528
Fixes #11728
[NO NEW TESTS NEEDED] We have no podman machine test at the moment.
Please test this manually on your system.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
| |
A rootless container created with a custom userns and forwarded ports
did not work. I refactored the network setup to make the setup logic
more clear.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|\
| |
| | |
libpod: deduplicate ports in db
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The OCICNI port format has one big problem: It does not support ranges.
So if a users forwards a range of 1k ports with podman run -p 1001-2000
we have to store each of the thousand ports individually as array element.
This bloats the db and makes the JSON encoding and decoding much slower.
In many places we already use a better port struct type which supports
ranges, e.g. `pkg/specgen` or the new network interface.
Because of this we have to do many runtime conversions between the two
port formats. If everything uses the new format we can skip the runtime
conversions.
This commit adds logic to replace all occurrences of the old format
with the new one. The database will automatically migrate the ports
to new format when the container config is read for the first time
after the update.
The `ParsePortMapping` function is `pkg/specgen/generate` has been
reworked to better work with the new format. The new logic is able
to deduplicate the given ports. This is necessary the ensure we
store them efficiently in the DB. The new code should also be more
performant than the old one.
To prove that the code is fast enough I added go benchmarks. Parsing
1 million ports took less than 0.5 seconds on my laptop.
Benchmark normalize PortMappings in specgen:
Please note that the 1 million ports are actually 20x 50k ranges
because we cannot have bigger ranges than 65535 ports.
```
$ go test -bench=. -benchmem ./pkg/specgen/generate/
goos: linux
goarch: amd64
pkg: github.com/containers/podman/v3/pkg/specgen/generate
cpu: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
BenchmarkParsePortMappingNoPorts-12 480821532 2.230 ns/op 0 B/op 0 allocs/op
BenchmarkParsePortMapping1-12 38972 30183 ns/op 131584 B/op 9 allocs/op
BenchmarkParsePortMapping100-12 18752 60688 ns/op 141088 B/op 315 allocs/op
BenchmarkParsePortMapping1k-12 3104 331719 ns/op 223840 B/op 3018 allocs/op
BenchmarkParsePortMapping10k-12 376 3122930 ns/op 1223650 B/op 30027 allocs/op
BenchmarkParsePortMapping1m-12 3 390869926 ns/op 124593840 B/op 4000624 allocs/op
BenchmarkParsePortMappingReverse100-12 18940 63414 ns/op 141088 B/op 315 allocs/op
BenchmarkParsePortMappingReverse1k-12 3015 362500 ns/op 223841 B/op 3018 allocs/op
BenchmarkParsePortMappingReverse10k-12 343 3318135 ns/op 1223650 B/op 30027 allocs/op
BenchmarkParsePortMappingReverse1m-12 3 403392469 ns/op 124593840 B/op 4000624 allocs/op
BenchmarkParsePortMappingRange1-12 37635 28756 ns/op 131584 B/op 9 allocs/op
BenchmarkParsePortMappingRange100-12 39604 28935 ns/op 131584 B/op 9 allocs/op
BenchmarkParsePortMappingRange1k-12 38384 29921 ns/op 131584 B/op 9 allocs/op
BenchmarkParsePortMappingRange10k-12 29479 40381 ns/op 131584 B/op 9 allocs/op
BenchmarkParsePortMappingRange1m-12 927 1279369 ns/op 143022 B/op 164 allocs/op
PASS
ok github.com/containers/podman/v3/pkg/specgen/generate 25.492s
```
Benchmark convert old port format to new one:
```
go test -bench=. -benchmem ./libpod/
goos: linux
goarch: amd64
pkg: github.com/containers/podman/v3/libpod
cpu: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
Benchmark_ocicniPortsToNetTypesPortsNoPorts-12 663526126 1.663 ns/op 0 B/op 0 allocs/op
Benchmark_ocicniPortsToNetTypesPorts1-12 7858082 141.9 ns/op 72 B/op 2 allocs/op
Benchmark_ocicniPortsToNetTypesPorts10-12 2065347 571.0 ns/op 536 B/op 4 allocs/op
Benchmark_ocicniPortsToNetTypesPorts100-12 138478 8641 ns/op 4216 B/op 4 allocs/op
Benchmark_ocicniPortsToNetTypesPorts1k-12 9414 120964 ns/op 41080 B/op 4 allocs/op
Benchmark_ocicniPortsToNetTypesPorts10k-12 781 1490526 ns/op 401528 B/op 4 allocs/op
Benchmark_ocicniPortsToNetTypesPorts1m-12 4 250579010 ns/op 40001656 B/op 4 allocs/op
PASS
ok github.com/containers/podman/v3/libpod 11.727s
```
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since we want to use the rootless cni ns also for netavark we should
pick a more generic name. The name is now "rootless network namespace"
or short "rootless netns".
The rename might cause some issues after the update but when the
all containers are restarted or the host is rebooted it should work
correctly.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
| |
| |
| |
| |
| |
| |
| | |
We should mount the full runtime directory into the namespace instead of
just the netns dir. This allows more use cases.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
| |
The check if cleanup is needed reads all container and checks if there
are running containers with bridge networking. If we do not find any we
have to cleanup the ns. However there was a problem with this because
the state is empty by default so the running check never worked.
Fortunately the was a second check which relies on the CNI files so we
still did cleanup anyway.
With netavark I noticed that this check is broken because the CNI files
were not present.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Podman 4.0 currently errors when you use network aliases for a network which
has dns disabled. Because the error happens on network setup this can
cause regression for old working containers. The network backend should not
validate this. Instead podman should check this at container create time
and also for network connect.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This matches what docker does. Also make sure the net aliases are also
shown when the container is stopped.
docker-compose uses this special alias entry to check if it is already
correctly connected to the network. [1]
Because we do not support static ips on network connect at the moment
calling disconnect && connect will loose the static ip.
Fixes #11748
[1] https://github.com/docker/compose/blob/0bea52b18dda3de8c28fcfb0c80cc08b8950645e/compose/service.py#L663-L667
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
| |
Remove ERROR: Error stutter from logrus messages also.
[ NO TESTS NEEDED] This is just code cleanup.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
We do not use the ocicni code anymore so let's get rid of it. Only the
port struct is used but we can copy this into libpod network types so
we can debloat the binary.
The next step is to remove the OCICNI port mapping form the container
config and use the better PortMapping struct everywhere.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Make use of the new network interface in libpod.
This commit contains several breaking changes:
- podman network create only outputs the new network name and not file
path.
- podman network ls shows the network driver instead of the cni version
and plugins.
- podman network inspect outputs the new network struct and not the cni
conflist.
- The bindings and libpod api endpoints have been changed to use the new
network structure.
The container network status is stored in a new field in the state. The
status should be received with the new `c.getNetworkStatus`. This will
migrate the old status to the new format. Therefore old containers should
contine to work correctly in all cases even when network connect/
disconnect is used.
New features:
- podman network reload keeps the ip and mac for more than one network.
- podman container restore keeps the ip and mac for more than one
network.
- The network create compat endpoint can now use more than one ipam
config.
The man pages and the swagger doc are updated to reflect the latest
changes.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
| |
When a container is automatically restarted due its restart policy and
the container uses rootless cni networking with ports forwarded we have
to start a new rootlessport process since it exits with conmon.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
When /etc/resolv.conf is a symlink to an absolute path use it and not
join it the the previous path.
[NO TESTS NEEDED] This depends on the host layout.
Fixes #11358
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
InfraContainer should go through the same creation process as regular containers. This change was from the cmd level
down, involving new container CLI opts and specgen creating functions. What now happens is that both container and pod
cli options are populated in cmd and used to create a podSpecgen and a containerSpecgen. The process then goes as follows
FillOutSpecGen (infra) -> MapSpec (podOpts -> infraOpts) -> PodCreate -> MakePod -> createPodOptions -> NewPod -> CompleteSpec (infra) -> MakeContainer -> NewContainer -> newContainer -> AddInfra (to pod state)
Signed-off-by: cdoern <cdoern@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Podman inspect has to show exposed ports to match docker. This requires
storing the exposed ports in the container config.
A exposed port is shown as `"80/tcp": null` while a forwarded port is
shown as `"80/tcp": [{"HostIp": "", "HostPort": "8080" }]`.
Also make sure to add the exposed ports to the new image when the
container is commited.
Fixes #10777
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a host uses systemd-resolved but not the resolved stub resolver the
following symlinks are created: `/etc/resolv.conf` ->
`/run/systemd/resolve/stub-resolv.conf` -> `/run/systemd/resolve/resolv.conf`.
Because the code uses filepath.EvalSymlinks we put the new resolv.conf
to `/run/systemd/resolve/resolv.conf` but the `/run/systemd/resolve/stub-resolv.conf`
link does not exists in the mount ns.
To fix this we will walk the symlinks manually until we reach the first
one under `/run` and use this for the resolv.conf file destination.
This fixes a regression which was introduced in e73d4829900c.
Fixes #11222
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
| |
[NO TESTS NEEDED] Just fixing spelling.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The rootlessport forwarder requires a child IP to be set. This must be a
valid ip in the container network namespace. The problem is that after a
network disconnect and connect the eth0 ip changed. Therefore the
packages are dropped since the source ip does no longer exists in the
netns.
One solution is to set the child IP to 127.0.0.1, however this is a
security problem. [1]
To fix this we have to recreate the ports after network connect and
disconnect. To make this work the rootlessport process exposes a socket
where podman network connect/disconnect connect to and send to new child
IP to rootlessport. The rootlessport process will remove all ports and
recreate them with the new correct child IP.
Also bump rootlesskit to v0.14.3 to fix a race with RemovePort().
Fixes #10052
[1] https://nvd.nist.gov/vuln/detail/CVE-2021-20199
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|\
| |
| | |
Fix race conditions in rootless cni setup
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There was an race condition when calling `GetRootlessCNINetNs()`. It
created the rootless cni directory before it got locked. Therefore
another process could have called cleanup and removed this directory
before it was used resulting in errors. The lockfile got moved into the
XDG_RUNTIME_DIR directory to prevent a panic when the parent dir was
removed by cleanup.
Fixes #10930
Fixes #10922
To make this even more robust `GetRootlessCNINetNs()` will now return
locked. This guarantees that we can run `Do()` after `GetRootlessCNINetNs()`
before another process could have called `Cleanup()` in between.
[NO TESTS NEEDED] CI is flaking, hopefully this will fix it.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix issue 10929 : `[Regression in 3.2.0] CNI-in-slirp4netns DNS gets broken when running a rootful container after running a rootless container`
When /etc/resolv.conf on the host is a symlink to /run/systemd/resolve/stub-resolv.conf,
we have to mount an empty filesystem on /run/systemd/resolve in the child namespace,
so as to isolate the directory from the host mount namespace.
Otherwise our bind-mount for /run/systemd/resolve/stub-resolv.conf is unmounted
when systemd-resolved unlinks and recreates /run/systemd/resolve/stub-resolv.conf on the host.
[NO TESTS NEEDED]
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The rootless cni namespace needs a valid /etc/resolv.conf file. On some
distros is a symlink to somewhere under /run. Because the kernel will
follow the symlink before mounting, it is not possible to mount a file
at exactly /etc/resolv.conf. We have to ensure that the link target will
be available in the rootless cni mount ns.
Fixes #10855
Also fixed a bug in the /var/lib/cni directory lookup logic. It used
`filepath.Base` instead of `filepath.Dir` and thus looping infinitely.
Fixes #10857
[NO TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|\
| |
| | |
getContainerNetworkInfo: lock netNsCtr before sync
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
`syncContainer()` requires the container to be locked, otherwise we can
end up with undefined behavior.
[NO TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Podman does not need to watch the cni config directory. If a network is
not found in the cache, OCICNI will reload the networks anyway and thus
even podman system service should work as expected.
Also include a change to not mount a "new" /var by default in the
rootless cni ns, instead try to use /var/lib/cni first and then the
parent dir. This allows users to store cni configs under /var/... which
is the case for the CI compose test.
[NO TESTS NEEDED]
Fixes #10686
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Network connect/disconnect has to call the cni plugins when the network
namespace is already configured. This is the case for `ContainerStateRunning`
and `ContainerStateCreated`. This is important otherwise the network is
not attached to this network namespace and libpod will throw errors like
`network inspection mismatch...` This problem happened when using
`docker-compose up` in attached mode.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Using the gvproxy application on the host, we can now port forward from
the machine vm on the host. It requires that 'gvproxy' be installed in
an executable location. gvproxy can be found in the
containers/gvisor-tap-vsock github repo.
[NO TESTS NEEDED]
Signed-off-by: Brent Baude <bbaude@redhat.com>
|
|\
| |
| | |
Add host.containers.internal entry into container's etc/hosts
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This change adds the entry `host.containers.internal` to the `/etc/hosts`
file within a new containers filesystem. The ip address is determined by
the containers networking configuration and points to the gateway address
for the containers networking namespace.
Closes #5651
Signed-off-by: Baron Lenardson <lenardson.baron@gmail.com>
|
|/
|
|
|
|
|
|
|
|
| |
Allow podman network reload to be run as rootless user. While it is
unlikely that the iptable rules are flushed inside the rootless cni
namespace, it could still happen. Also fix podman network reload --all
to ignore errors when a container does not have the bridge network mode,
e.g. slirp4netns.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The source ip for the rootlesskit port forwarder was hardcoded to the
standard slirp4netns ip. This is incorrect since users can change the
subnet used by slirp4netns with `--network slirp4netns:cidr=10.5.0.0/24`.
The container interface ip is always the .100 in the subnet. Only when
the rootlesskit port forwarder child ip matches the container interface
ip the port forwarding will work.
Fixes #9828
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
| |
Add a new --rootless-cni option to podman unshare to also join the
rootless-cni network namespace. This is useful if you want to connect
to a rootless container via IP address. This is only possible from the
rootless-cni namespace and not from the host namespace. This option also
helps to debug problems in the rootless-cni namespace.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
| |
The CNI plugins need access to iptables in $PATH. On debian /usr/sbin
is not added to $PATH for rootless users. This will break rootless
cni completely. To prevent breaking existing users add /usr/sbin to
$PATH in podman if needed.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
| |
If a user only has a local dns server in the resolv.conf file the dns
resolution will fail. Instead we create a new resolv.conf which will use
the slirp4netns dns.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
| |
Delte the network namespace and kill the slirp4netns process when it is
no longer needed.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
| |
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
| |
Do not invoke the rootlesskit port forwarder when the container has no
ports.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
| |
With the new rootless cni supporting network connect/disconnect is easy.
Combine common setps into extra functions to prevent code duplication.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
| |
This should make maintenance easier.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
| |
This is supported with the new rootless cni logic.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of creating an extra container create a network and mount
namespace inside the podman user namespace. This ns is used to
for rootless cni operations.
This helps to align the rootless and rootful network code path.
If we run as rootless we just have to set up a extra net ns and
initialize slirp4netns in it. The ocicni lib will be called in
that net ns.
This design allows allows easier maintenance, no extra container
with pause processes, support for rootless cni with --uidmap
and possibly more.
The biggest problem is backwards compatibility. I don't think
live migration can be possible. If the user reboots or restart
all cni containers everything should work as expected again.
The user is left with the rootless-cni-infa container and image
but this can safely be removed.
To make the existing cni configs work we need execute the cni plugins
in a extra mount namespace. This ensures that we can safely mount over
/run and /var which have to be writeable for the cni plugins without
removing access to these files by the main podman process. One caveat
is that we need to keep the netns files at `XDG_RUNTIME_DIR/netns`
accessible.
`XDG_RUNTIME_DIR/rootless-cni/{run,var}` will be mounted to `/{run,var}`.
To ensure that we keep the netns directory we bind mount this relative
to the new root location, e.g. XDG_RUNTIME_DIR/rootless-cni/run/user/1000/netns
before we mount the run directory. The run directory is mounted recursive,
this makes the netns directory at the same path accessible as before.
This also allows iptables-legacy to work because /run/xtables.lock is
now writeable.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Make sure we do not display the expected error when using podman network
reload. This is already done for iptables-legacy however iptables-nft
creates a slightly different error message so check for this as well.
The error is logged at info level.
[NO TESTS NEEDED] The test VMs do not use iptables-nft so there is no
way to test this. It is already tested for iptables-legacy.
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
|
| |
Make sure to pass the cni interface descriptions to cni teardowns.
Otherwise cni cannot find the correct cache files because the
interface name might not match the networks. This can only happen
when network disconnect was used.
Fixes #9602
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
| |
Only the the network mode bridge supports cni networks.
Other network modes cannot use network connect/disconnect
so we should throw a error.
Fixes #9496
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The libpod network logic knows about networks IDs but OCICNI
does not. We cannot pass the network ID to OCICNI. Instead we
need to make sure we only use network names internally. This
is also important for libpod since we also only store the
network names in the state. If we would add a ID there the
same networks could accidentally be added twice.
Fixes #9451
Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
|