Hooks supports two directories, process default and override

ALso cleanup files section or podman man page

Add description of policy.json
Sort alphabetically.
Add more info on  oci hooks

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>

Closes: #1487
Approved by: umohnani8

5 files changed, 48 insertions, 33 deletions

diff --git a/cmd/podman/libpodruntime/runtime.go b/cmd/podman/libpodruntime/runtime.go
index be15d138d..a0d497e8e 100644
--- a/cmd/podman/libpodruntime/runtime.go
+++ b/cmd/podman/libpodruntime/runtime.go
@@ -136,7 +136,9 @@ func GetRuntimeWithStorageOpts(c *cli.Context, storageOpts *storage.StoreOptions
 	if c.GlobalIsSet("default-mounts-file") {
 		options = append(options, libpod.WithDefaultMountsFile(c.GlobalString("default-mounts-file")))
 	}
-	options = append(options, libpod.WithHooksDir(c.GlobalString("hooks-dir-path"), c.GlobalIsSet("hooks-dir-path")))
+	if c.GlobalIsSet("hooks-dir-path") {
+		options = append(options, libpod.WithHooksDir(c.GlobalString("hooks-dir-path")))
+	}
 
 	// TODO flag to set CNI plugins dir?
 
diff --git a/docs/podman.1.md b/docs/podman.1.md
index eb07ed491..3a0943d6b 100644
--- a/docs/podman.1.md
+++ b/docs/podman.1.md
@@ -54,14 +54,14 @@ Path to the OCI compatible binary used to run containers
 
 **--storage-driver, -s**=**value**
 
-Storage driver.  The default storage driver for UID 0 is configured in /etc/containers/storage.conf, and is *vfs* for other users.  The `STORAGE_DRIVER` environment variable overrides the default.  The --storage-driver specified driver overrides all.
+Storage driver.  The default storage driver for UID 0 is configured in /etc/containers/storage.conf (`$HOME/.config/containers/storage.conf` in rootless mode), and is *vfs* for other users.  The `STORAGE_DRIVER` environment variable overrides the default.  The --storage-driver specified driver overrides all.
 
 Overriding this option will cause the *storage-opt* settings in /etc/containers/storage.conf to be ignored.  The user must
 specify additional options via the `--storage-opt` flag.
 
 **--storage-opt**=**value**
 
-Storage driver option, Default storage driver options are configured in /etc/containers/storage.conf. The `STORAGE_OPTS` environment variable overrides the default. The --storage-opt specified options overrides all.
+Storage driver option, Default storage driver options are configured in /etc/containers/storage.conf (`$HOME/.config/containers/storage.conf` in rootless mode). The `STORAGE_OPTS` environment variable overrides the default. The --storage-opt specified options overrides all.
 
 **--syslog**
 
@@ -153,37 +153,41 @@ the exit codes follow the `chroot` standard, see below:
 
 **libpod.conf** (`/etc/containers/libpod.conf`)
 
-libpod.conf is the configuration file for all tools using libpod to manage containers.  When Podman runs in rootless mode, then the file `$HOME/.config/containers/libpod.conf` is used.
+    libpod.conf is the configuration file for all tools using libpod to manage containers.  When Podman runs in rootless mode, then the file `$HOME/.config/containers/libpod.conf` is used.
 
-**storage.conf** (`/etc/containers/storage.conf`)
+**mounts.conf** (`/usr/share/containers/mounts.conf` and optionally `/etc/containers/mounts.conf`)
 
-storage.conf is the storage configuration file for all tools using containers/storage
+    The mounts.conf file specifies volume mount directories that are automatically mounted inside containers when executing the `podman run` or `podman start` commands. When Podman runs in rootless mode, the file `$HOME/.config/containers/mounts.conf` is also used. Please refer to containers-mounts.conf(5) for further details.
 
-The storage configuration file specifies all of the available container storage options for tools using shared container storage.
+**OCI hooks JSON** (`/etc/containers/oci/hooks.d/*.json`, `/usr/share/containers/oci/hooks.d/*.json`)
 
-When Podman runs in rootless mode, the file `$HOME/.config/containers/storage.conf` is also loaded.
+    Each `*.json` file in `/etc/containers/oci/hooks.d` and `/usr/share/containers/oci/hooks.d` configures a hook for Podman containers, with `/etc/containers/oci/hooks.d` having higher precedence.  For more details on the syntax of the JSON files and the semantics of hook injection, see `oci-hooks(5)`.
 
-**mounts.conf** (`/usr/share/containers/mounts.conf` and optionally `/etc/containers/mounts.conf`)
+    Podman and libpod currently support both the 1.0.0 and 0.1.0 hook schemas, although the 0.1.0 schema is deprecated.
+
+    For the annotation conditions, libpod uses any annotations set in the generated OCI configuration.
 
-The mounts.conf files specify volume mount directories that are automatically mounted inside containers when executing the `podman run` or `podman start` commands. When Podman runs in rootless mode, the file `$HOME/.config/containers/mounts.conf` is also used. Please refer to containers-mounts.conf(5) for further details.
+    For the bind-mount conditions, only mounts explicitly requested by the caller via `--volume` are considered.  Bind mounts that libpod inserts by default (e.g. `/dev/shm`) are not considered.
 
-**hook JSON** (`/usr/share/containers/oci/hooks.d/*.json`)
+    Hooks are not used when running in rootless mode.
 
-Each `*.json` file in `/usr/share/containers/oci/hooks.d` configures a hook for Podman containers.  For more details on the syntax of the JSON files and the semantics of hook injection, see `oci-hooks(5)`.
+**policy.json** (`/etc/containers/policy.json`)
 
-Podman and libpod currently support both the 1.0.0 and 0.1.0 hook schemas, although the 0.1.0 schema is deprecated.
+    Signature verification policy files are used to specify policy, e.g. trusted keys, applicable when deciding whether to accept an image, or individual signatures of that image, as valid.
 
-For the annotation conditions, libpod uses any annotations set in the generated OCI configuration.
+**registries.conf** (`/etc/containers/registries.conf`)
 
-For the bind-mount conditions, only mounts explicitly requested by the caller via `--volume` are considered.  Bind mounts that libpod inserts by default (e.g. `/dev/shm`) are not considered.
+    registries.conf is the configuration file which specifies which container registries should be consulted when completing image names which do not include a registry or domain portion.
 
-Hooks are not used when running in rootless mode.
+    When Podman runs in rootless mode, the file `$HOME/.config/containers/registries.conf` is used.
 
-**registries.conf** (`/etc/containers/registries.conf`)
+**storage.conf** (`/etc/containers/storage.conf`)
+
+    storage.conf is the storage configuration file for all tools using containers/storage
 
-registries.conf is the configuration file which specifies which container registries should be consulted when completing image names which do not include a registry or domain portion.
+    The storage configuration file specifies all of the available container storage options for tools using shared container storage.
 
-When Podman runs in rootless mode, the file `$HOME/.config/containers/registries.conf` is used.
+    When Podman runs in rootless mode, the file `$HOME/.config/containers/storage.conf` is also loaded.
 
 ## Rootless mode
 Podman can also be used as non-root user.  When podman runs in rootless mode, an user namespace is automatically created.
@@ -209,7 +213,7 @@ Currently it is not possible to create a network device, so rootless containers
 then only the loopback device will be available.
 
 ## SEE ALSO
-`oci-hooks(5)`, `containers-mounts.conf(5)`, `containers-registries.conf(5)`, `containers-storage.conf(5)`, `crio(8)`, `libpod.conf(5)`
+`containers-mounts.conf(5)`, `containers-registries.conf(5)`, `containers-storage.conf(5)`, `crio(8)`, `libpod.conf(5)`, `oci-hooks(5)`, `policy.json(5)`
 
 ## HISTORY
 Dec 2016, Originally compiled by Dan Walsh <dwalsh@redhat.com>
diff --git a/libpod/container_internal.go b/libpod/container_internal.go
index 79bc49c37..e5e871d6f 100644
--- a/libpod/container_internal.go
+++ b/libpod/container_internal.go
@@ -1246,7 +1246,7 @@ func (c *Container) saveSpec(spec *spec.Spec) error {
 }
 
 func (c *Container) setupOCIHooks(ctx context.Context, config *spec.Spec) (extensionStageHooks map[string][]spec.Hook, err error) {
-	if c.runtime.config.HooksDir == "" {
+	if len(c.runtime.config.HooksDir) == 0 {
 		return nil, nil
 	}
 
@@ -1277,16 +1277,25 @@ func (c *Container) setupOCIHooks(ctx context.Context, config *spec.Spec) (exten
 		}
 	}
 
-	manager, err := hooks.New(ctx, []string{c.runtime.config.HooksDir}, []string{"poststop"}, lang)
-	if err != nil {
-		if c.runtime.config.HooksDirNotExistFatal || !os.IsNotExist(err) {
+	var allHooks map[string][]spec.Hook
+	for _, hDir := range c.runtime.config.HooksDir {
+		manager, err := hooks.New(ctx, []string{hDir}, []string{"poststop"}, lang)
+		if err != nil {
+			if c.runtime.config.HooksDirNotExistFatal || !os.IsNotExist(err) {
+				return nil, err
+			}
+			logrus.Warnf("failed to load hooks: {}", err)
+			return nil, nil
+		}
+		hooks, err := manager.Hooks(config, c.Spec().Annotations, len(c.config.UserVolumes) > 0)
+		if err != nil {
 			return nil, err
 		}
-		logrus.Warnf("failed to load hooks: {}", err)
-		return nil, nil
+		for i, hook := range hooks {
+			allHooks[i] = hook
+		}
 	}
-
-	return manager.Hooks(config, c.Spec().Annotations, len(c.config.UserVolumes) > 0)
+	return allHooks, nil
 }
 
 // mount mounts the container's root filesystem
diff --git a/libpod/options.go b/libpod/options.go
index e6751d68d..1a29c0705 100644
--- a/libpod/options.go
+++ b/libpod/options.go
@@ -181,7 +181,7 @@ func WithStaticDir(dir string) RuntimeOption {
 // WithHooksDir sets the directory to look for OCI runtime hooks config.
 // Note we are not saving this in database, since this is really just for used
 // for testing.
-func WithHooksDir(hooksDir string, dirNotExistFatal bool) RuntimeOption {
+func WithHooksDir(hooksDir string) RuntimeOption {
 	return func(rt *Runtime) error {
 		if rt.valid {
 			return ErrRuntimeFinalized
@@ -191,8 +191,8 @@ func WithHooksDir(hooksDir string, dirNotExistFatal bool) RuntimeOption {
 			return errors.Wrap(ErrInvalidArg, "empty-string hook directories are not supported")
 		}
 
-		rt.config.HooksDir = hooksDir
-		rt.config.HooksDirNotExistFatal = dirNotExistFatal
+		rt.config.HooksDir = []string{hooksDir}
+		rt.config.HooksDirNotExistFatal = true
 		return nil
 	}
 }
diff --git a/libpod/runtime.go b/libpod/runtime.go
index 736169932..c69854a17 100644
--- a/libpod/runtime.go
+++ b/libpod/runtime.go
@@ -143,7 +143,7 @@ type RuntimeConfig struct {
 	// to attach pods to
 	CNIDefaultNetwork string `toml:"cni_default_network,omitempty"`
 	// HooksDir Path to the directory containing hooks configuration files
-	HooksDir string `toml:"hooks_dir"`
+	HooksDir []string `toml:"hooks_dir"`
 	// HooksDirNotExistFatal switches between fatal errors and non-fatal
 	// warnings if the configured HooksDir does not exist.
 	HooksDirNotExistFatal bool `toml:"hooks_dir_not_exist_fatal"`
@@ -199,7 +199,7 @@ var (
 			"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
 		},
 		CgroupManager:         SystemdCgroupsManager,
-		HooksDir:              hooks.DefaultDir,
+		HooksDir:              []string{hooks.DefaultDir, hooks.OverrideDir},
 		StaticDir:             filepath.Join(storage.DefaultStoreOptions.GraphRoot, "libpod"),
 		TmpDir:                "",
 		MaxLogSize:            -1,