From 9fad55c87f0ae6d2d01c77d8c67c5883ccbdbfde Mon Sep 17 00:00:00 2001 From: Ed Santiago Date: Thu, 23 Jul 2020 10:54:50 -0600 Subject: CI - various fixes Primary purpose: upgrade crun to 0.14 on f31, in hopes of eliminating the 'cgroups.freeze' flake that is plaguing CI. While I'm at it: - remove a no-longer-needed dnf upgrade that was running in CI itself (not image building, in each actual CI run). The purpose was to upgrade conmon, but that was added a long time ago and the required conmon is now in stable. The effect of this dnf upgrade today was simply to cause flakes when fedora repos were offline. - remove a no-longer-needed check for varlink. - networking.sh : add a timeout! 'openssl s_client' will happily hang forever if a host is unreachable, which means we waste two hours waiting for Cirrus to time out. - timestamp.awk : include date (not just time) in START/END msgs. There are times when I'm looking at a CI log and it is ultra important to know if it is from yesterday or today. - add progress messages in some places where I've previously struggled to understand context in logs; and improve some unlikely error messages to include script name. ...then, after all that, wrote a new README about how to to all this. Hope it helps someone. Signed-off-by: Ed Santiago --- .../cirrus/packer/README.how-to-update-cirrus-vms | 89 ++++++++++++++++++++++ contrib/cirrus/packer/fedora_packaging.sh | 9 +++ 2 files changed, 98 insertions(+) create mode 100644 contrib/cirrus/packer/README.how-to-update-cirrus-vms (limited to 'contrib/cirrus/packer') diff --git a/contrib/cirrus/packer/README.how-to-update-cirrus-vms b/contrib/cirrus/packer/README.how-to-update-cirrus-vms new file mode 100644 index 000000000..ac2902ffb --- /dev/null +++ b/contrib/cirrus/packer/README.how-to-update-cirrus-vms @@ -0,0 +1,89 @@ +This document briefly describes how to update VMs on Cirrus. + +Examples of when you need to do this: + + - to update crun, conmon, or some other package(s) + - to add and/or remove an OS (eg drop f31, add f33) + - to change system config (eg containers.conf or other /etc files) + - to change kernel command-line (boot time) options + +This is a TWO-STEP process: you need to submit a PR with a magic [CI:IMG] +description string, wait for it to finish, grab a magic string from the +results, then resubmit without [CI:IMG]. + +Procedure, Part One of Two: + + 1) Create a working branch: + + $ git co -b my_branch_name + + 2) Make your changes. Typically, zero or more of the following files: + + .cirrus.yml + contrib/cirrus/packer/*_packaging.sh + + I said zero because sometimes you just want to update VMs + with the latest in dnf or ubuntu repos. That doesn't require + changing anything here, simply running new dnf/apt installs. + + 3) Commit your changes. Be sure to include the magic [CI:IMG] string: + + $ git commit -asm'[CI:IMG] this is my commit message' + + 4) Submit your PR: + + $ gh pr create --fill --web + + + -------------------------- INTERMISSION -------------------------- + ...in which we wait for CI to turn green. In particular, although + we only really need 'test_build_cache_images' (45 minutes or so) + to get the required magic number strings, please be a decent + human being and wait for 'verify_test_built_images' (another hour) + so we can all have confidence in our process. Thank you. + -------------------------- INTERMISSION -------------------------- + + +Procedure, Part Two of Two: + + 1) When 'test_build_cache_images' completes, click it, then click + 'View more details on Cirrus CI', then expand the 'Run build_vm_image' + accordion. This gives you a garishly colorful display of lines. + Each color is a different VM. + + 2) Verify that each VM has the packages you require. (The garish log + doesn't actually list this for all packages, so you may need to + look in the 'verify_test_built_images' log for each individual + VM. Click the 'package_versions' accordion.) + + 3) At the bottom of this log you will see a block like: + + Builds finished. The artifacts of successful builds are: + ubuntu-19: A disk image was created: ubuntu-19-podman-6439450735542272 + fedora-31: A disk image was created: fedora-31-podman-6439450735542272 + ..... + + The long numbers at the end should (MUST!) be all identical. + + 4) Edit .cirrus.yml locally. Find '_BUILT_IMAGE_SUFFIX' near the + top. Copy that long number ("6439450735542272", above) and paste + it here, replacing the previous long number. + + 5) Wait for CI to turn green. I know you might have skipped that, + because 'test_build_cache_images' finishes long before 'verify', + and maybe you're in a hurry, but come on. Be responsible. + + 6) Edit the PR description in github: remove '[CI:IMG]' from the + title. Again, *in github*, in the web UI, use the 'Edit' button + at top right next to the PR title. Remove the '[CI:IMG]' string + from the PR title, press Save. If you forget to do this, the + VM-building steps will run again (taking a long time) but it + will be a waste of time. + + 7) Update your PR: + + $ git add .cirrus.yml (to get the new magic IMAGE_SUFFIX string) + $ git commit --amend (remove [CI:IMG] for consistency with 6) + $ git push --force + +You can probably take it from here. diff --git a/contrib/cirrus/packer/fedora_packaging.sh b/contrib/cirrus/packer/fedora_packaging.sh index b4a3a2062..f19932a9f 100644 --- a/contrib/cirrus/packer/fedora_packaging.sh +++ b/contrib/cirrus/packer/fedora_packaging.sh @@ -153,6 +153,15 @@ DOWNLOAD_PACKAGES=(\ echo "Installing general build/test dependencies for Fedora '$OS_RELEASE_VER'" $BIGTO ooe.sh $SUDO dnf install -y ${INSTALL_PACKAGES[@]} +# AD-HOC CODE FOR SPECIAL-CASE SITUATIONS! +# On 2020-07-23 we needed this code to upgrade crun on f31, a build +# that is not yet in stable. Since CI:IMG PRs are a two-step process, +# the key part is that we UN-COMMENT-THIS-OUT during the first step, +# then re-comment it on the second (once we have the built images). +# That way this will be dead code in future CI:IMG PRs but will +# serve as an example for anyone in a similar future situation. +# $BIGTO ooe.sh $SUDO dnf --enablerepo=updates-testing -y upgrade crun + [[ ${#REMOVE_PACKAGES[@]} -eq 0 ]] || \ $LILTO ooe.sh $SUDO dnf erase -y ${REMOVE_PACKAGES[@]} -- cgit v1.2.3-54-g00ecf