Another case of "x509: certificate signed by unknown authority"

[bmbeverst]

Version

system-upgrade-controller version v0.15.2 (b1a9781)

Platform/Architecture

linux-amd64 Flatcar Container Linux stable 4152.2.2 for QEMU (immutable)

Describe the bug

I0324 02:10:05.643137       1 event.go:389] "Event occurred" object="system-upgrade/plan-k3s-server-upgrade" fieldPath="" kind="Plan" apiVersion="upgrade.cattle.io/v1" type="Warning" reason="ResolveFailed" message="Failed to resolve latest version from Spec.Channel: Get \"https://update.k3s.io/v1-release/channels/stable\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

system-upgrade-controller cannot upgrade the k3s cluster, do the TLS verify failure.

To Reproduce

Instead of a version use channel as below:
channel: https://update.k3s.io/v1-release/channels/stable

Expected behavior

Expect that the SSL certificate can be verified from the container.

Actual behavior

The SSL configuration is unable to verify the certificate and can't connect.

I can connect with curl, but I only have one symlink file. There is no /etc/pki or /etc/tls. Any assistance would be appreciated!

I have one syslinked file:

$ ls -l /etc/ssl/
drwxr-xr-x. 1 root root 14388 Feb 13 18:28 certs
-rw-r--r--. 1 root root   241 Feb 13 18:26 fipsmodule.cnf
lrwxrwxrwx. 1 root root    31 Feb 13 16:59 openssl.cnf -> ../../usr/share/ssl/openssl.cnf
drwx------. 1 root root     0 Feb 13 18:36 private

I tried adding a file mount

        - mountPath: /usr/share/ssl/openssl.cnf
          name: usr-ssl
          readOnly: true
 . . . 
      - hostPath:
          path: /etc/ssl/openssl.cnf
          type: File
        name: usr-ssl

curl

curl https://update.k3s.io/v1-release/channels/stable
<a href="https://github.com/k3s-io/k3s/releases/tag/v1.31.6+k3s1">Found</a>.

Additional context

The bug was reported multiple times in different constellations:

• Error: WithLatestTag(): invalid tag format (on rancher/k3s-upgrade) #94
• error getting latest release because "x509: certificate signed by unknown authority" #105
• Update RKE2 v1.24.12+rke2r1 #243
• Yet another case of "x509: certificate signed by unknown authority, requeuing" #278

I was unable to access the container to see what files are actually in the /etc/ssl/ folder.

I was unable to figure out the required files for the container to configure SSL. Why doesn't the container come with working SSL?

[oakey-dev]

you actually need /usr/share/ca-certificates too

the diff of my change

diff --git a/01-system-upgrade-controller.yaml b/01-system-upgrade-controller.yaml
index d8da40c..9b7d08c 100644
--- a/01-system-upgrade-controller.yaml
+++ b/01-system-upgrade-controller.yaml
@@ -281,12 +281,18 @@ spec:
         - mountPath: /etc/ssl
           name: etc-ssl
           readOnly: true
+        - mountPath: /usr/share/ssl
+          name: usr-share-ssl
+          readOnly: true
         - mountPath: /etc/pki
           name: etc-pki
           readOnly: true
         - mountPath: /etc/ca-certificates
           name: etc-ca-certificates
           readOnly: true
+        - mountPath: /usr/share/ca-certificates
+          name: usr-share-ca-certificates
+          readOnly: true
         - mountPath: /tmp
           name: tmp
       serviceAccountName: system-upgrade
@@ -310,6 +316,10 @@ spec:
           path: /etc/ssl
           type: DirectoryOrCreate
         name: etc-ssl
+      - hostPath:
+          path: /usr/share/ssl
+          type: DirectoryOrCreate
+        name: usr-share-ssl
       - hostPath:
           path: /etc/pki
           type: DirectoryOrCreate
@@ -318,5 +328,9 @@ spec:
           path: /etc/ca-certificates
           type: DirectoryOrCreate
         name: etc-ca-certificates
+      - hostPath:
+          path: /usr/share/ca-certificates
+          type: DirectoryOrCreate
+        name: usr-share-ca-certificates
       - emptyDir: {}
         name: tmp

now it started updating

and thanks for the hint