Skip to main content
Version: 1.35.2

Using an HTTP proxy

Nodes in a private network often reach the Internet through an HTTP proxy. SD configures the proxy on each node from the configuration file of the cluster.

note

The field is in a different place for each provider. The Immutable provider reads spec.infrastructure.proxy and applies it to the whole node. The OnPremises provider reads spec.kubernetes.proxy and applies it to containerd only. This page is about the Immutable provider.

Configuration​

spec:
  infrastructure:
    proxy:
      http: "http://proxy.example.dev:3128"
      https: "http://proxy.example.dev:3128"
      noProxy: "localhost,127.0.0.1,.example.dev,192.168.1.0/24,10.32.0.0/18,10.96.0.0/12,.svc,.cluster.local"

All three fields are optional. If you omit the proxy section, SD makes no change to the nodes.

warning

The scheme of the URL gives the transport to the proxy. It does not give the traffic that the proxy carries. Most proxies accept plain HTTP, so the https field usually also holds a http:// URL. A https:// URL makes each client start a TLS session with the proxy itself, and the client must then trust the certificate of the proxy. Use a https:// URL only when the proxy listens for TLS.

What uses the proxy​

furyctl writes the proxy variables for systemd and for login shells. Each service on the node inherits them:

  • containerd, for image pulls
  • kubelet
  • etcd
  • command line tools such as curl

What does not use the proxy​

The installation of the operating system does not use the proxy. The installer downloads the Flatcar image from the PXE server of furyctl, and it sends the status of the node to the same server. Each node must reach that server directly.

The download of the system extensions does not use the proxy at the creation of a node. Ignition gets them from the PXE server of furyctl, before the proxy configuration is in use. At an upgrade the node downloads them from the release server, and that download does use the proxy.

Exclude the addresses of the cluster​

The components of a cluster use TLS between them, with the CA of the cluster. A proxy must not carry this traffic. Add these addresses to noProxy:

  • localhost and 127.0.0.1
  • The network of the nodes.
  • The address of the furyctl server, from spec.infrastructure.ipxeServer.url. After the first boot each node sends its status to that server with curl, and that request uses the proxy.
  • The host of the control plane address, from spec.kubernetes.controlPlane.address, without the port.
  • The pod CIDR, from spec.kubernetes.networking.podCIDR.
  • The service CIDR, from spec.kubernetes.networking.serviceCIDR.
  • .svc and .cluster.local
warning

If an address of the cluster is absent from noProxy, kubelet tries to reach the API server through the proxy. A proxy that decrypts TLS then gives kubelet a certificate from a different CA. kubelet refuses that certificate, and the cluster does not start.

How noProxy entries match​

An entry matches the host as the URL writes it. There is no name resolution during the comparison. This is true for curl and for the Go programs on the node.

  • A CIDR entry such as 10.96.0.0/12 matches a URL that holds an IP address. A URL with a hostname that resolves into that range still uses the proxy, so add the hostname to the list too.
  • An entry with a port such as api.example.test:6443 never matches. Write the host without the port.

Proxies that decrypt TLS​

Many company proxies decrypt the TLS traffic and encrypt it again. Such a proxy replaces the certificate of each host with a certificate from a private CA. A node trusts only the public CA bundle of the Flatcar image.

containerd stops with this error when it pulls an image from a host that the proxy decrypts:

tls: failed to verify certificate: x509: certificate signed by unknown authority

The pod then stays in ImagePullBackOff. Each other program on the node gets the same result, because the cause is the trust store of the node and not containerd.

SD does not add a private CA certificate to the nodes. Use one of these three options:

  • Configure the proxy to send the traffic for the container registries through without decryption. This option needs no work on the nodes.
  • Use a registry mirror that the proxy does not decrypt, with spec.kubernetes.advanced.containerd.registryConfigs.
  • Add the CA certificate to the trust store of each node with your own tools, for example with ansible. The procedure is below.

Add the CA certificate to the nodes​

Copy the certificate in PEM format to /etc/ssl/certs/ on each node. Then build the CA bundle again and restart the services:

sudo cp corporate-proxy.pem /etc/ssl/certs/
sudo update-ca-certificates
sudo systemctl restart containerd kubelet

update-ca-certificates writes the bundle to /etc/ssl/certs/ca-certificates.crt. On a new node this path is a link to the read-only /usr partition. The command replaces the link with a file in /etc, so the image of the operating system stays unchanged.

The restart is necessary. containerd and kubelet read the CA bundle one time and keep it in memory. They do not use a new certificate before a restart.

warning

This procedure is outside the lifecycle of SD. A node that you add later does not have the certificate, and a node that you install again loses it. You keep the certificate on each node.