Kubelet Streaming Connection Idle Timeout Not Set To 0

Manual Steps

On every worker node, back up the existing kubelet config file:

sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak.$(date +%F-%H%M%S)

On every worker node, edit /var/lib/kubelet/config.yaml and set a non‑zero timeout (example: 5 minutes). If the key exists and is 0, change it; if it does not exist, add it under the top‑level config:
sudo sed -i 's/^[[:space:]]*streamingConnectionIdleTimeout: *0[[:space:]]*$/streamingConnectionIdleTimeout: 5m/' /var/lib/kubelet/config.yaml
If the line does not already exist, open the file with an editor and add, at top level (aligned with other keys like authentication / authorization):
streamingConnectionIdleTimeout: 5m
If your kubelet also uses command-line flags via systemd drop‑in, ensure it does not override this to 0. On every worker node, open the systemd drop‑in file:
sudo sed -n '1,160p' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
If you see --streaming-connection-idle-timeout=0 anywhere, edit the file with a text editor and change it to:
--streaming-connection-idle-timeout=5m
On every worker node, reload systemd and restart kubelet (this will temporarily disrupt kubelet on that node):
sudo systemctl daemon-reload sudo systemctl restart kubelet.service
On every worker node, verify kubelet is running and the timeout is no longer 0:
/bin/ps -fC kubelet
Confirm that either the kubelet command line does not contain --streaming-connection-idle-timeout=0 (and if present, shows a non‑zero value like 5m), and that /var/lib/kubelet/config.yaml contains streamingConnectionIdleTimeout: 5m (or another non‑zero duration).

Using kubectl

kubectl cannot modify kubelet process flags or its config file at /var/lib/kubelet/config.yaml, so this setting cannot be fixed via the Kubernetes API. Apply the remediation directly on every worker node’s host configuration as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Fix kubelet streamingConnectionIdleTimeout on all worker nodes.
#
# Usage:
#   1) Place this script on each worker node and run as root, OR
#   2) Run centrally with SSH access to each worker node:
#        WORKER_NODES="node1 node2" ./fix-kubelet-timeout.sh ssh
#
# This script is idempotent and safe to re-run.

set -euo pipefail

STREAM_TIMEOUT_VALUE="5m"
KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

mode="${1:-local}"   # 'local' (default) or 'ssh'
workers="${WORKER_NODES:-}"

run_local() {
  echo "=== Running on worker node: $(hostname) ==="

  # 1. Ensure kubelet config file has streamingConnectionIdleTimeout != 0
  if [ -f "$KUBELET_CONFIG" ]; then
    echo "Updating $KUBELET_CONFIG ..."

    # If key exists, replace it; otherwise, add under top-level.
    if grep -qE '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG"; then
      # If already set to desired non-zero value, do nothing
      if grep -qE "^[[:space:]]*streamingConnectionIdleTimeout:[[:space:]]*${STREAM_TIMEOUT_VALUE}[[:space:]]*$" "$KUBELET_CONFIG"; then
        echo "  streamingConnectionIdleTimeout already set to ${STREAM_TIMEOUT_VALUE}"
      else
        # Replace any existing value (including 0 or other) with desired non-zero
        sed -i -E "s/^[[:space:]]*streamingConnectionIdleTimeout:.*/streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}/" "$KUBELET_CONFIG"
        echo "  Set streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
      fi
    else
      # Add key if missing
      printf "\nstreamingConnectionIdleTimeout: %s\n" "$STREAM_TIMEOUT_VALUE" >> "$KUBELET_CONFIG"
      echo "  Added streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
    fi
  else
    echo "WARNING: $KUBELET_CONFIG not found; skipping file-based config."
  fi

  # 2. Ensure systemd drop-in (command-line args) has non-zero timeout if used
  if [ -f "$SYSTEMD_DROPIN" ]; then
    echo "Checking $SYSTEMD_DROPIN for command-line timeout flag ..."

    if grep -q -- '--streaming-connection-idle-timeout' "$SYSTEMD_DROPIN"; then
      # If already desired value, do nothing
      if grep -q -- "--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}" "$SYSTEMD_DROPIN"; then
        echo "  --streaming-connection-idle-timeout already set to ${STREAM_TIMEOUT_VALUE}"
      else
        # Replace any existing value (including 0 or other)
        sed -i -E "s/--streaming-connection-idle-timeout=[^[:space:]]*/--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}/" "$SYSTEMD_DROPIN"
        echo "  Updated --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
      fi
    else
      # Append flag to existing KUBELET_SYSTEM_PODS_ARGS or Environment line if present
      if grep -q 'KUBELET_SYSTEM_PODS_ARGS' "$SYSTEMD_DROPIN"; then
        sed -i -E "s/(KUBELET_SYSTEM_PODS_ARGS=\"[^\"]*)\"/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
        echo "  Appended flag to KUBELET_SYSTEM_PODS_ARGS"
      elif grep -q '^Environment=' "$SYSTEMD_DROPIN"; then
        sed -i -E "s/^(Environment=.*)\"$/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
        echo "  Appended flag to Environment line"
      else
        cat <<EOF >> "$SYSTEMD_DROPIN"

Environment="KUBELET_SYSTEM_PODS_ARGS=--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
EOF
        echo "  Added new Environment line with timeout flag"
      fi
    fi
  else
    echo "INFO: $SYSTEMD_DROPIN not found; kubelet may be fully config-file driven."
  fi

  # 3. Reload and restart kubelet (operational impact: kubelet restart on this node)
  echo "Reloading systemd and restarting kubelet ..."
  systemctl daemon-reload
  systemctl restart kubelet.service

  # 4. Verification: inspect kubelet process args to ensure timeout is not 0
  echo "Verification: kubelet process arguments on $(hostname):"
  /bin/ps -fC kubelet || true

  # Fail if we still see an explicit '=0'
  if /bin/ps -fC kubelet 2>/dev/null | grep -q -- '--streaming-connection-idle-timeout=0'; then
    echo "ERROR: kubelet still running with --streaming-connection-idle-timeout=0 on $(hostname)" >&2
    exit 1
  fi

  echo "=== Completed on $(hostname) ==="
}

run_ssh() {
  if [ -z "$workers" ]; then
    echo "ERROR: In ssh mode, set WORKER_NODES=\"node1 node2\" environment variable." >&2
    exit 1
  fi

  for node in $workers; do
    echo "##### Processing worker node: $node #####"
    ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$node" 'bash -s' <<'EOF'
set -euo pipefail

STREAM_TIMEOUT_VALUE="5m"
KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

echo "=== Running on worker node: $(hostname) ==="

if [ -f "$KUBELET_CONFIG" ]; then
  echo "Updating $KUBELET_CONFIG ..."
  if grep -qE '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG"; then
    if grep -qE "^[[:space:]]*streamingConnectionIdleTimeout:[[:space:]]*${STREAM_TIMEOUT_VALUE}[[:space:]]*$" "$KUBELET_CONFIG"; then
      echo "  streamingConnectionIdleTimeout already set to ${STREAM_TIMEOUT_VALUE}"
    else
      sed -i -E "s/^[[:space:]]*streamingConnectionIdleTimeout:.*/streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}/" "$KUBELET_CONFIG"
      echo "  Set streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
    fi
  else
    printf "\nstreamingConnectionIdleTimeout: %s\n" "$STREAM_TIMEOUT_VALUE" >> "$KUBELET_CONFIG"
    echo "  Added streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
  fi
else
  echo "WARNING: $KUBELET_CONFIG not found; skipping file-based config."
fi

if [ -f "$SYSTEMD_DROPIN" ]; then
  echo "Checking $SYSTEMD_DROPIN for command-line timeout flag ..."
  if grep -q -- '--streaming-connection-idle-timeout' "$SYSTEMD_DROPIN"; then
    if grep -q -- "--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}" "$SYSTEMD_DROPIN"; then
      echo "  --streaming-connection-idle-timeout already set to ${STREAM_TIMEOUT_VALUE}"
    else
      sed -i -E "s/--streaming-connection-idle-timeout=[^[:space:]]*/--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}/" "$SYSTEMD_DROPIN"
      echo "  Updated --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
    fi
  else
    if grep -q 'KUBELET_SYSTEM_PODS_ARGS' "$SYSTEMD_DROPIN"; then
      sed -i -E "s/(KUBELET_SYSTEM_PODS_ARGS=\"[^\"]*)\"/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
      echo "  Appended flag to KUBELET_SYSTEM_PODS_ARGS"
    elif grep -q '^Environment=' "$SYSTEMD_DROPIN"; then
      sed -i -E "s/^(Environment=.*)\"$/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
      echo "  Appended flag to Environment line"
    else
      cat <<EOS >> "$SYSTEMD_DROPIN"

Environment="KUBELET_SYSTEM_PODS_ARGS=--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
EOS
      echo "  Added new Environment line with timeout flag"
    fi
  fi
else
  echo "INFO: $SYSTEMD_DROPIN not found; kubelet may be fully config-file driven."
fi

echo "Reloading systemd and restarting kubelet ..."
systemctl daemon-reload
systemctl restart kubelet.service

echo "Verification: kubelet process arguments on $(hostname):"
/bin/ps -fC kubelet || true

if /bin/ps -fC kubelet 2>/dev/null | grep -q -- '--streaming-connection-idle-timeout=0'; then
  echo "ERROR: kubelet still running with --streaming-connection-idle-timeout=0 on $(hostname)" >&2
  exit 1
fi

echo "=== Completed on $(hostname) ==="
EOF
  done
}

if [ "$mode" = "local" ]; then
  run_local
elif [ "$mode" = "ssh" ]; then
  run_ssh
else
  echo "Usage: $0 [local|ssh]" >&2
  exit 1
fi

Kubernetes

Kubernetes Misconfigurations

Kubelet Streaming Connection Idle Timeout Not Set To 0

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation