Skip to content

Conversation

@youwalther65
Copy link
Contributor

@youwalther65 youwalther65 commented Dec 5, 2025

Fixes #8738

Description

Karpenter supports only a subset of available K8s related Bottlerocket API attributes using a struct BottlerocketKubernetes as defined in pkg/providers/amifamily/bootstrap/bottlerocketsettings.go.

Currently if the EC2NodeClass contains an attribute in Bottlerocket userData which is not supported it will be silently ignored without any log info in Karpenter logs, see Karpenter docs here.

Karpenter uses the go-toml v2 which supports a strict mode to throw a toml.StrictMissingError error.

Using strict mode, EC2NodeClass validation is still returning status: "True" for condition type: Ready if an unknown attribute is used in spec.userData, but throws a log of "level":"ERROR" with details.

How was this change tested?
Yes, successfully tested using:

Prerequisite:

EC2NodeClass default with the following setting:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
...
  name: default
...
spec:
  amiFamily: Bottlerocket
...
  userData: |
    [settings.kubernetes]
    log-level = 2
    cpu-manager-policy = "static"
    unknown = "yes"
...

Run patched Karpenter locally with make run and got log entry:

{"level":"ERROR","time":"2025-12-08T13:42:58.699+0100","logger":"controller","caller":"bootstrap/bottlerocketsettings.go:31","message":"Unknown parameter in userData K8s settings","reason":"1| [settings.kubernetes]\n2| unknown = \"yes\"\n | ~~~~~~~ missing field\n3| log-level = 2","error":"strict mode: fields in the document are missing in the target struct"}

and corresponding EC2NodeClass stayed in Ready: true status:

$ kubectl get ec2nodeclass default 
NAME      READY     AGE
default   True   47d

@youwalther65
Copy link
Contributor Author

youwalther65 commented Dec 8, 2025

Last commit introduces backwards compatibility and just logs an error but proceeds as usual i.e. EC2NodeClass still stays Ready

Example:

EC2NodeClass definition

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
...
  name: default
...
spec:
...
  userData: |
    [settings.kubernetes]
    unknown = "yes"
    log-level = 2

logs error message in Karpenter logs like:

{"level":"ERROR","time":"2025-12-08T13:42:58.699+0100","logger":"controller","caller":"bootstrap/bottlerocketsettings.go:31","message":"Unknown parameter in userData K8s settings","reason":"1| [settings.kubernetes]\n2| unknown = \"yes\"\n | ~~~~~~~ missing field\n3| log-level = 2","error":"strict mode: fields in the document are missing in the target struct"}

and corresponding EC2NodeClass stayed in Ready: true status:

$ kubectl get ec2nodeclass default 
NAME      READY     AGE
default   True   47d

@youwalther65 youwalther65 changed the title Use go-toml v2 in strict mode to reject unknown Bottlerocket k8s parameters" Use go-toml v2 in strict mode to log ERROR for unknown Bottlerocket k8s parameters" Dec 8, 2025
Copy link
Contributor

@ryan-mist ryan-mist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter snapshot

@ryan-mist
Copy link
Contributor

LGTM. Can you also change the PR description to align with the new implementation?

@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-edcd4b44ec72a9abcf8f148247c969d67773f9fa.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-edcd4b44ec72a9abcf8f148247c969d67773f9fa" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

Preview deployment ready!

Preview URL: https://pr-8745.d18coufmbnnaag.amplifyapp.com

Built from commit edcd4b44ec72a9abcf8f148247c969d67773f9fa

@coveralls
Copy link

Pull Request Test Coverage Report for Build 20028523045

Details

  • 10 of 10 (100.0%) changed or added relevant lines in 1 file are covered.
  • 13 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.09%) to 67.563%

Files with Coverage Reduction New Missed Lines %
pkg/fake/iamapi.go 5 57.69%
pkg/providers/instanceprofile/instanceprofile.go 8 81.22%
Totals Coverage Status
Change from base Build 19946739564: -0.09%
Covered Lines: 7836
Relevant Lines: 11598

💛 - Coveralls

Copy link
Contributor

@ryan-mist ryan-mist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add log info when EC2NodeClass contains a Bottlerocket attribute which is not supported by Karpenter

3 participants