This is obviously not the right way of doing that, as we can't change defaults.

The question is how exactly we should do that:

• we seem to already have v1 API, but that don't even have any defaulting:
  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/apis/config/v1
• given the config is already in v1 - what options do we have to change it?

I'm not sure this is so terrible in a config type?

Do you want to remove the limit or increase it a lot? At 1k you might as well remove it completely?

In that case an alternative could be to add a "UseRateLimit" field and default it to false?

I'd get kubelet authors to weigh in first.

I don’t know why we wouldn’t go to 25 or 50 as a first step, then a release later with soak jump another bit. I struggle to think of kubelet even on 512 pod nodes being able to saturate 50qps anyway

I struggle to think of kubelet even on 512 pod nodes being able to saturate 50qps anyway

For extended period of time - I agree. As a spike - I actually can - let's say some large pod just finished there and we have a place to start next new small 30 pods, with 10 secrets each. That gives O(400) API calls...

That said - I would be fine with doing it in steps, if it's not a big overhead like creating a new config version to avoid changing defaults within a single (group, version) or sth.

When I looked at this for EKS, I found that I got most of the benefit just by doubling the QPS/burst. We've used 10 QPS 20 Burst as the default kubelet values for EKS AMIs since K8s 1.22. I'm happy to see the defaults increased.

The below graph is 3k pods and 30 nodes showing time to pod readiness.

The other situation where it helps is in auto-scaling. E.g. New pods are created, causing new nodes to be launched and large numbers of pods then schedule to the same node at roughly the same time as it goes ready.

Refs:

• increase the kube-api-server QPS from 5/10 to 10/20 awslabs/amazon-eks-ami#1030
• k8s: increase the kube-api-server QPS from 5/10 to 10/20 bottlerocket-os/bottlerocket#2436

At 1k you might as well remove it completely?

I think having a safeguard at a very high, but not extremely high value would protect API server from errors in kubelet. Virtually unlimited normal operation sounds reasonable to me.

@wojtek-t when I looked yesterday I thought the limit increased to 100/500. I might not remember correctly since I don't se force push there =). For me 100/500 was the good first step as @smarterclayton suggested.

One risk of 1k is potentially we can make kubelet to become a noisy neighbor in the moment of rescheduling (like @wojtek-t described in #116121 (comment) with 400 requests). It may be a problem when there are other API server clients on the node or network is very limited and close to be saturated... But we are keeping configuration settings and for set ups with limited network connectivity one can change defaults back.

I'd vote for 100/500 as a first step in 1.27.

This seems better than --use-rate-limit that is false by default because at least if someone previously specified these flags they would continue to work as the author expected.

I think the question for sig-node is more about whether we can change the default at all. And to us (api-machinery) about how confident we are in disabling the client-side rate limiter. We (redhat) have not yet tried disabling the rate limiter for kubelets.

@wojtek-t when I looked yesterday I thought the limit increased to 100/500. I might not remember correctly since I don't se force push there =). For me 100/500 was the good first step as @smarterclayton suggested.

I didn't change that in the meantime.
FWIW - Clayton suggested even smaller, but for now I switched to 50/100 - if we're ok with changing the defaults on the config level - this isn't much work so it's for sure ok to go gradually..

One risk of 1k is potentially we can make kubelet to become a noisy neighbor

Agree - but the fact that we're allowing clients to send requests, doesn't mean that kube-apiserver won't reject them anyway. So it's on kube-apiserver to decide whether it has capacity to process it, and APF should do the job here.

This seems better than --use-rate-limit that is false by default because at least if someone previously specified these flags they would continue to work as the author expected.

Agree - this is why I also started with this WIP PR.

And to us (api-machinery) about how confident we are in disabling the client-side rate limiter. We (redhat) have not yet tried disabling the rate limiter for kubelets.

As long as I don't have full confidence in other components, I think we're pretty confident in bumping Kubelets at this point from Google side.

@wojtek-t when I looked yesterday I thought the limit increased to 100/500. I might not remember correctly since I don't se force push there =). For me 100/500 was the good first step as @smarterclayton suggested.

I didn't change that in the meantime.
FWIW - Clayton suggested even smaller, but for now I switched to 50/100 - if we're ok with changing the defaults on the config level - this isn't much work so it's for sure ok to go gradually..

One risk of 1k is potentially we can make kubelet to become a noisy neighbor

Agree - but the fact that we're allowing clients to send requests, doesn't mean that kube-apiserver won't reject them anyway. So it's on kube-apiserver to decide whether it has capacity to process it, and APF should do the job here.

This seems better than --use-rate-limit that is false by default because at least if someone previously specified these flags they would continue to work as the author expected.

Agree - this is why I also started with this WIP PR.

And to us (api-machinery) about how confident we are in disabling the client-side rate limiter. We (redhat) have not yet tried disabling the rate limiter for kubelets.

As long as I don't have full confidence in other components, I think we're pretty confident in bumping Kubelets at this point from Google side.