Hello,

We are experiencing a very specific situation with our Kubernetes cluster that results in an unusually high volume of network traffic generated by Flannel.

In our setup, approximately 250 nodes do not require inter-node & inter-pod communication. Only a subset of 5 nodes (with custom labels) need to exchange traffic between pods (such as ingress-nginx, prometheus, etc.)

Upon investigation, we observed that each node receives around 100KB/s of traffic from the API server. This traffic is processed by Flannel and appears to be related to the node watch functionality, as referenced in the code here:
https://github.com/flannel-io/flannel/blob/master/pkg/subnet/kube/kube.go#L185

We are using Kubespray with Flannel configured to use the UDP backend.

As a next step, we experimented with switching the Flannel backend to alloc, which seems to work well for the majority of nodes. However, we encountered two issues:

1. DNS Resolution Failure: Nodes are unable to resolve DNS. Kubespray configures nodelocaldns to forward DNS requests to coredns, but this becomes unreachable due to the lack of pod-to-pod communication.
2. Pod Reachability Across Nodes: The 5 nodes that require inter-pod communication (e.g., for ingress-nginx) are unable to reach pods on other nodes.

After reviewing the Flannel codebase, we are considering introducing an additional configuration flag to disable the node watch functionality — similar to how the alloc backend operates. However, we are uncertain about the safety and potential consequences of such a modification.

Would disabling the watch functionality be a viable and safe approach in our case?
Could this lead to inconsistencies or delayed updates in the network configuration?
Alternatively, would implementing periodic node polling (rather than continuous watching) be a more reliable solution?

We would greatly appreciate any insights, recommendations, or best practices you could share regarding this situation.

Thank you in advance for your support.

Best regards,
Tomasz