This message was deleted.

do you use datadog? or prometheus?

If you want something more flexible https://keda.sh/ also helps with that

We use prometheus in our cluster, the approach @Philip Clark mentioned is using External metrics of load balancer to scale but AWS load balancer has 5mins delay in adding external metrics to cloudwatch from where we can scale and thus us unreliable. Is there any other wasy way we can achieve this?

I’ve head of keda, haven’t used it though. I think it all depends on how flexible of a solution you need, and if you want to maintain yet another kube service

I thought it was a simple problem that should have been solved already and a lot of people might be using this but i guess it is not🥲

Autoscaling is simple yes, but measuring network traffic on cloud resources outside Kubernetes as an input to autoscaling isn't built in (for good reason, Kube only knows what's going on inside it), so you need to rely on something outside the Kube to monitor that traffic. More often, people use a service mesh or some kind distributed tracing.

@Prashant Banthia Instead of using an ALB, why don't you use an NLB in front of an ingress controller like nginx or istio? Then you can use the instantaneous metrics as scraped by nginx/envoy proxies running in your cluster to power the HPA. And you reduce your costs by not having an LB dedicated per application.

All depends "The NLB does not support SNI, which means you can only associate one SSL/TLS certificate per NLB" so it has some limitations.

Not at all, because you wouldn't be using the NLB to handle SNI or redirection to any of your services. The NLB is just a dumb L4 LB that directs EVERYTHING at the nginx/envoy proxies inside your cluster. Those proxies are the ones handling SNI and routing, and if you terminate at the proxy, loads of L7 routing functions.

True that's a good point

We’re actually looking at using this pattern to get around the timeout limitations with nlb, seems kind of weird to have two ingress controllers (nginx and aws lb controller, but 🤷🏻‍♂️

Ya that's our current setup as well but we're moving to just albs ( and some cloudflare magic ) to make things easier

@Philip Clark Just see it as an L4 proxy load-balancing between your L7 proxies. AWS even forces you to do this outside of k8s if you're connecting API gateway to e.g. ECS containers. API gateway only supports linking to NLBs, which obviously have no path-based routing. You have to route the gateway to an NLB which then points to an ALB in order to route different paths to different containers lol. Thankfully the new HTTP API type in gateway allows directly routing to an ALB.

Yeah that’s a really good way of thinking about it