For the experienced folks here, While using auto scaling in k8s, which metrics did you look at in real world scenarios? I know CPU, memory etc. are the usual ones to look at. But, did you ever encounter a situation where these metrics did not help when scaling was actually needed. I am inquisitive post a recent conversation where a team is using "oldest message in message queue" as a metric for scaling pods. would like to hear more such stories.

We had a need that required scaling on a specific custom Prom metric which was exposed by the Pod(API) v1.20 of K8s supported it in a buggy manner but as of v1.22 we were all good

Yes, we use 'oldest message in queue' or 'queue latency' to scale background processing workers. It works really well, and lets us scale down to zero for things that don't run often.