I had a coworker cook up some ugly scripts to do essentially this exact thing (and we were on TF 0.11/0.12 at the time). We had created hundreds of google storage buckets in a single state file, along with IAM policies, service accounts, a bunch of crap (the oldest being bare resources, the newer ones contained in a module). We made some tactical errors in the early going with storage ACLs and IAM policies "fighting" over the bucket permissions, resulting in permadiffs and in the worst case, broken permissions after an apply when you changed something completely unrelated 😞 . Total mess and huge blast radius. The mitigation was to move each bucket to its own state. I don't recall how he did it, and I no longer work there so can't even give you the broad strokes without divulging any IP 😞 . But we used a Consul k/v state backend so he may have been pulling the state JSON and twiddling with it directly, not sure.

Honestly, when I hear about what's recommended by Hashicorp and what's not, I kind of just stop listening... They've done a tremendous job of writing a language that's impossible to implement at any kind of scale without some kind of anti-pattern so I take their recommendations with a grain of salt. I'd be willing to bet that the OP is in this mess in the first place because TF naturally tends to be a monolith.

My custom solution is indeed greping the output of terraform state list. I was wondering if there was not a better tool since this is presumably a common problem. Thank you all for you input.

For example, I can have a template that runs many workspaces and environments that codes for a VPC. Other templates/workspaces/environments may depend on this. However, running a change to the VPC template doesn't necessitate a change to the dependencies and we shouldn't have to run through the entire dependency graph. Figuring out ways to read changes from a producing environment and evaluate its impact on the dependent environment before running it is WAAAAY harder than it should be.

Of course, gotta be on TFC or (eventually) TFE

To be clear I haven't been hands-on with Stacks, we had just gotten in to the private preview when my employment ended 😐

The cross-state dependency thing is an interesting one. The way we implemented decoupled TF workspaces, they were each their own hermetic thing. Communication with other components was over a service mesh using some GCP shared DNS zones, and then we implemented something insane with Istio/Envoy (I didn't do the implementation of this part). But anyway, each TF org/workspace didn't really have any need to interact with the other orgs/workspaces. All separate VPC networks, GCP projects, etc.

Two features that are being worked on now that I'm especially excited about are: 1. dynamic providers (being able to declare the providers in a for loop, then refer to them and pass them dynamically to modules) 2. the ability to define module versions outside the module source definition

the ability to define module versions outside the module source definition

Do you mean like as a local or variable?

Also:

I think in my past we avoided that need by just having lots of mini-monolith state files in our huge monolith repo lolsob

Seriously, I will give no shade here. Managing TF at scale is terrible and you have to do some shitty things to get it done. Its just a matter of what shitty thing can you live with.

Now time for an afternoon full of interviews