I also have a few ideas on your questions @Ivo this too that are maybe helpful: 1. Very few. I work for Syntasso and we are building Kratix.io which is a FOSS orchestrator and actually one of the examples on the original Thoughtworks Tech Radar Blip. In addition to our product, I think that Crossplane often gets raised as an option here. I am a big fan of the reconcile loops with infra as code! My experience is that the ability to loop in older tech and processes (think

scp

style commands or manual approvals) requires a fairly high level of skill to write your own controllers for each integration. Both Kratix and Crossplane have an enterprise option also. For Kratix, even that is self hosted as what we provide is a composable framework rather than a single solution, so our customers (so far) prefer the security and flexibility of self hosting (including in air gapped environments!). 2. Platform teams. I will definitely unequivocally say platform teams. Platform teams own the interfaces between capabilities and users. a request for something comes in, how does it get delivered? Platform orchestrator. Platform teams very often have a whole heck of a lot of other responsibilities including capability implementation (think Terraform, Crossplane, Ansible, etc) and interface creation (think Backstage, CLIs, etc). But the orchestration is is squarely in that platform bucket. 3. You use the term IDP here, and I could read that as portal or platform. In regards to

portal

, I think that it is frequent platforms will provide some number of interfaces including possibly a portal, but part of the power of a platform orchestrator is that it can and should support any number of interfaces as it should expose all capability via APIs. In regards to

platform

, I think that platform and platform orchestrator can and should be interchangeable. Except many IDPlatforms today can’t support enough of a companies estate so they try to divide them. 4. I am interested in your perspective on my thoughts on #3 as I think this is tightly coupled. 5. Giving any implementation straight to a user has two major impacts: 1) it requires them to research and deeply understand the implementation and not just the use of it. And (quite possibly more importantly) 2) it reduces the flexibility to change implementation as needed. While Terraform is great, so was CFEngine before Puppet and Ansible took over. It is natural for technology to improve and we want to be able to use the right tool for the job which requires an interface between things. That being said, creating a thick layer that requires learning some crazy intense proprietary ideas is also hard. So it is about drawing that interface and contract in the right way which, to be fair, is the tricky part! 6. I am not sure what you mean by environment. If you mean test env (e.g. dev, staging, prod) I think it should span all of them, though you may choose to de-risk through blast radius instances etc). If you mean environment in regards to like company, then I think early stage companies can outsource their processes and DevEx to a 3rd party, but IME a company of sufficient scale / complexity ends up needing to customise 3rd party tools and they need a way to do this better.

ex. If i use TF and create a file with tfvars values i then ingest in my terrafrom code, I can use exactly the same tfvars file in PowerShell script that uses for example the AZ CLI to create the resources (yes i have to deal with inter-depedency between resources on create and remove). The problem starts with us no owning what the provider is doing or even its existence or support through its lifecycle

you can always take the imperative out of tf and into your pipeline. Its what i usually do with config values that need to happen after provisioning and do not belong in a state file

An exception would be your example of db migrations that have to be tied to same git commit / tag / build artifact

For me, using TF means git is no longer your source of truth, its actually the state file. It can become your source of truth if you use GitOps and infra managment is controlled via it.

Then the challenge is the upgrading of the pipelines people use across the org. But I know with gitlab you can reference shared pipelines and things so def doable.

And you don't even need to provide the devs any tfvars files, when you think about it, its just configuration values in some configuration database. Whether it is in a real database or saved as a json file is not relevant as long as your pipeline engine knows how to read it and extract the values. The only thing you have to remember is that any type of abstraction you place on top of a tech means the user has no idea how it works, how to debug, how to fix. If you have more than one tool across the company, it also makes it hard to pass knowledge between teams or when team members move between teams, they don't retain the knowledge of how to do things. And last is that when you recruit someone it is unlikely they had something similar, so expectations and learning curves.

IMHO, if you're creating a generic value files that could feed into a tf provider or a pipeline or any other technology you might want to implement the back end with, then you have an API. As Arie said, CRUD activities for that config file could end go through an API gateway so you could get those features (security, scaling, observability). Then, again IMO, whatever is taking that config file and managing it's execution, whether it be terraform, helm, a ci/cd pipeline etc is your orchestrator. The term is new, doesn't mean the concept is. The idea of "orchestrating" where and how to fulfill a platform request is something we have always done. Just sometimes it manual (via tickets etc) of via tools using different terms in their marketing (e.g. your use of tf)

In particular when you start talking to teams that are a bit more advanced in their processes, it can get super confusing because you have generated the kind of outcomes that these tools (including mine) are offering to help people with. And in time, these tools will grow up to the point where you see them as a commodity and you want to offload some of your custom internal glue. But until you see a 10x improvement, it is all just marketing noise and confusion. Many (MANY) teams aren’t yet at decoupling the vars from the code and not yet able to manage the resulting infra as a fleet due to how the creation process spawns outputs. In those cases, a new tool that is specialised in those capabilities can help.