This message was deleted.

Appreciate my stack may not be the most cutting edge but we are tightly constrained in a large highly regulated enterprise

We are using a mix of Terraform and Helm for standing up our kubernetes clusters (EKS) that come provisioned with a bunch of cluster-scoped services.

I'm a strong proponent of Pulumi for greenfield infra projects

Terraform here to set up kubernetes and other bits in GCP

Tf for cluster creation for me and gitlab pipelines with helm for apps

Hi 🙂 We are currently using Terraform to provision the clusters(but considering to switch to Pulumi soon. We use ArgoCD with Helm to deploy the applications and are really happy with it.

TF+ArgoCD and a mix of Helm charts for 3rd party apps and kustomize for in-house services

@John Fox Hi John! A few things that may help you: 1. Have you seen the Humanitec’s Platform Orchestrator yet? a. You can use it (as a glue) to transform your existing IDP (Internal Developer Platform aka. your platforming stack) into an enterprise-grade IDP. It also enables the standardisation by design, allowing you to pre-bake security standards into dev self-service paths, etc. 2. We’ve recently released the reference architectures for Azure (image below) 3. Humanitec is also optimised for GitOps workflows. Given your requirements I think it would make sense to chat - I’ll message you privately.

Maybe @Mickaël Gentil can share his xp

Thanks @Romaric Philogène 🤜 🤛 @John Fox We are using Qovery for our IDP after searching and testing similar SaaS tools. We would like an easy to go platform both for devops controls and for developers adoption and DevXperience with complete GItOPS workflow. We are a mid size company and we don't have resources time and money to create our own IDP based on Platform Orchestrator like Humanitec, which is by the way a very great platform for that. In one week we had our Developers completely autonomous. Qovery allow us to quickly and easily deploy our services like a breeze on AWS Cloud. They manage K8 clusters with their engine (the funny part? in production we d'ont have K8, we are Serverless!) and permit us to generate Preview environment on Github PR directly accessible by our Q&A after Slack and Jira tickets status change. We let behind us all the hard work to focus on what the IDP nature is for: Simplicity, Centralized Catalog, Autonomy of our devs to deliver continuous code deployment but letting them to create their own POC (thanks to the RBAC systems). All is never perfect with any tools, but the quality of the platform, the simplicity and the wonder team always responding is a guarantee of quality. And they are definitely focused on the customer centric approach. We always participate to what should be next features by voting adoption. I don't say other IDP are not great. Just our experience with Qovery. I can answer to any questions.

Thanks for all the updates everyone. Much appreciated. It does feel like the route we are going down is pushing the boundaries of the tools we are using (and work to be fair). It's just layering a lot of custom organisational uniqueness on top of the tools/patterns.

Have you looked at Crossplane? If so, what led you not to use it?

HI @Kief Morris 👋 To be clear, Crossplane is a very good tool! One of the balance to go with Qovery was the very simplicity to use (KISS). Their K8 engine take care of all. We don't manage it. No need to install with helm or using kubectl. All is GitOPs. We create only Docker files into our repository, or using our own AWS ECR, or the Public Docker Registry, depends on our workflow. No need to another yaml/json or whatever schema to learn. If we want to use kubectl we can because Qovery build the K8 cluster on AWS EKS., but, We don't have the needs ^^ And with their Terraform provider all is IAAC if you need (I know crossplane have official too now by upbound). The Gtihub PR preview environment creation is a must to perform Q&A on branches for example (idk if it's native with crossplane). We are trying to keep all things KISS even for DevOPS.

crossplane is more of a "I want to go off and build a platform", qovery and some others is more of a batteries included but with a lot of escape hatches if needed for custom stuff

I mainly mentioned crossplane because it sounded like John is using AKS with a fair bit of DIY. My impression of Crossplane is that it's a lot like AKS + extra stuff, so I wondered if it would be a logical progression in his situation.

ahh gotcha

I can imagine it might not be if, for example, the extra stuff that Crossplane adds (compositions and such) don't actually replace the DIY stuff

Crossplane sounds like it is the next step for us to explore. Held fire on additional tooling at the moment and working within a large financial services org and very highly regulated and tricky to get new stuff in. Also there's a number of org bespoke tools that have been built, they are complicated/obscure enough. Feels like we are naturally hitting the boundaries of our diy solution and need to look for further help

I have found the hard part is transitioning the stuff that makes money to any new systems. IMO if you can’t see a migration path of existing solutions / tech to the new ones it will really struggle to take off

Agreed. I'm being really harsh and insisting this platform is for greenfield stuff (clusters certainly)

As far as I know for crossplane paid you get a nice UI and a lot of support from upbound I think thats pretty much it, for qovery its a managed service ( although the core engine is open source but unless you got a highly technical team it would be hard to use that by itself )

We tried Crossplane a year ago and struggle to make it work properly. We found it to be sometimes a bit aggressive with the resources it managed (for example terminating an instance just to change a tag...). For these reasons, we switched to Pulumi which is more stable and a better fit for our use-case. We are on the path of building things ourselves right now but curious to see what the managed platforms can offer.

When you say building yourself Vs a managed offering, what bits of work/tech/challenges do you wish you could offload to a managed platform? And which bits are making you hesitate to try one? I've never used a managed platform for more than a PoC but have definitely had pain managing template drift on my custom platforms 😕

For the bits I would like to offload: • UI -> we don’t have the skills in the team • access management -> having sound and compliant access management is always a struggle Also I see that as a way to speed up the implementation of our platform as building by ourselves takes time and we have limited resources. For the bits that make me hesitate: • integration -> when building our platform we are trying to glue together a lot of different technologies to create the golden paths for product teams. It isn’t only about provisioning the cloud resources that teams need but also making sure those are instrumented (logs in our Elasticsearch, metrics in our Prometheus, dashboards and alerts in Grafana, ...) and compliant with our internal policies (tagging for cost reporting, vulnerability scanning, ...) • customisation -> we support a lot of different product teams that have various ways of working (some are data scientists, others are working on customer projects that are deployed on the premise of the customer, some are building SaaS products, some on internal services, ...). We need a different approach for each of these teams. What do you mean by template drift @Abby Bangser?

FYI Qovery can be used with Crossplane 🙂

At $lastjob we had helm templates and terraform modules that would enable teams to make their services themselves. But once a team pulled that code into their repos it was hard to then progress it and keep things in alignment.

@Abby Bangser what do you mean by “keep things in alignment”?

We had something like 50 service repos that all used a helm chart, makefiles, github action files, and terraform modules that they depended on. When we needed to change something (introduce a new option, update the way we deployed, patch a security issue, etc) it should have been as easy as updating the pieces, but in reality everyone had crafted them to their use cases and it meant each change required a lot of faff. This is part of why I have been saying “code isn’t an API” a lot recently. And yes that includes helm charts and terraform modules 😅

Super interesting Abby. Thanks for the details. I’d love to learn more about your setup and see how you are solving/not solving those issues and why. If you have 30 minutes in the next days. I’d love to dig deeper.

As I mentioned, it is $lastjob so maybe less relevant right now. But loving this thread and learning a lot with people here!

Sounds like what I've been calling "Snowflakes as Code". People copy and customize templates and code, and before long, you've got 50 service repos, or 4 environments, or whatever, that all ought to be pretty much the same thing with some configuration differences, but are all tweaked to the point where making a change across all of them is a huge hassle.

Definitely. Does feel like we are heading down that road where things are so complex and difficult to maintain/track/understand

IMO much of the problem in this space has been driven by reinvention of the wheel. Early on, folks definitely needed to invent new ways to deliver IaC, but so so much of it should have been commoditized by now.

Agreed Craig, the challenges now are on how to use commodity infra (clouds and SaaS options) while still supporting the (sometimes frustrating or crazy seeming) internal requirements. When I solved that before it was the “snowflake as code” style as Kief described. Now I am looking at ways that internal teams can mimic the layers of value with API interactions just as clouds have. Allowing the internal platforms to continue to progress without impacting users (just as the clouds have).

I think a big part of the problem is that there are no (widely adopted) tools, or even conventions, for building IaC. Managing multiple projects, setting configuration values, etc. - these are things that every team creates their own in-house scripts to do. There are some tools out there, but most still seem to be designed to support custom development.

@Abby Bangser 💯 I was just having the same conversation with someone yesterday: through all of this IaC work over the past decade, we have spent very little time (dare I say, almost none?) on the systems software interfaces.

I think a big part of the problem is that there are no (widely adopted) tools, or even conventions, for building IaC.

can there ever be on though? I mean, any company around for a reasonable amount of time/making a reasonable amount of money is on a number of different infras and architectures and everything. Do we want to build a tool that can IaC a mainframe and serverless? 🤔

There is a tension when it comes to standardized interfaces though...how will

$vendor

differentiate if the interface is a standard?

on feature merit? 🧌😂

@Abby Bangser those are very broad interfaces you describe. I am speaking more to components of a software stack. The average developer will spend non-trivial amounts of time trying to navigate "which tool?" instead of just writing the business logic.

do you have another example that can help me make sure I understand? Are you more thinking what Dapr provides with app level “talk to db, I don’t care what that means” style?

In fact "which tool?" is how this thread started. And there are probably half a dozen (or more) patterns here 🙂

can there ever be on though? I mean, any company around for a reasonable amount of time/making a reasonable amount of money is on a number of different infras and architectures and everything. Do we want to build a tool that can IaC a mainframe and serverless? 🤔

I'm not thinking of the IaC tools themselves, but all the scripts, makefiles, and whatnot that teams use to glue their IaC projects together, configure environments and applications, etc.

Nice, I got you now. I do think there are scenarios where it does matter, but there are many many scenarios where it doesn’t. The trick is how to start at the high level and then allow pulling back if/when it matters, all in a smooth experience

Abby, you remember those Makefiles on that project ...

but kief, it would all have been better if we had just used Rake instead!

Oh, man, a project or two after that one I set up a build with Rake, then came back a few months later and the line count for the Rakefiles and related code were on par with the actual Terraform code

Related to this conversation - I just scheduled a new event on how to combine Crossplane with Qovery and Kubernetes and the benefits for platform engineering and engineering teams.