Heyo peeps! I'm brainstorming a process to streamline the app delivery experience in our future IDP and looking for a certain kind of tool, but let me start with the context and my thoughts so far: • The main product will be composed of several workloads (could range from monolith + a few other bricks to many microservices, but ideally this design choice shouldn't impact the experience) • hard requirement to build our own IDP from opensource and self host, no SaaS or cloud provider dependency possible (though the infra side is already solved, we have an internal private cloud and can spin new clusters on demand) • security/compliance and policy enforcement are major topics. One thing we would like is that the thing given to a cluster is a single CRD to centralize policy enforcement and RBAC hardening, instead of a bag of yaml rendered by helm/kustomize/[...]. • In line with the secure context, the road to prod needs to be extremely strict, with a strict separation of people between build and run roles (so no self deploy to prod, that's a regulatory requirement) • Of course, the fun challenge is combining all those constraints with a kickass developer experience, short feedback loops, maximum autonomy as long as it's not prod, all the things we love and that brought us here ❤️ The approach I'm thinking of: have a "fake prod" environment that represents the last step visible to developers (we called it "road to prod"), and once validated on it, the whole stack can then evolve as-is through the environments of the run team and eventually to prod (in contrast to previous environments where the individual versions of components in the stack can vary). So what I'm looking for: a tool with a kubernetes operator to ingest a high-level CRD representing a single workload or a versioned composition of workloads, app-centric enough so that developers can manipulate it without knowing anything about kube. But before mentioning oam and score, let me complicate the requirements 😄 • The CRD must specify exactly the version of each component, so that it's taken as-is with confidence past "road to prod" • As said before, this resource must be the final thing given to kube and cannot render plain yaml as part of a deploy process • On local workstations, the component being worked on can be substituted by any dev tool like tilt or devspace, and avoid the whole build/version feedback killer • I would like this resource to be the same one used for local inner loop hacking, self service dev environments, and prod • But! I'd also like this tool to be agnostic of the deployment workflow and not impose a specific process or operate with a push-based model (regulatory requirement forbids having a central "brain" that can reach both prod and non-prod environments). Instead, any plain old gitops tool could watch a repo with this resource and sync it on a cluster. I've looked at several products than could fit the bill, but they either render plain yaml (score) or are part of an orchestration machine incompatible with our requirements (kubevela). or negatively impact the inner loop of the developers. I'd love to hear your inputs on this problem and if you know any tool that could help. If such a tool doesn't exist we'd be glad to create and open source it! (at the very least our final workflow and reference architecture will be given back to the community, along with any tooling developed along the way)

Those are quite the requirements for your IDP. A lot yours align with mine except for the local development since I am looking at hybrid between kubernetes workloads and serverless workloads (functions/cloudrun). I have spent a bunch of time with kubevela and using crossplane to manage resource on the backend. The one thing I actually like from kubvela is that it just doesn't output yaml for a component/definiton etc. You can actually call out to rest APIs etc for deployments. The OAM application CRD is pretty robust at the moment for developers to defined their application with multiple components and dependencies between components. As well as defining complex workflow for the deployment. Depending on how a component is defined it can have a version input for the artifact it's going to deploy (container, tarball, terrform module version, anything). I also plan to use argocd for the gitops part of applying the application definition. Then to separate pre-prod from prod, I would just use. the technique of using seperate control clusters for each environment and seperate the application with argocd.

sorry for the late response! indeed, maybe I brushed off kubevela too quick, I was suspicious of the workflow part of the application spec, I'm still unclear regarding is the deployment path is a platform team or an appdev team concern

Hey both, this is the type of platform problem I love to dive into 🫶 It is also the type of platform problem that led to Kratix.io. First take a serving of 🧂 since I am on the core contributing team, and then hopefully you will hear me out 🤞 Kratix is all about enabling platform teams to build YOUR platform YOUR way. You can think of Kratix as a vending machine that you can then load up with the services you want to offer. Delivery of those services can include YAML wrangling but also business process following like security scans, manual approvals, budget checks etc. Basically we enable both declarative AND imperative commands. Where you are able to declare the outputs the only thing Kratix does is write those outputs to the correct state store, then it is up to your infrastructure how you apply it (we encourage gitops using Flux or Argo, but a CI/CD pipeline is also possible!). As @Kevin Scheunemann shared, Crossplane + KubeVela can do a lot. You can actually look at this talk from KubeCon to get an example: https://kccnceu2023.sched.com/event/1HyXP/building-a-platform-engineering-fabric-with-the-kube-api-at-autodesk-jesse-sanford-greg-haynes-autodesk While this can work, the idea from Kratix is to remove the wrangling and elevate this conversation to a first class citizen. I think Kratix framework may tick a lot of boxes and we are currently working with a small number of design partners to build out any missing pieces, so if you want to discuss what that looks like I am around 👍 Things you need / Kratix approach: • support an array of software architectures ✅ • opensource and self host ✅ • security/compliance and policy enforcement (continuous and as code!) ✅ • strict separation of people between build and run roles ✅ • kickass developer experience 👷 (this is the bit you get to build using Kratix!)

@Abby Bangser I have been definitely been following Kratix since last year when you did the Enlightning show with Whitney and the video from platformcon. I like the idea of kratix and promises however the OAM application abstraction just hits home for developers and creates I nice declarative abstraction between teams and platform. How does Kratix handle that? Also I have notices there really isn't a release of kratix yet. Do you have a timeline when you might a defined version to use?

thank you for your answer! Inded, kratix is something we've been eyeing, but we didn't think it was a candidate for the local dev aspect. Do you have users making use of kratix on a local (kind, k3s...) cluster on their workstation? It's entirely possible that this is a silly question, and that kratix is rather made for remote dev environments as a service.

Oh wow, both familiar is a great to hear! So first of all, Kratix is definitely in the “sandbox” style project. We release continuously with SHAs per commit so you an use at any time. That being said, we are in the process of a renaming right now to make the domain objects a bit more consumable (e.g.

xaasCrd

makes a lot more sense to be just simply called

api

). These changes should appear fairly soon. As for general roadmapping, we are currently working with a small group of design partners to identify the most valuable work which right now is looking like adding delete pipelines (ability to run commands when someone deletes a resource just like you can on create), and ability to add more organisation/permission controls with clearer namespacing and RBAC patterns. Other things that are in discussion are using Tekton for pipelines as well as better integrations to other technology like Terraform and Hashi suite. There isn’t a clear timeline on this is our core focus right now is getting these design partners successful with their use cases. For the specific challenges you two mentioned, I will try and share ideas but likely a bit of context lost in text so happy to chat further. • OAM: This model is, as you said, a good way for the application teams to define their apps. You may be interested in about how the team used Crossplane + KubeVela but still saw gaps in their delivery. Kratix fills those and more. • Local development: Kratix can run on any cluster including k3s and KinD. We use that as our development environment and as a quickstart to intro people to Kratix. But what I think you are actually asking is if people run Kratix on their developer boxes at their companies. I have not seen this, but given the right use case, it is very possible. • Kratix for Infra: Yes and no. Kratix is a framework which can enable your infrastructure provisioning and we see tools like Terraform, Cloudformation and Crossplane the most. The key is that Kratix makes “as-a-Service” a first class concern and manages the API between a service provider (the platform team) and consumer (the app teams) to enable scalable on demand offerings. Things like “test-environment-as-a-Service” is right now the most common use case. In some cases it includes a cluster (or vcluster, or namespace) and a number of baseline services (istio, cert manager, etc) and then a deployment process (e.g. configured gitops etc). Then the intention of that process rolling into long running environments and production environments as confidence grows with the process. If you think that you have a use case today and want to have a chat to see what it may look like to collaborate together on your goals happy to do that!