Hey everyone! I'm new to platform engineering and was hoping to get some ideas here on how to incorporate the ideas into my company because it looks like exactly what we want to setup. We are a software contracting company and often times end up making Kubernetes deployments that we end up handing off to our customers. We deploy air-gapped (no internet) which makes things harder... so often times we have a lot of repeated work such as setting up and configuring bare-metal clusters, deployment configuration, selecting helm charts, CI/CD pipelines, etc.. I've been seeing a ton of ideas on platform engineering but wanted to understand how people are actually implementing platform engineering, especially since most 'code' is kept in a team's git repository. Take for example CI/CD or Helm charts. These are files that live within a team's git repository, so how is an IDP or platform helping to provide these things to a team without getting someone with the relevant experience to manually spend time and help out with development and integration of these resources? Would development teams essentially just be copying code into their repo based on suggested "templates" that the IDP is offering?

My experience is with AWS & Azure and both will allow you to integrate your GitHub to allow CI/CD directly from the Git. Also, you can set up templates that will allow folks with basic knowledge to deploy.

often times end up making Kubernetes deployments that we end up handing off to our customers

What customer role is expected to own these deployments? Ops? Upgrades? Patches? Ongoing security and compliance?

We deploy air-gapped (no internet)

How do you get your bits in there in the first place, not just from a technical perspective but from a customer security process perspective? And can you specify what you mean by bare metal?

Would development teams essentially just be copying code into their repo based on suggested "templates" that the IDP is offering?

Maybe, but after you leave, who will keep those up to date? Or modify them as needed? Based on "without getting someone with the relevant experience to manually spend time and help out with development and integration of these resources?" this sounds like setting a customer up to fail over the long haul, or get swamped with technical debt over time and end up 4 years of releases behind the times.

Yeah, I will say it's definitely not perfect.

What customer role is expected to own these deployments?

Generally it's a more technical role in our customer, but typically they don't know how to use Kubernetes... They ask us to supply them with kubernetes deployments but have no idea how to maintain a cluster themselves... it's a bit weird I will admit.

How do you get your bits in there in the first place

Our dev environments have internet, but deploying to our customer environment requires deploying k8s with no internet. We typically use a provided VPN that gets us into an environment with no internet. We could reverse-proxy an internet in from our host machine but that would be frowned upon... lol. Currently we're doing a lot of no-internet deploys which involce manually downloading docker images, tarballs, rpm packages, etc. for deployment... it's painful.

Maybe, but after you leave, who will keep those up to date?

Yeah that is the main reason we want to go to an IDP. With an IDP we could have our platform team manage, say, a helm chart, and if they release a new version or change how it's deployed, teams could be alerted with something like "Hey, there's a new update to PostgresDB which your app is using" for example? But agreed, there is a lot of problems we need to solve here lol

A full fledged IDP with no internet access and therefore no full-cycle feedback loop is likely an antipattern here. You'd end up building YAFD (Yet Another F&%$in' Dependency) inside of a customer's environment that someone would have to own and manage, without getting a lot of benefit from doing so. Swatting flies with a bazooka. It sounds like your team is deploy and pop smoke, but on a cadence. Is that cadence weekly, monthly, quarterly? Customer has a use-it-or-lose-it bucket of hours? That might inform how to approach the problem. A potential option: • Depending on how dynamic their environment and build dependencies are, help the customer inventory their portfolio and keep it straight. If they keep their helm charts and other artifacts in consistent places, what keeps you from polling that for your recommendations? • Build up automation on your side of the house that aligns with that cadence. If the dev environments are at reasonable parity with their production environments, there's not much stopping you from building tooling that takes the polled information from their various repos, upgrades packages as necessary, builds any artifacts and validates at whatever security and compliance levels they require, and outputs a big pile of bits to drag into their prod environment over VPN. Additional benefits are that you've also ◦ built their DR plan and turned Mean Time To Repair into Mean Time To Repave ◦ built tooling to deploy (with modifications as necessary) into any environment ◦ reduced burden on their developers How big is their stable of developers?

> It sounds like your team is deploy and pop smoke Pretty much. We make the software, and disappear unless they sign another contract with us for support or fixes. > How big is their stable of developers? Probably fairly large but the real question is: Do they have time to understand and maintain k8s clusters they didn't provision? Usually the answer is no, because our clusters usually include databases, message queues, etc... basically nothing is cloud provided/managed. Honestly, a lot of this is probably on our customer for not wanting to use the cloud and being stuck in the stone age. But I think that my idea is not to build an IDP for each customer, but rather build an IDP for our internal developer teams so that internally we can streamline the process of creating & assembling apps for our customers (almost all of which tend to be air-gapped). Excuse my crappy drawing, but this is sort of what I imagined. So our tooling and development processes all have internet, we just lose it once we deploy. For example, most of our customers require kubernetes, databases (usually), Kafka is usually a common ask, etc...

look at https//managed.delivery from netflix it describes so much of this better than IDPs do

I get it, that makes a lot more sense now. If you aren't already, you could conceivably sell a subscription for updates based on that model as well. If they're the kind of customers who don't mind incremental line items and who have the sorts of security and compliance requirements that make them airgap environments, then you're well positioned to say, "Oh, by the way, we'll help you keep it patched and secure for $pile_of_money_per_annum." IDP may still not be the best solution, central repos of automation with per-customer forks (to include IaaS paving) may be a better way to go.

history > howiedidit.sh

is a shockingly effective place to start for that kind of effort 🤣

yeah

I think the question of "Do I use Spinnaker, $some_other_tool, or roll-your-own-repos?" will be answered by complexity and expected scale. How alike do those customer environments look? How many of them are there/will there be? How similar are the apps we're building and deploying for them?

Hey Quinn! Here’s a recording from OpenShift Commons this year - side event of KubeconEU. Bechtle, one of the largest European SIs is tackling pretty much the challenges you describe. The architecture depicted in the recording can be freely reconfigured with components from the platform tooling landscape you can find as part of this community. Happy to have a chat on it if you have any questions.