This message was deleted.

if you are splitting up the monolith into smaller "containerable" services, you should concentrate on testing those services and only those services, as they are now independent of everything else that it was a part of in the monolith. now you can employ your previous strategy of e2e testing but on a much smaller scale. you can now decide if you want an overall integration test suite or a targeted unit testing approach, or both

they are now independent of everything else that it was a part of in the monolith

Not necessarily, most services depend on other services. And if a service consumes 5 other services, it starts getting unreasonable spinning all of them up in an isolated environment to run e2e tests (because then you basically have to spin up the entire transitive graph of dependencies). Mocks (possibly auto-generated) of the dependencies might alleviate some of that, but will only take us so far. What I'm starting to get inclined to is to do bulk of the testing in the form of unit and integration tests and enable the teams to be able to test their services in production.

Presumably services come together to form a product, there’s certainly a place for testing the interfaces your customers use in either a continuous or pre/post release gating fashion. Ideally against production. Test as your customers use. It’s not a replacement for unit or integration testing, but it’s valuable. Often times this is referred to as synthetic testing, smoketesting or e2e testing.

IMO once you reach the scale where microservices make sense; the only environment that behaves like production is… production. Thus, I’d advocate for investing in additional monitoring and alerting for the user journeys the e2e tests used to exercise (possibly with some additional synthetic traffic for important user flows that aren’t hit regularly) tldr; unit + API contract testing / microservice; replace e2e tests with production monitoring/synthetic traffic.

If "code in production" is the obvious consequence of breaking up the repository, then I think you should consider coming back to monorepo 🙂

The best approach I've seen to this problem is consumer-driven contract testing, most popularly implemented via Pact: https://pact.io/ . The idea is that the consumer of an API writes tests for its assumptions about the API provider's behavior, which are then run as part of the provider's test suite. The consumer has to update its expectations before the provider is allowed to change those aspects of its behavior. A stub server is used to represent the consumer in the provider's tests and the provider in the consumer's tests, so you don't need a full multi-service e2e test environment. The main problem is that there's a bit of a learning curve, so it takes a while for developers to wrap their brain around how/when to employ it effectively.

The other side to that coin is mitigate contract related issues by reliance on a spec (Proto, OpenAPI etc) The docs of pact make a good point though, the approach pact takes is incremental adoption whilst defining the API via a spec is all in and can be an investment for some (tooling to learn, enforcement to include in pipelines etc) Necessarily it depends on your engineer organisation. I’ve found the act of defining a proto/openapi spec ahead of time quite useful for the design phase of major initiatives like the one OP describes. Reliance on a spec could be a way to force that discussion to happen regardless of organisational culture.

Thanks folks, I also read about Pact in Building Microservices, it seems like a reasonable approach for distributed systems.

you could become a design partner for our auto-k8s AI workflows which can handle complex devops task and this problem becomes non-existent https://docs.zeus.fyi/docs/zK8s/intro extremely powerful ai control systems + auto-eval + api controlled k8s orchestration = no devops