Subject:

Challenges in End-to-End Integration/App Testing

Problem Statement: I am currently leading a developer experience charter at a company with a development team of around 500 engineers. My team and I are responsible for improving developer experience and have developed an Internal Developer Platform (IDP) to enable the creation of ephemeral environments and service deployments. While the IDP has resolved many feature testing challenges, allowing service owners to independently deploy and test services in smaller groups, we continue to face significant challenges with end-to-end app testing (end to end flow through app): 1. Service Stability: Deploying a large number of services (e.g., 50) simultaneously in a testing environment often results in multiple failure scenarios, such as: ◦ Deployment issues due to network interruptions. ◦ Resource limitations. ◦ Failures caused by other unforeseen factors. 2. Test Data Consistency: Ensuring consistent and reliable test data across multiple services remains a bottleneck. For example: ◦ Services like the wallet service must have the user’s wallet correctly initialized for the user under test. ◦ Maintaining consistent test data across ephemeral environments is challenging and often leads to unreliable test outcomes. 3. Up-to-Date Service Definitions: Incremental changes to service code, database schemas, seed data, etc., are occasionally missed by service owners. This results in outdated service definitions, leading to issues in establishing a stable environment with fully functional business flows. Research and Insights: I’ve explored solutions to address these challenges and came across Uber’s approach of creating a sandbox environment. In their approach, the base environment mirrors production, and services with new feature code changes are deployed in a sandbox (referred to as a "slate"). Request for Input: Has anyone faced similar challenges and successfully resolved them? I’d love to hear about your experiences, insights, or any best practices you can share.

@Deepak Chougule If Uber’s approach is appealing, you may want to check out Signadot as it’s a more general/k8s-native implementation of Uber’s approach. (Full disclosure, i’m the founder). Thanks

Sounds to me lime typical problems to imperative implementation, I would look deeper into more declarative way of building stuff (K8s Operator Pattern)

Do you need to be deploying all those services to get confidence in the thing being deployed? It may be worth siloing off some of the more problematic services and using something like Consumer Driven Contract Testing so that services are coupled to an interface, not an implementation. Sometimes it's not a platform engineering problem but an architectural one.

The way I've handled this in the past is to maintain a "stable" collection of services alongside the dev/test deploys using service mesh config to isolate the test service(s) from the stable things. Requests to a test service would route to stable unless there was a test version in their namespace/network segment

the approach that @Michael Weinberg recommends is pretty neat if you have a very complex environment. What we’ve seen is that, for most cases, deploying an end-to-end environment with all the services that your test needs gets the job done, is more straightforward, and it requires less ‘magic’ (e.g. service mesh or other types of networking redirection). In my experience, kubernetes is much better than people anticipate at running a lot of services on a very small footprint.

For sure. The approach I described was an evolution from deploy all the services. YMMV on how easy that is in practice

@Shane Dowling Thanks for your response. In most of the cases, deploying all of the services and doing end to end testing is not required. But in case of a big feature releases, where multiple services are involved to complete a connected flow, in that case we need testing through app to make sure that end to end flow is working as expected.