This message was deleted.

Effectively what I want to do is: - receive a webhook - process that webhook (i.e. new

push

to the default branch of that repo) - trigger an event - event invokes a short-lived worker, in a container, which: - takes a few configuration items from the event - downloads the code for that repo - runs a process against that codebase - write the result to a database Has anyone done similar before? Not tied to Cloud Providers or anything yet 😄

Hm… what stops you from doing it in a single application as opposed to having an app which receives the webhook and a worker?

One main thing is that I want to split the codebase into two - the worker is going to be Typescript, to use existing libraries, whereas everything else will be Go. I'd also like to be able to scale them independently, as from experience, the traffic will be different, and the workers require much more resources wise than the webhooks

Fair enough. Other than that… it’s really up to you and the constraints you may or may not have: • What’s the expected traffic? • What’s the desired reliability/is it ok for it to be down short amounts of time? • Who’s going to maintain it? • What tech are you familiar with and do you have a reason why you would want to deviate from that? • What’s the desired build/ship timeline?

Thanks, some great questions 🙌 So a little more info - this is planned to be a SAAS platform I've previously built a version of this, using EventBridge and Lambda (but SQS would've also been a valid option). This had ~1000 repos'

push

events sent to the app, and then processed similarly to mentioned above.

What’s the expected traffic?

Unfortunately don't have any numbers to hand, but I seem to think it was roughtly 30-60 requests a second sounds about right.

What’s the desired reliability/is it ok for it to be down short amounts of time?

I'd prefer for the webhook layer to be more resilient, and at least allow processing the events, but if the workers are down for a little while, that'd be OK

Who’s going to maintain it?

One person team, me

What tech are you familiar with and do you have a reason why you would want to deviate from that?

Go, AWS, some Typescript. Typescript required for the worker process, would prefer to use Go if possible, but not necessarily tied to it.

What’s the desired build/ship timeline?

This is a side project, so pretty flexible with timelines.

From the top of my head, you could do this very easily on top of AWS with lambdas and SQS.

Yeah I was thinking this too, and it tracks with a previous implementation. From a previous version I've built on this, I'm not sure I'll be able to use Lambda, as I was hitting the 15 minute invocation limit...

Another consideration – do you want the worker to be downloading a repository every time? That’s a somewhat expensive operation which can be cached by making the workers not short-lived.

... because we were downloading very large Git repos, with their full history. The idea is that we'd either something like a

git clone --depth 1

, or download a ZIP of the code, which would remove a lot of the lookup needed. Short-lived is also because we were hitting disk space issues after multiple runs execute.

And one last thing that comes to my mind, can there be concurrency issues? Is it ok if two workers do an op on the repository simultaneously or should operations always be linear? You’ll have to deal with conflicts if not.

The idea is that only the default branch of a repo is going to be processed, so if it's a repo with lots of regular pushes to the default branch - that may be pushed quicker than the scan can execute - we may lead to some overwriting. It'd be prefered if that didn't happen, but not so bad if the results aren't quite right However, it'd probably be good to allow pruning unnecessary worker processing if possible, just to save work.

You could use event bridge event to kick off a code pipeline that can run longer than 15 minutes.

The pattern you describe works well, it’s worth answering if you might want to replay the webhooks you receive more than once.

I'd say no, ideally we'd only process a duplicate webhook event once

Yeah only once processing semantics is one thing, but I was trying to indicate that their could be situations where you want to reprocess old webhooks due to some code change or issue on your end. Webhooks are only emitted at time of change of course. Just something I see come up with webhook driven systems sometimes.

Oh yes, I wasn't sure if you were thinking that too - then yeah absolutely, I think it'd be good to be able to replay / retry old events especially if there's downtime / you want to reprocess it