Support for parallel processing of SQSEvents with multiple records

[jiw-mh]

When looking at the queue code, all operations happen in sync.

https://github.com/brefphp/bref/blob/2c8ec120d0d4779ec4b5890276cbbd21c85711cf/src/FunctionRuntime/Main.php#L45C25-L45C38

Having support for asynchronous code could reduce our lambda cost quite a bit https://reactphp.org particularly when processing SQS entries.

[mnapoli]

@jiw-mh what do you mean by that? How would costs be reduced? I think there's some misunderstanding on how Lambda works.

[jiw-mh]

sorry for the late reply 🙇 - The issue is indeed a bit confused (me re-thinking it) and I am not sure how to best approach it, but indeed: react php might be the wrong approach.

The laravel bridge will automatically executes all records of an event in series: https://github.com/brefphp/laravel-bridge/blob/997296994ddf1c931b85a3cc34346404e35e1d60/src/Queue/QueueHandler.php#L65-L78

I imagine that bref could identify if the queue is a FIFO or parallel queue and in case it is a parallel queue split up the SQS event into one event per record and then execute the items in parallel in separate threads on the machine. There could be an environmental flag for enabling this.

[mnapoli]

Ah ok I see. So configure "batch" handling of SQS records, and then in a single invocation your PHP code would process each record of the batch in parallel.

The only thing you would gain here is on costs. But not on speed. AWS Lambda is made to process stuff in parallel, so instead of batching you could "not batch" and have all those records processed in parallel by Lambda.

I.e. instead of 1 batch of a 1000 records processed serially in 1 invocation, you could have 1000 records processed in parallel in 1000 invocations.

[jiw-mh]

The things that get executed for each queue item are HTTP requests. If each record is executed in a separate thread running in parallel, the waiting for the HTTP response would mean that most cpu cycles are idle. This should reduce the total execution time by a lot.

[deleugpn]

If each record is executed in a separate thread running in parallel [...]

If each record is executed in a separate Lambda Function running in parallel...

When you produce individual SQS messages, each message will go to one instance of a Lambda function and they're all executed in parallel. So the reduced total execution time you're looking for will be achieved the same way.

[jiw-mh]

When you produce individual SQS messages, each message will go to one instance of a Lambda function and they're all executed in parallel. So the reduced total execution time you're looking for will be achieved the same way.

Each Lambda instance will cost money. I prefer to pay less. 😓

[mnapoli]

In that case you can already do that today:

• configure Lambda to handle batches (e.g. 1000 records in 1 invocation)
• in your code, parallelize HTTP requests with Amp, ReactPHP, Laravel's HTTP client, Symfony's HTTP client, Guzzle, etc.

I've done that already, works well. You don't need Bref to do some async, only our code.

[jiw-mh]

Laravel Queue Jobs (not http requests) do not distinguish between parallel/serial execution and leave this up to the configuration. The SQS workers in regular Laravel SQS implementations parallelize by the number of workers. In a lambda execution environment the obvious way I can see is to re-implement the Laravel bridge's QueueHandler that starts a thread for every record and in each thread runs a LaravelWorker with a single record. Since the LaravelJobs are expecting to be run in series. Of course, depending on the queue settings, I would need to check if that particular queue is FIFO or not (for a FIFO I would use a single thread, even for multiple records).

I could implement this myself, but this seems like an expected behavior for bref that would benefit everyone, which is why I posted this issue. That said, I initially thought of posting it in the Laravel-bridge repository but after consideration I think it might be something that all bridges need and that maybe the SQSHandler could just abstract this away for all implementations?

[mnapoli]

Ahhh ok. So if we're talking specifically about Laravel, Laravel doesn't support handling multiple jobs in the same worker process.

And 1 Lambda invocation = 1 process. Theoretically you could run multiple processes on the machine, but I don't see the point. Instead:

• either keep Laravel jobs and just have more Lambda workers run in parallel 🤷
• or if you want to optimize things, don't use Laravel jobs and instead work with SQS natively (and then you can do parallel stuff)

What is the scale of your jobs? What is the SQS + Lambda costs? Are this $1k or $10k per month?

If we're talking theoretically then it's a moot discussion. Just let Lambda scale and be done with it.

[jiw-mh]

We did have a bit of a stumble and it did cost us a good amount last month, we needed to move some parts around to avoid the same cost this month, but it is high enough for me to need to rethink it since we spent a lot of money on cpu idling 😅 The biggest part to me was that I didn't notice the serial processing and assumed it would use the idle time in a different thread. My bad, but it might be worth putting in docs. "Work with SQS events natively" creates a custom code that will cause effort to maintain. Running them in parallel (separate threads) seems like the way to try both this way and - as you suggest - to just implement queues without laravel. Maybe Laravel want's to implement an optionally async interface in the queue specification.