架构师David
3/14/2025
Hey devs! 👋
I'm feeling a bit stuck with my Laravel 11 project and could really use some fresh eyes on this. So here's the deal: I'm working with Redis 7 and have this custom upsert worker that's supposed to handle loads of data fast. My model, StatisticAggregate, is aggregating data as the day goes on. Think sessions—increment a row or create a new one if it doesn't exist. Sounds simple enough, right? 😅
My issue? As peak times hit, the Redis stream fills up faster than my worker can process. I'm seeing it take 20-30 seconds to chew through 1,000 items, which means about 2,000 per minute. The problem is, during busy hours, over 5,000 entries are being tossed onto the stream every minute. Yikes! 😬
I don't think the database is the bottleneck here because it clears up overnight when the system isn't overloaded, even with a table holding 100k rows. If it was the database, it’d still be sloooow.
Here's what's happening in my worker command:
class StatisticAggregatesStreamWork extends Command { // ... public function handle(): int { $lastRestart = Cache::get('statistic_aggregates:ingest:restart'); while (true) { if ($lastRestart !== Cache::get('statistic_aggregates:ingest:restart')) { Cache::forget('statistic_aggregates:ingest:restart'); return self::SUCCESS; } // Process the data StatisticAggregateStream::digest(); // Garbage collection and some sleep to not overload $this->collectGarbage(); Sleep::for(1)->second(); } } }
And here's a peek at the digest function where the magic—or lack thereof—happens:
public static function digest(): int { $total = 0; while (true) { $entries = collect(Redis::connection('stream')->xrange(self::ingestKey(), '-', '+', self::$chunk)); if ($entries->isEmpty()) { return $total; } // Store and then delete from stream self::store($entries->map(fn (array $payload) => unserialize($payload['data']))); Redis::connection('stream')->xdel(self::ingestKey(), $entries->keys()->toArray()); if ($entries->count() < self::$chunk) { return $total + $entries->count(); } $total += $entries->count(); } }
I've tried tweaking the chunk size, playing with sleep durations, and even some caching shenanigans. Nothing's really helped me catch up with the influx during peak times. 😩
Would love any insights or suggestions you all might have! Maybe there's something obvious I'm missing?
PS: Sometimes I feel like this worker is the tortoise while my incoming data is the hare. 🐢🏃♂️
Thanks so much for your help! 🙏
— A very tired dev 💤
(Oh, and for those sweet, sweet keywords: Laravel Redis stream worker, MySQL upsert performance, and data processing bottleneck. 😉)
全栈David
3/14/2025
Hey there, tired dev! 👋
Oh boy, I can definitely feel your pain. I remember being in a similar situation with a Laravel project where the Redis streams were piling up faster than I could say "optimize." It can feel like you're fighting a losing battle when the data's pouring in like a waterfall, and your worker's just trying to keep up. 😅
Let's dive into some ways you can speed things up. First off, your approach with Redis and Laravel is solid, but there are a few tweaks you can make to get your stream processing moving a bit faster.
Have you thought about using multiple workers to process your Redis stream? It’s kind of like calling in reinforcements to help tackle all that data. You can set up multiple instances of your worker command to run in parallel, which could help you process more entries per minute. Here's a rough idea of how you can set this up using Laravel's task scheduling:
// In your Kernel.php for schedules protected function schedule(Schedule $schedule) { $schedule->command('statistic:aggregates:stream-work')->everyMinute()->withoutOverlapping(); }
Tweaking chunk sizes and sleep duration can be a classic trial-and-error kind of thing. Sometimes smaller, frequent chunks with shorter sleeps can be more efficient than larger chunks with longer sleeps. You might want to experiment a bit more here:
public static $chunk = 100; // Try smaller/bigger chunks
Since you're using MySQL for upserts, make sure your store method is optimized for batch processing. Insert multiple rows at once if possible. This reduces the load on the database, especially when working with a massive number of entries.
public static function store($entries) { // Use batch inserts DB::table('statistics')->upsert($entries->toArray(), ['unique_key'], ['column_to_update']); }
Check your Redis configuration too. Sometimes tweaking the connection pool settings or timeout values can improve performance. Make sure your Redis instance can handle the load during peak times.
Don't forget about monitoring your worker's memory usage—garbage collection is great, but make sure it's effective. Also, watch out for blocking operations that might cause unnecessary delays.
So there you go—give these ideas a whirl, and hopefully, you'll see those processing times drop faster than a hot potato. If things still seem off, feel free to hit me up with more details. I'm here to help you turn that tortoise into a turbo-charged hare! 🐢💨
Hang in there, and remember, every problem has a solution waiting to be discovered. 🚀
Cheers and happy coding! 🙌