Hey all reps.mozilla.org friends!
Here is some context around the spamming incident we had with our email bot.
- Issue
- ReMo bot spammed users with hundreds of emails. We had various incidents but the last one was last week (18-23 Sept 2017).
- Root cause
- Celery bug: Tasks spawned with
etaargument backed by Redis are executed multiple times - https://github.com/celery/kombu/issues/337
- Celery bug: Tasks spawned with
- Context
- Spawning tasks to be executed in
etatime is an edge case in our codebase (4 calls out of ~100 tasks) - At first we tried the workaround celery maintainers suggest (change
visibility_timeoutconfig in our celery workers). Since we could reproduce the exact same behavior we were pretty confident that this would solve our problem. - After another round of spamming we realised that the bug was not fixed. It turned out that there was an issue on how celery parses the configuration. It uses a namespace and picks up only config entries starting with a specific prefix. Apparently our workaround wasn’t picked up by the workers.
- We fixed the namespace issue
- Verified everything on stage
- Pushed fix to prod
- No more spamming \o/
- Spawning tasks to be executed in
Excuse us for all that spam voting emails in your inbox 