What happened
On 14 Feb between 10:32:51PM and 11:05:51PM we had an outage on reps.mozilla.org
portal. The issue was triggered because of the way we handle RSS feeds for the mozilla portal. Posts from planet.mozillareps.org
are aggregated to reps.mozilla.org
homepage. In the past this was being done in the frontend but after PR 1304 we switched to feedparser
and started parsing RSS feeds in the backend. The outage was caused by planet.mozillareps.org
server being down.
Workaround
planet.mozillareps.org
server got back up after 30 mins so webops considered this issue resolved
Next steps
Improve reps.mozilla.org
homepage code to gracefully cache/timeout reps planet feed parsing.