Post Header
Mini-release notes: Battling the 502s errors and responding to user feedback
Our coders and sys-admins have been working hard to deal with the performance issues we've been experiencing over the last few weeks. Releases 0.8.15 and 0.8.16, only a few days after deploying 0.8.14, see the introduction of several tweaks to our code and server setup that should help alleviate the site slowness we addressed in our recent post, AO3 performance issues.
New code by andreja, Ariana, Elz, Enigel, and Naomi. Tested by Elz, Enigel, Jenn, Kylie, Sarken.
Performance fixes
As noted on our Known Issues page, one of the major bottlenecks were tag listings for very popular fandoms, which would either load very slowly or throw up 502 errors. We've added caching for the first five pages of results - these will expire when a work using that tag is posted or revised, so the listings will still be up to date.
Another significant bottleneck were the bookmarks listings, both the main one and the bookmarks on tags. We've simplified the functionality, as we look into ways of reworking the bookmarks for a much better performance.
With the help of New Relic, a web service that monitors and analyzes site performance in great detail, we found another source of slowness: works with a large amount of (guest) kudos. Showing an updated kudos count for every work as it was accessed was putting an undue strain on the server, so for now the number of kudos on a work will be cached (i.e. not fetched from the database in real time) and only updated every five minutes.
As per a user suggestion (thank you, fydyan), we also looked into ways to prioritize certain user actions over others, so that trying to post a work or a comment would be less likely to throw up an error (potentially taking all input with it) than, say, accessing a user profile or browsing the site. Many thanks to Sidra from Systems for implementing this!
Invitation emails
We had a bug which was preventing notification emails for invitations being sent. We've fixed this bug, and the emails which were affected have been resent. This may result in some people receiving their invitations twice - invitations can only be used once, so please take note of the invitation code and don't pass the extra email along to a friend if it's a code you've already used!
Plans for our next few deploys
We're continuing to work hard on performance fixes and will soon be implementing much more caching across the site. We'll be posting with more details about this shortly.
We've had lots of feedback about our recent changes to notification emails. Unfortunately, we cannot roll back these changes for performance reasons (for more details see our post on Email changes and USER STATS!). However, based on feedback we've received so far, we will be adjusted the way subscription emails are batched and labeled, hopefully in the next deploy. Thanks for bearing with us while we work to improve this!
Details
- Bug fixes:
- Invite emails were not being sent; this has now been fixed and the delayed ones resent.
- The work count on the front page used to vary depending on whether the user was logged in; it now shows the total of all published works.
- Previewing a work before posting made the wordcount not show; this has been fixed.
- Improvements:
- Kudos counts on works are now cached, but will change when new kudos is added by logged in users. Guest kudos counts will update every five minutes. Thus it will cause less of a load on the database.
- Works listings under tags are now cached up to the fifth page. The listings will update when a work with that tag is posted or revised.
- We've removed the grouping of bookmarks by work. Also, before it was trying to get the bookmarks using tag synonyms, or bookmarks that were not using the tag directly. The main bookmarks listing now grabs the most recent public bookmarks on the site, and the tag listings show the bookmarks that are tagged directly with the requested tag.
- Text on the collections page has been changed to clarify what characters are allowed in collection names.
- Small change to text on the tag edit pages: the label for the synonym field was changed to "Choose an existing tag or add a new tag name here to create a new canonical and make this tag its synonym." for the sake of clarity.