Forum Updates – Making the move from MySQL to MongoDB

Before I started working at Discogs, the team asked if there was anything I’d like to work on in particular. My response was the Forum. Finally my dream became a reality.

I spent the last two months rewriting the Forum from the ground up. Most of you may have only noticed some minor interface changes, but the backend has been completely redesigned. It is now easier for us to maintain, and add new features like allowing users to create polls, tag threads as questions or bugs and so on.

We have moved our Forum data from MySQL into a MongoDB replica set. Three of our major reasons for doing so:

1. Transitioning from a relational database to a document oriented database allows us to have a much more flexible data model. Most of the changes we have wanted to make to the Forum required schema changes which would have been much more work with our data in MySQL.

2. MongoDB has full text search capabilities out of the box. Every forum thread from over 12 years ago is now indexed and searchable. In fact, MongoDB now runs the blog, forum, and help documentation searches.

3. The Forum is now stored in it’s own dedicated “cluster” which allows for scalability via our N+1 architecture, and will not impact the rest of the site.

We are currently using Pymongo, which is the most popular way to interact with MongoDB using Python. We also use Mongoengine as an ORM to help give us more control over how to organize our documents.

Next on my list of things to tackle are Groups. Since they are so similar to the Forum, they will be rewritten as well using the same technology. I will follow some of the same patterns I used when building the Forum in order to simplify the codebase yet preserve the same functionality. There is a surprising amount of data in Groups, and I am hoping that making the same changes I did with the Forum will allow us to make them easier for users to communicate, collaborate, and seek out information. Make sure you’re watching the Development Forum for updates!