Forum Updates – Making the move from MySQL to MongoDB
Before I started working at Discogs, the team asked if there was anything I’d like to work on in particular. My response was the Forum. Finally my dream became a reality.
I spent the last two months rewriting the Forum from the ground up. Most of you may have only noticed some minor interface changes, but the backend has been completely redesigned. It is now easier for us to maintain, and add new features like allowing users to create polls, tag threads as questions or bugs and so on.
We have moved our Forum data from MySQL into a MongoDB replica set. Three of our major reasons for doing so:
-
Transitioning from a relational database to a document oriented database allows us to have a much more flexible data model. Most of the changes we have wanted to make to the Forum required schema changes which would have been much more work with our data in MySQL.
-
MongoDB has full text search capabilities out of the box. Every forum thread from over 12 years ago is now indexed and searchable. In fact, MongoDB now runs the blog, forum, and help documentation searches.
-
The Forum is now stored in it’s own dedicated “cluster” which allows for scalability via our N+1 architecture, and will not impact the rest of the site.
We are currently using Pymongo, which is the most popular way to interact with MongoDB using Python. We also use Mongoengine as an ORM to help give us more control over how to organize our documents.
Next on my list of things to tackle are Groups. Since they are so similar to the Forum, they will be rewritten as well using the same technology. I will follow some of the same patterns I used when building the Forum in order to simplify the codebase yet preserve the same functionality. There is a surprising amount of data in Groups, and I am hoping that making the same changes I did with the Forum will allow us to make them easier for users to communicate, collaborate, and seek out information. Make sure you’re watching the Development Forum for updates!
Not familiar with MongoDB but here is a possible resource.
All the things is discussed properly.
Hope it will help… :-)
Visit Here https://zappysys.com/blog/ssis-loading-data-into-mongodb-upsert-update-delete-insert/
Not familiar with MongoDB but here is a possible resource.
All the things is discussed properly.
Hope it will help… :-)
Visit Here https://zappysys.com/blog/ssis-loading-data-into-mongodb-upsert-update-delete-insert/
I am Not familiar with MongoDB but my friend suggest me this for understand all about MongoDB
https://zappysys.com/blog/ssis-loading-data-into-mongodb-upsert-update-delete-insert/
I also faced the same problem related to MongoDB then one of my friend suggest me a good resource for better understanding of mongoDB http://goo.gl/zMgzbC and http://docs.mongodb.org/manual/tutorial/
@pianoman74, yes a groups rewrite is in our sights. It will be similar to the Forum rewrite and will have much more relevant search results!
Just one more question to you guys:
I was jumping for joy when you mentioned the User Groups. So are you also planning to rewrite the backend for them, so that searches like e. g. http://www.discogs.com/groups/search/9?s=vocal+house&btn= will finally work again instead of spitting out 9- to 4-year-old posts but none of the current ones. Oddly enough, ‘house music’ instead of ‘vocal house’ works. It has been behaving that strange for several years now.
nice one ! :)
depends on what you need
ACID vs BASE ! :D
Don’t think we’ve given up on MySQL – it’s still very much our core datastore. But when it comes to performing text searches, MySQL was not cutting it. This may change in MySQL 5.6, when full-text search is supported with the InnoDB engine. But with simple “docs”, such as forum, groups, and blogs, storing them as JSON in MongoDB is more flexible then trying to do the same in MySQL. Horses for courses.
No “bad press” if it outperforms the old situation in speed and ease of maintainance… I’m sure as long as you don’t need complex aggregate queries MongoDB will do fine. I also fully understand reason #1. Just looked into Pymongo which indeed seems to make the map reduce hell a bit more bearable.
cb, make sure you grow yourself a thick skin with those major changes :) Since it will surely produce a load of “bad press”. Regarding MySQL: since I never liked this pieced-together crap of ancient code that has been made a real mess of constantly rewriting it within 20+ years (instead of getting rid of the old cruft once and for all), let alone their abysmal user support — I say ANYTHING is better than MySQL (even PostgreSQL).
Hopefully all goes well.
I mean, when I read MongoDB I had to think of http://howfuckedismydatabase.com/nosql/ :P
I still don’t really get the current MongoDB (or document based db) hype. Mongo full text indexes don’t look too inviting either http://docs.mongodb.org/manual/core/text-search/ What am I missing?
Cool stuff indeed!