Archive for the 'Uncategorized' Category

Why memcacheDB sucks (and it’s not just the server side)

Over the last week, at Delta Projects, I've been working on replacing MemcacheDB with Cassandra. We have run with MemcacheDB for a while, because we store data that we can't store client side and we need some way of storing this serverside. Since the data needs to be available really quickly and has to be persistent, MemcacheDB (a server that exposes more or less the same API as Memcache, but persists to disk) was put in place some time ago. This worked well for a while, but since we're expanding, the requirements changed for storing data server side. The initial requirements were pretty clear: we can get away with a key/value store, latency is important and the data should be persistent. However, since we're going to run with multiple data centers, the requirements have changed. The initial requirements are still valid, but when running over multiple locations, we would like to have all data available in all locations. Next to that, scaling MemcacheDB is hard. We ran with two MemcacheDB servers, where data was sharded, but the databases on both machines outgrew the physical memory in the servers, which caused a huge impact on performance. Since data is sharded and not replicated, you can't just pop in an extra box and lessen the load on the other servers. You have to rebalance manually. So we went on a quest for different server side storage solutions that would fulfill our requirements.

Long story short, we decided on Cassandra.

Cassandra is more than a key/value store. It's a distributed data store that uses Bigtable's ColumnFamily design, which will allow us to use our data differently than just storing and retrieving data by using keys; we can actually query the data. Anyway, the decision was made to move from MemcacheDB to Cassandra and we had to come up with a migration path. The path we envisioned would be:

  • Keep reading and writing from and to MemcacheDB but also write to Cassandra.
  • Stop writing to MemcacheDB and start reading from Cassandra first and have MemcacheDB as a fall-back if we can't read from Cassandra.
  • In the mean time copy data from MemcacheDB. Keys that are not in Cassandra will be copied, the others will be skipped
  • Disable reading from MemcacheDB.

This meant that we had to run with both Cassandra and MemcacheDB at the same time.

While investigating the use of Cassandra, we found out that the Ruby client for Cassandra doesn't deliver. Mainly because Cassandra uses Thrift and the Thrift client for Ruby isn't too good. However, Hector, a Java client for Cassandra, worked flawlessly. This, together with some other discoveries1 made us decide to move our application from MRI Ruby (the standard Ruby interpreter) to JRuby (a Ruby interpreter on the JVM). The nice thing about JRuby is that we were able to use best of both the Java and the Ruby world. Most Ruby code runs well in Ruby, but sometimes code has to be adjusted a little or sometimes it's better to use a JRuby version of a specific library; one that's more suited for running on the JVM.

So, we added Cassandra support to our application and moved from the standard Ruby memcache client to the JRuby version. The JRuby version is a small wrapper that exposes the same API as the Ruby version and wraps an existing Java Memcache client. However, after some testing, we found out that the way keys are distributed over the MemcacheDB servers is not the same in both implementations. Both implementations used different algorithms to determine which key should be placed on which server. The Java version provides the ability to choose between three different algorithms, while the Ruby version only provides one, but neither one is compatible with the others.

For getting our JRuby application to talk to our MemcacheDB servers, we had two options; patch the Java version so it behaved like the Ruby version, or just run with the Ruby version (JRuby is still Ruby, right?). Initially, we chose the latter option, which seemed to be the quickest fix. Just pop in the gem and you're done.

Right.

In the MRI Ruby version of our application, we used one MemcacheDB client per process (and then running with multiple processes), while in the JRuby version, we had to run with one client per Java thread. The standard Ruby version of the client is not very good with Java threads and apparently, TCP sockets are handled differently in Java then when using the MRI interpreter. Sockets would linger for a very long time and when the MemcacheDB server would be unresponsive for a while and we would try to reconnect, we would end up with a lot of established connections that were never going to be used, but also never closed. The JRuby version didn't seem to have all these problems and is (obviously) much better suited to run in the JVM; it uses nice socket pools and such.

Not to waste time, we started patching the JRuby client to mimic the behavior of it's Ruby counterpart, which worked out pretty well. We still have to test it a little bit more thoroughly, but the first tests look promising.

The last couple of days have been a head-breaking quest, but I think we've finally nailed it. We really need to move to Cassandra soon, since MemcacheDB is only going to give us a lot more headaches. At least I got to code some Java again and I learned quite a few things about MemcacheDB and how sharding actually works. I also learned that MemcacheDB is a bad choice. Implementing it is done fairly quickly (since you can just use a Memcache client), but it doesn't scale and apparently the sharding differs from one implementation to the other (what if you run multiple applications accessing MemcacheDB and all are written in different languages?). It's too bad that we spent a lot of time finding a solution for something that is going to be replaced very soon.

If you're writing a new application and need persistent cache or something other than a SQL database, check out alternatives like Cassandra or MongoDB.

1. We decided not to use Scribe for writing to HDFS, but rolled our own code using ActiveMQ, in-process.
Published at June 12, 2010 in Uncategorized. 0 Comments

Is it worth contributing to open source software?

Over the last couple of weeks, my colleagues and me at Delta Projects have been working on some extra functionality for Facebook's Scribe server. Scribe is a server that logs messages. It can do this to multiple different "stores" and does failover from one store to the other if one fails. In our case, we wanted Scribe to log messages from our ad servers to HDFS and in case of an HDFS failure, write to local disk and replay those logs, once HDFS becomes available again.
After some investigation, we noticed that Scribe was very static in the way file paths were constructed and we couldn't fulfill our requirements; we want to have our log files structured in a directory structure like /year/month/day/hour/adserver.log.
Here is where open source software is great: you have the code, so you can modify it yourself. So we did. We spent a couple of hours implementing dynamic file paths and then committed the feature to our own github fork of scribe. We posted the patch to the Scribe mailing list and spoke about it on IRC with some guys of Facebook and Twitter, but not much happened.
Another requirement is that we store our data compressed on HDFS, but Scribe doesn't come with compression. There is a patch for LZO from Twitter, but because of license issues, this patch will never make it into the main branch of Scribe. Because of this, we decided that we would either use FastLZ or BZip2 for our purposes. But again, Scribe has no built-in support for compression.
Since we already poked around in the Scribe code, we decided to implement compression ourselves. The problem with Scribe however is that it basically untested. There are a couple of tests written in PHP, but the don't cover the whole application. Next to that, the Scribe source (written in C++) is a mess. It's one big hack. No neat design patterns, a lot of dead code everywhere, classes with way too much responsibility, etc. So, for our implementation of compression, we figured that it would be nice not to hack it in there (even the LZO patch from Twitter seemed like a quick hack), but do it correctly, so that we wouldn't only introduce new functionality, but also clean up a lot and refractor ugly parts of the code. We extracted and abstracted a lot of code and put it in the right place (using a lot of practices that Uncle Bob preaches in "Clean Code"). And as icing on the cake, we threw in tests. We introduced Google Tests, a C++ testing framework and wrote tests for every piece of code that we wrote.
Then the day came that our work was finished, so we committed our changes to our github branch and posted our patch to the scribe mailing list. The patch was around 200k (including the tests), which is big and apparently, it scared the guys at Facebook, because after that, nothing happened. The only response was that the patch was too big and we we're asked if we could cut it up in smaller pieces, so reviewing the code would be easier. Unfortunately, we couldn't do this, since so many interweaved concepts were pulled apart and put into different places. We would have to retrace our steps and create a patch for every one of them, with having a working system after every step.
The end result is that we have a lot of (imho really nice) code, that probably nobody is going to use. I even think that we're not going to use it either, since it's in a branch of Scribe that is only maintained by us. If it would be in the main branch of Facebook's Scribe, the code would be maintained by "the community", but keeping it only in our branch will cause a divergence of our branch in the future and we don't want to maintain a spin-off of Scribe for ages to come.
I think it's great that companies like Facebook and Twitter open source a lot of software, but when there is no large community outside of those companies, adding features might not be worth it. Scribe only seems to be used (on a large scale) by Twitter and Facebook and whatever suits them will make it to the main branch. If only it was covered by tests, then the maintainers could have quickly verified our work, but now, there is no reason for Facebook to merge in our patch, so there is no need for allocating resources to review our changes. And again, forking the whole project and maintaining it isn't an option for us. We'll move on to other technology..
Published at June 5, 2010 in Uncategorized. 0 Comments

Finger part II

Last time I blogged, I was sitting at the hospital (Huddinge Sjukhus, for the people who like to know where I go), waiting for my doctors appointment to check out how my finger was. Well, the fractures in my finger (about five or six in the little bone) were healing, but my finger was rotated. When making a fist, my finger doesn't go in the normal direction, but a few degrees too much to the other side. The orthopedic doctor didn't really know what to do with it and told me that I should go to the hand surgeon in a different hospital (Södersjukhuset this time) and let him check what we should do. The doctor made me and appointment on Monday morning, which meant that they would wrap up my finger in plaster again for the weekend. She told me that I should be at the hand surgeon sober, since if this guy wanted to operate on my finger, he might do it the same day. Well that kind of sucked. I wasn't really looking forward to getting my finger operated, since I want to use a computer. On Monday, I went there and after another set of x-rays the surgeon told me that my finger was rotated a bit and that there were 3 options: operate now and make it straight, operate later (which means breaking it again) and make it straight or just live with it and accept the fact that I have a crooked finger. The doctor wasn't a big fan of operating now, since the results might not be that great and there would be a chance of having scar tissue, since my muscles are still recovering. The same goes for operating later, but then the results might be slightly better. But operating would mean a stiff finger for 3 to 6 months. Keeping it as it is would give me back 100% strength and grip with the help of physiotherapy, but the only thing would be that my finger will always be a little bit rotated. No biggie I think, so I decided to leave it for now and do physiotherapy. The doctor gave me a splint that I can take off at work and that I have to wear for two weeks, outside and when I sleep, since my finger is still broken. After that, I should use the thing for 4 more weeks when doing something dangerous (whatever that means). Så, all well, I think.. I can type and thus code again, which was (apart from the difficulty of wiping my ass with my left hand) the biggest annoyance of not being able to use my right fingers..

Published at February 12, 2010 in Uncategorized. 0 Comments

The first week

For everybody I didn't speak, or who doesn't follow me through other ways: I have arrived and I'm great! The last week has been pretty busy. On Thursday, I moved. Wayne drove me and it took about 15 hours to get from Utrecht to Stockholm. It was kind of strange to close the door behind me, but I was really prepared for the moment. We drove through Germany, took the ferry to Denmark (probably the most boring country in Europe) and from there drove to Sweden, using the tunnel and bridge between Copenhagen and Malmö. Malmö is about halfway, so it took around 7 hours to drive up to Stockholm. In the evening we finally arrived and it was great to see my girl again. My sister and my mom were already there and the following weekend we spent strolling around Stockholm. My girl threw me a welcome party on Saturday, which was awesome. Some people I already knew where there and I met some new ones. On Sunday, my mom and sister said goodbye and went back to the Netherlands. Because of all the traveling, moving and stress of the last couple of months, we didn't do a lot in the evening. On Monday, I went to some government offices to take care of the paperwork and on Tuesday, I started my new job. The company is great; a lot of very smart people and a great atmosphere. In the next couple of days/weeks I'll get more adjusted, but I really feel home here. Life is great!

Published at December 2, 2009 in Uncategorized. 0 Comments

Ready.. set.. go!

I'm almost done. Yesterday, I said goodbye to a lot of friends and some family during the party I threw. It was amazing to see how many people came and to talk to all of them. Of course I'm not gone, but I know that I will see many a lot less and will probably not speak with them so often, so it was good to speak and remember great times together. Thanks to all of you who joined me in celebrating the beginning of a new chapter in my life! I had a lovely day and will remember this for the rest of my life.

My task list is almost empty. I sold my car, my dad took care of the beds and tomorrow I will bring the cats to their temporary home. In February I will fly back to get them. The last thing to do is to actually drive. Thursday the 26th at 5 o'clock in the morning, we drive. It's a 16 hour drive through Germany and Denmark and I really can't wait. Over the last weeks I felt a little nervous and excited, I felt stressed, but after yesterday, I feel only excitement. I took care of everything and everything worked out very well, so I am ready!

Published at November 22, 2009 in Uncategorized. 0 Comments

Only 2 weeks to go..

I thought I'd write more about the progress of moving to Sweden, but I've been terribly busy lately. In short, the status is that almost everything is done! At least the most important stuff. First, I found a job. I'll start the 1st of December at a company called The Delta Projects, where I'll start as a System Developer. I'll probably start working on their flagship product AdAction, with which they serve about 30 million online ads a day. I've met with my new manager 3 times already and spoken to my new colleagues twice and I'm really excited to start. I'll probably code in Ruby a lot and play with new technology.

The second biggy is that my house is rented. Thanks to Wayne, I was able to rent it to a company. The house worried me a bit, since it's not that easy to rent it for the price I wanted to get for it, but it finally worked out.

My cats will stay in the Netherlands for about 3,5 months, since they need a rabies check, 120 days after their vaccination. Through http://clubvan100.rvu.nl I found someone that is willing to take care of them for a couple of months.

The rest was basically canceling  lot of subscriptions, getting the right documents from the Dutch government, canceling my company and so on.

My friend Wayne is going to drive me to Stockholm on the 26th of November. Today I started packing my stuff, since next week we're going to try if everything fits in his car, but from the looks of it, it might fit.

The only things I still need to do is to sell my car and take care of my bed. Since 4 expats will live in my house, I would like to trade my bed (boxspring) for 4 single beds. So if you happen to know anyone, let me know.

The final date comes is really in sight and I can't wait to go. Me and the misses are great and it feels more and more that this has been the right choice.

Published at November 13, 2009 in Uncategorized. 2 Comments

Pre-migration party!

At the 21st of November, I'm throwing a party. This is of course because because I'm migrating. I don't like the words farewell or goodbye party, since I'm not saying goodbye to anyone. I'm just moving to another country. If you feel that you should celebrate with me, you should be there. The location isn't clear yet. Could be my house, could be somewhere else. Details will follow. If you want to join me, let me know (I have an event page on Facebook, or you can just email me, twitter or comment below), so I can make sure that there will be enough drinks. It will probably be the last time that I'll be able to throw a party with (relatively) cheap booze.

Published at September 26, 2009 in Uncategorized. 1 Comment