Why memcacheDB sucks (and it’s not just the server side)

Over the last week, at Delta Projects, I've been working on replacing MemcacheDB with Cassandra. We have run with MemcacheDB for a while, because we store data that we can't store client side and we need some way of storing this serverside. Since the data needs to be available really quickly and has to be persistent, MemcacheDB (a server that exposes more or less the same API as Memcache, but persists to disk) was put in place some time ago. This worked well for a while, but since we're expanding, the requirements changed for storing data server side. The initial requirements were pretty clear: we can get away with a key/value store, latency is important and the data should be persistent. However, since we're going to run with multiple data centers, the requirements have changed. The initial requirements are still valid, but when running over multiple locations, we would like to have all data available in all locations. Next to that, scaling MemcacheDB is hard. We ran with two MemcacheDB servers, where data was sharded, but the databases on both machines outgrew the physical memory in the servers, which caused a huge impact on performance. Since data is sharded and not replicated, you can't just pop in an extra box and lessen the load on the other servers. You have to rebalance manually. So we went on a quest for different server side storage solutions that would fulfill our requirements.

Long story short, we decided on Cassandra.

Cassandra is more than a key/value store. It's a distributed data store that uses Bigtable's ColumnFamily design, which will allow us to use our data differently than just storing and retrieving data by using keys; we can actually query the data. Anyway, the decision was made to move from MemcacheDB to Cassandra and we had to come up with a migration path. The path we envisioned would be:

  • Keep reading and writing from and to MemcacheDB but also write to Cassandra.
  • Stop writing to MemcacheDB and start reading from Cassandra first and have MemcacheDB as a fall-back if we can't read from Cassandra.
  • In the mean time copy data from MemcacheDB. Keys that are not in Cassandra will be copied, the others will be skipped
  • Disable reading from MemcacheDB.

This meant that we had to run with both Cassandra and MemcacheDB at the same time.

While investigating the use of Cassandra, we found out that the Ruby client for Cassandra doesn't deliver. Mainly because Cassandra uses Thrift and the Thrift client for Ruby isn't too good. However, Hector, a Java client for Cassandra, worked flawlessly. This, together with some other discoveries1 made us decide to move our application from MRI Ruby (the standard Ruby interpreter) to JRuby (a Ruby interpreter on the JVM). The nice thing about JRuby is that we were able to use best of both the Java and the Ruby world. Most Ruby code runs well in Ruby, but sometimes code has to be adjusted a little or sometimes it's better to use a JRuby version of a specific library; one that's more suited for running on the JVM.

So, we added Cassandra support to our application and moved from the standard Ruby memcache client to the JRuby version. The JRuby version is a small wrapper that exposes the same API as the Ruby version and wraps an existing Java Memcache client. However, after some testing, we found out that the way keys are distributed over the MemcacheDB servers is not the same in both implementations. Both implementations used different algorithms to determine which key should be placed on which server. The Java version provides the ability to choose between three different algorithms, while the Ruby version only provides one, but neither one is compatible with the others.

For getting our JRuby application to talk to our MemcacheDB servers, we had two options; patch the Java version so it behaved like the Ruby version, or just run with the Ruby version (JRuby is still Ruby, right?). Initially, we chose the latter option, which seemed to be the quickest fix. Just pop in the gem and you're done.

Right.

In the MRI Ruby version of our application, we used one MemcacheDB client per process (and then running with multiple processes), while in the JRuby version, we had to run with one client per Java thread. The standard Ruby version of the client is not very good with Java threads and apparently, TCP sockets are handled differently in Java then when using the MRI interpreter. Sockets would linger for a very long time and when the MemcacheDB server would be unresponsive for a while and we would try to reconnect, we would end up with a lot of established connections that were never going to be used, but also never closed. The JRuby version didn't seem to have all these problems and is (obviously) much better suited to run in the JVM; it uses nice socket pools and such.

Not to waste time, we started patching the JRuby client to mimic the behavior of it's Ruby counterpart, which worked out pretty well. We still have to test it a little bit more thoroughly, but the first tests look promising.

The last couple of days have been a head-breaking quest, but I think we've finally nailed it. We really need to move to Cassandra soon, since MemcacheDB is only going to give us a lot more headaches. At least I got to code some Java again and I learned quite a few things about MemcacheDB and how sharding actually works. I also learned that MemcacheDB is a bad choice. Implementing it is done fairly quickly (since you can just use a Memcache client), but it doesn't scale and apparently the sharding differs from one implementation to the other (what if you run multiple applications accessing MemcacheDB and all are written in different languages?). It's too bad that we spent a lot of time finding a solution for something that is going to be replaced very soon.

If you're writing a new application and need persistent cache or something other than a SQL database, check out alternatives like Cassandra or MongoDB.

1. We decided not to use Scribe for writing to HDFS, but rolled our own code using ActiveMQ, in-process.
Published at June 12, 2010 in Uncategorized. 0 Comments

Is it worth contributing to open source software?

Over the last couple of weeks, my colleagues and me at Delta Projects have been working on some extra functionality for Facebook's Scribe server. Scribe is a server that logs messages. It can do this to multiple different "stores" and does failover from one store to the other if one fails. In our case, we wanted Scribe to log messages from our ad servers to HDFS and in case of an HDFS failure, write to local disk and replay those logs, once HDFS becomes available again.
After some investigation, we noticed that Scribe was very static in the way file paths were constructed and we couldn't fulfill our requirements; we want to have our log files structured in a directory structure like /year/month/day/hour/adserver.log.
Here is where open source software is great: you have the code, so you can modify it yourself. So we did. We spent a couple of hours implementing dynamic file paths and then committed the feature to our own github fork of scribe. We posted the patch to the Scribe mailing list and spoke about it on IRC with some guys of Facebook and Twitter, but not much happened.
Another requirement is that we store our data compressed on HDFS, but Scribe doesn't come with compression. There is a patch for LZO from Twitter, but because of license issues, this patch will never make it into the main branch of Scribe. Because of this, we decided that we would either use FastLZ or BZip2 for our purposes. But again, Scribe has no built-in support for compression.
Since we already poked around in the Scribe code, we decided to implement compression ourselves. The problem with Scribe however is that it basically untested. There are a couple of tests written in PHP, but the don't cover the whole application. Next to that, the Scribe source (written in C++) is a mess. It's one big hack. No neat design patterns, a lot of dead code everywhere, classes with way too much responsibility, etc. So, for our implementation of compression, we figured that it would be nice not to hack it in there (even the LZO patch from Twitter seemed like a quick hack), but do it correctly, so that we wouldn't only introduce new functionality, but also clean up a lot and refractor ugly parts of the code. We extracted and abstracted a lot of code and put it in the right place (using a lot of practices that Uncle Bob preaches in "Clean Code"). And as icing on the cake, we threw in tests. We introduced Google Tests, a C++ testing framework and wrote tests for every piece of code that we wrote.
Then the day came that our work was finished, so we committed our changes to our github branch and posted our patch to the scribe mailing list. The patch was around 200k (including the tests), which is big and apparently, it scared the guys at Facebook, because after that, nothing happened. The only response was that the patch was too big and we we're asked if we could cut it up in smaller pieces, so reviewing the code would be easier. Unfortunately, we couldn't do this, since so many interweaved concepts were pulled apart and put into different places. We would have to retrace our steps and create a patch for every one of them, with having a working system after every step.
The end result is that we have a lot of (imho really nice) code, that probably nobody is going to use. I even think that we're not going to use it either, since it's in a branch of Scribe that is only maintained by us. If it would be in the main branch of Facebook's Scribe, the code would be maintained by "the community", but keeping it only in our branch will cause a divergence of our branch in the future and we don't want to maintain a spin-off of Scribe for ages to come.
I think it's great that companies like Facebook and Twitter open source a lot of software, but when there is no large community outside of those companies, adding features might not be worth it. Scribe only seems to be used (on a large scale) by Twitter and Facebook and whatever suits them will make it to the main branch. If only it was covered by tests, then the maintainers could have quickly verified our work, but now, there is no reason for Facebook to merge in our patch, so there is no need for allocating resources to review our changes. And again, forking the whole project and maintaining it isn't an option for us. We'll move on to other technology..
Published at June 5, 2010 in Uncategorized. 0 Comments

Ruby’s garbage collector and caching

My current job at Delta Projects is great in terms of working with high volumes. The fact that we serve around 50+ million ads a day creates the need for different approaches in terms of storing and retrieving data. Our ad servers for example are completely independent from the backend. Business logic and models are exported as code in our backend systems and pushed to our array of ad servers. Content itself is put into a git repository and is updated regularly by the ad servers. The backend system is unaware of which ad servers there are. Raw data is pushed back from the ad servers and processed later.

Because we want to serve ads as quickly as possible (and creating the least possible delay on the page that uses ads that are served by us), we cache ads on the ad servers. Some weeks ago however, we noticed that CPU usage was increasing at a very high rate over time. Memory usage was also increasing, but not as quickly as the CPU usage. The memory increase was caused by the fact that we stored ad meta data in memory inside our application. Since we run around 10 Unicorn processes, an ad would be in memory 10 times, but that wasn't such a big deal, since we run with a lot of RAM. The CPU consumption was more worrying. After some investigation we found out that when all ads are loaded in memory, we had a lot of string objects in memory (around 1.2 million) that were retained, since a global cache array would keep a reference to them. In other words, the objects are never garbage collected. But, since ruby 1.8 has a non-generational GC, all objects are inspected by the GC and having 10 processes performing a GC run over 1.2 million objects every now and then, caused a lot of CPU load.

So, we needed a better way of caching. Memcached was our first idea, but having yet another process on which we depend didn't feel like such a good idea. Since it's a local cache, my colleague Kalle came up with the idea of storing our cache data on tmpfs. Our application takes care of filling the cache (since it serializes ad meta data) and reading from it. Invalidating cache items is now done through a git hook, that simply removes a file that has been updated or deleted from tmpfs.

This al lead to a tenth of the memory consumption and  a lot less and constant cpu usage.

Published at February 20, 2010 in english, ruby and work. 0 Comments

Finger part II

Last time I blogged, I was sitting at the hospital (Huddinge Sjukhus, for the people who like to know where I go), waiting for my doctors appointment to check out how my finger was. Well, the fractures in my finger (about five or six in the little bone) were healing, but my finger was rotated. When making a fist, my finger doesn't go in the normal direction, but a few degrees too much to the other side. The orthopedic doctor didn't really know what to do with it and told me that I should go to the hand surgeon in a different hospital (Södersjukhuset this time) and let him check what we should do. The doctor made me and appointment on Monday morning, which meant that they would wrap up my finger in plaster again for the weekend. She told me that I should be at the hand surgeon sober, since if this guy wanted to operate on my finger, he might do it the same day. Well that kind of sucked. I wasn't really looking forward to getting my finger operated, since I want to use a computer. On Monday, I went there and after another set of x-rays the surgeon told me that my finger was rotated a bit and that there were 3 options: operate now and make it straight, operate later (which means breaking it again) and make it straight or just live with it and accept the fact that I have a crooked finger. The doctor wasn't a big fan of operating now, since the results might not be that great and there would be a chance of having scar tissue, since my muscles are still recovering. The same goes for operating later, but then the results might be slightly better. But operating would mean a stiff finger for 3 to 6 months. Keeping it as it is would give me back 100% strength and grip with the help of physiotherapy, but the only thing would be that my finger will always be a little bit rotated. No biggie I think, so I decided to leave it for now and do physiotherapy. The doctor gave me a splint that I can take off at work and that I have to wear for two weeks, outside and when I sleep, since my finger is still broken. After that, I should use the thing for 4 more weeks when doing something dangerous (whatever that means). Så, all well, I think.. I can type and thus code again, which was (apart from the difficulty of wiping my ass with my left hand) the biggest annoyance of not being able to use my right fingers..

Published at February 12, 2010 in Uncategorized. 0 Comments

Awesomeness and a broken finger

Sorry, I'm a bad blogger. I promised a lot of people to blog about my move to Sweden, but I haven't. And since I still don't have a mobile phone subscription, I can't use twitter as I used to, every moment of the day. The short version is in the title: Sweden is awesome and I broke a finger.

The somewhat longer version is that life is great here. Me and Kasia are doing great; I'm having such a great time being with her. Work is great too. I'm doing pretty cool stuff, both coding and tinkering with networks and unix systems to squeeze out every possibility to get even better performance. We're currently serving 50 million ads a day (and yes, online ads.. I sold my soul) with just a hand full of servers, running a Ruby application. The team I'm working with consists of only smart an experienced people, which makes it even better. I'm learning every day.

My Swedish isn't really great yet. I've signed up for a course, but I have to wait some months to start, since it's fully booked. In the mean time I'm doing some Rosetta Stone, but not so much. I hear a lot of Swedish at work, but everybody speaks English, so there is no real need to speak Swedish. Reading is not so hard, since there are a lot of commonalities with Dutch, but speaking is a lot more difficult.  However, I had a small breakthrough yesterday, when I went to the hairdresser. She doesn't speak English, so I was forced to speak Swedish and I had an actual (and pretty decent) conversation with her! I guess it's all about not being shy and just try.

Swedes are friendly people and very modest. Business is somewhat different than in the Netherlands. For example, meetings are different. In the Netherlands, often people go into a meeting with a goal. Decisions should be made. A general consensus is OK, but there should be a result after a meeting. Here it's different. Meetings are for listening to people and talk about the stuff that needs attention, but decisions are made at the coffee machine. I read some articles about this and some said that Swedish mentality is closer to the way people do business in Belgium, but I'm not sure. Belgians seem far more hierarchical; one person is the boss. Here, it's more about compromises and talking.

Some people asked me if Sweden is expensive. I guess it can be, but apart from the alcohol, it feels a bit like the Netherlands. Of course my salary is in Swedish Kronor, so it's harder to compare, but I don't really have the feeling that products are that more expensive here. I must admit that I'm not very price aware, so it's more a gut feeling. VAT is higher; 25% and 12% instead of 19% and 6%, but I guess you get a lot back from that. Alcohol is excessively expensive though. Half a liter (note to Wayne: "I'm sorry sir, I don't know what a liter is") of beer costs roughly Euro 6,50 in a bar and a Mojito is roughly Euro 13,-. Next to that, you can't buy wine or (normal) beer in the supermarket. There is only one company that sells alcohol and it's run by the government and there a bottle of beer is about Euro 1,50. In the supermarket you can buy 3.5% vol. beer, but that just tastes like water.

But you get a lot back for the stuff you pay. Public transport is good. I think it's better than in the Netherlands. Metro runs pretty much on time (in the Dutch meaning of the word) and often. Swedes tend to have a thing for time. If the metro is 3 minutes late, it's late. 13 past the hour isn't a quarter past, it's 13 past. Interesting difference.

Another thing you get is health care. Paying for that is done through taxes, directly. Not through strange systems with insurance companies. If you need health care, you will get it. Everybody is equal. The first 900 kr (90 euro's) per year, you'll have to pay by yourself, and then it's free for 365 days.

How do I know this? Experience. 3 weeks ago I broke my right ring finger. I went snow boarding with a friend and at the beginning of the first run, I touched the ground with my hand while trying to keep balance. It hurt, but I thought that it would be just a contusion. It wasn't even a cool crash or anything and I had been standing on my board for only 5 seconds. I went to the first aid at the slope to have it checked, but the guy there said it was probably only a contusion and that it was just a little bit swollen. so, I continued boarding for a couple of hours. My finger turned blue and purple the next day, but when the swelling disappeared after a couple of days. I still couldn't move it and it still hurt, so I decided to see a doctor and after some x-rays it turned out to be broken. At the hospital, I got a cast that is coming off today. I'm actually sitting at the hospital while writing this piece. I just had x-rays and I have a doctors appointment in an hour. Hope it healed a bit and that they'll take off the cast, because it's really uncomfortable and typing is pretty hard.

I'll try to make some more time in the future to blog and I hope I can get a mobile subscription soon, so I'm able to be online a bit more. But I'm alright!

Published at February 5, 2010 in english, life and sweden. 2 Comments

The first week

For everybody I didn't speak, or who doesn't follow me through other ways: I have arrived and I'm great! The last week has been pretty busy. On Thursday, I moved. Wayne drove me and it took about 15 hours to get from Utrecht to Stockholm. It was kind of strange to close the door behind me, but I was really prepared for the moment. We drove through Germany, took the ferry to Denmark (probably the most boring country in Europe) and from there drove to Sweden, using the tunnel and bridge between Copenhagen and Malmö. Malmö is about halfway, so it took around 7 hours to drive up to Stockholm. In the evening we finally arrived and it was great to see my girl again. My sister and my mom were already there and the following weekend we spent strolling around Stockholm. My girl threw me a welcome party on Saturday, which was awesome. Some people I already knew where there and I met some new ones. On Sunday, my mom and sister said goodbye and went back to the Netherlands. Because of all the traveling, moving and stress of the last couple of months, we didn't do a lot in the evening. On Monday, I went to some government offices to take care of the paperwork and on Tuesday, I started my new job. The company is great; a lot of very smart people and a great atmosphere. In the next couple of days/weeks I'll get more adjusted, but I really feel home here. Life is great!

Published at December 2, 2009 in Uncategorized. 0 Comments

Ready.. set.. go!

I'm almost done. Yesterday, I said goodbye to a lot of friends and some family during the party I threw. It was amazing to see how many people came and to talk to all of them. Of course I'm not gone, but I know that I will see many a lot less and will probably not speak with them so often, so it was good to speak and remember great times together. Thanks to all of you who joined me in celebrating the beginning of a new chapter in my life! I had a lovely day and will remember this for the rest of my life.

My task list is almost empty. I sold my car, my dad took care of the beds and tomorrow I will bring the cats to their temporary home. In February I will fly back to get them. The last thing to do is to actually drive. Thursday the 26th at 5 o'clock in the morning, we drive. It's a 16 hour drive through Germany and Denmark and I really can't wait. Over the last weeks I felt a little nervous and excited, I felt stressed, but after yesterday, I feel only excitement. I took care of everything and everything worked out very well, so I am ready!

Published at November 22, 2009 in Uncategorized. 0 Comments

Only 2 weeks to go..

I thought I'd write more about the progress of moving to Sweden, but I've been terribly busy lately. In short, the status is that almost everything is done! At least the most important stuff. First, I found a job. I'll start the 1st of December at a company called The Delta Projects, where I'll start as a System Developer. I'll probably start working on their flagship product AdAction, with which they serve about 30 million online ads a day. I've met with my new manager 3 times already and spoken to my new colleagues twice and I'm really excited to start. I'll probably code in Ruby a lot and play with new technology.

The second biggy is that my house is rented. Thanks to Wayne, I was able to rent it to a company. The house worried me a bit, since it's not that easy to rent it for the price I wanted to get for it, but it finally worked out.

My cats will stay in the Netherlands for about 3,5 months, since they need a rabies check, 120 days after their vaccination. Through http://clubvan100.rvu.nl I found someone that is willing to take care of them for a couple of months.

The rest was basically canceling  lot of subscriptions, getting the right documents from the Dutch government, canceling my company and so on.

My friend Wayne is going to drive me to Stockholm on the 26th of November. Today I started packing my stuff, since next week we're going to try if everything fits in his car, but from the looks of it, it might fit.

The only things I still need to do is to sell my car and take care of my bed. Since 4 expats will live in my house, I would like to trade my bed (boxspring) for 4 single beds. So if you happen to know anyone, let me know.

The final date comes is really in sight and I can't wait to go. Me and the misses are great and it feels more and more that this has been the right choice.

Published at November 13, 2009 in Uncategorized. 2 Comments

Pre-migration party!

At the 21st of November, I'm throwing a party. This is of course because because I'm migrating. I don't like the words farewell or goodbye party, since I'm not saying goodbye to anyone. I'm just moving to another country. If you feel that you should celebrate with me, you should be there. The location isn't clear yet. Could be my house, could be somewhere else. Details will follow. If you want to join me, let me know (I have an event page on Facebook, or you can just email me, twitter or comment below), so I can make sure that there will be enough drinks. It will probably be the last time that I'll be able to throw a party with (relatively) cheap booze.

Published at September 26, 2009 in Uncategorized. 1 Comment

A new date – moving early

In my last post, I wrote about my plans for moving to Stockholm, Sweden. I forgot to mention the exact date, but later, I edited the post and said that it would be the 1st of February 2010. Over the last few weeks I've been speaking to people to find a job and I already had a job interview in Stockholm with a company that seems very interesting. Because I sold my shares in Jewel Labs and staying in the Netherlands will only cost me money, I decided (after discussing this with my colleagues, my family and my girl) to move to Stockholm early. The planning now is to move before the 1st of December. Of course, I still need to rent my apartment to someone (contacted some companies that rent stuff to expats) and find a job, but I know that everything will work out.

Published at September 15, 2009 in life, sweden and work. 0 Comments
« Previous Entries