Archive for February, 2010

Ruby’s garbage collector and caching

My current job at Delta Projects is great in terms of working with high volumes. The fact that we serve around 50+ million ads a day creates the need for different approaches in terms of storing and retrieving data. Our ad servers for example are completely independent from the backend. Business logic and models are exported as code in our backend systems and pushed to our array of ad servers. Content itself is put into a git repository and is updated regularly by the ad servers. The backend system is unaware of which ad servers there are. Raw data is pushed back from the ad servers and processed later.

Because we want to serve ads as quickly as possible (and creating the least possible delay on the page that uses ads that are served by us), we cache ads on the ad servers. Some weeks ago however, we noticed that CPU usage was increasing at a very high rate over time. Memory usage was also increasing, but not as quickly as the CPU usage. The memory increase was caused by the fact that we stored ad meta data in memory inside our application. Since we run around 10 Unicorn processes, an ad would be in memory 10 times, but that wasn't such a big deal, since we run with a lot of RAM. The CPU consumption was more worrying. After some investigation we found out that when all ads are loaded in memory, we had a lot of string objects in memory (around 1.2 million) that were retained, since a global cache array would keep a reference to them. In other words, the objects are never garbage collected. But, since ruby 1.8 has a non-generational GC, all objects are inspected by the GC and having 10 processes performing a GC run over 1.2 million objects every now and then, caused a lot of CPU load.

So, we needed a better way of caching. Memcached was our first idea, but having yet another process on which we depend didn't feel like such a good idea. Since it's a local cache, my colleague Kalle came up with the idea of storing our cache data on tmpfs. Our application takes care of filling the cache (since it serializes ad meta data) and reading from it. Invalidating cache items is now done through a git hook, that simply removes a file that has been updated or deleted from tmpfs.

This al lead to a tenth of the memory consumption and  a lot less and constant cpu usage.

Published at February 20, 2010 in english, ruby and work. 0 Comments

Finger part II

Last time I blogged, I was sitting at the hospital (Huddinge Sjukhus, for the people who like to know where I go), waiting for my doctors appointment to check out how my finger was. Well, the fractures in my finger (about five or six in the little bone) were healing, but my finger was rotated. When making a fist, my finger doesn't go in the normal direction, but a few degrees too much to the other side. The orthopedic doctor didn't really know what to do with it and told me that I should go to the hand surgeon in a different hospital (Södersjukhuset this time) and let him check what we should do. The doctor made me and appointment on Monday morning, which meant that they would wrap up my finger in plaster again for the weekend. She told me that I should be at the hand surgeon sober, since if this guy wanted to operate on my finger, he might do it the same day. Well that kind of sucked. I wasn't really looking forward to getting my finger operated, since I want to use a computer. On Monday, I went there and after another set of x-rays the surgeon told me that my finger was rotated a bit and that there were 3 options: operate now and make it straight, operate later (which means breaking it again) and make it straight or just live with it and accept the fact that I have a crooked finger. The doctor wasn't a big fan of operating now, since the results might not be that great and there would be a chance of having scar tissue, since my muscles are still recovering. The same goes for operating later, but then the results might be slightly better. But operating would mean a stiff finger for 3 to 6 months. Keeping it as it is would give me back 100% strength and grip with the help of physiotherapy, but the only thing would be that my finger will always be a little bit rotated. No biggie I think, so I decided to leave it for now and do physiotherapy. The doctor gave me a splint that I can take off at work and that I have to wear for two weeks, outside and when I sleep, since my finger is still broken. After that, I should use the thing for 4 more weeks when doing something dangerous (whatever that means). Så, all well, I think.. I can type and thus code again, which was (apart from the difficulty of wiping my ass with my left hand) the biggest annoyance of not being able to use my right fingers..

Published at February 12, 2010 in Uncategorized. 0 Comments

Awesomeness and a broken finger

Sorry, I'm a bad blogger. I promised a lot of people to blog about my move to Sweden, but I haven't. And since I still don't have a mobile phone subscription, I can't use twitter as I used to, every moment of the day. The short version is in the title: Sweden is awesome and I broke a finger.

The somewhat longer version is that life is great here. Me and Kasia are doing great; I'm having such a great time being with her. Work is great too. I'm doing pretty cool stuff, both coding and tinkering with networks and unix systems to squeeze out every possibility to get even better performance. We're currently serving 50 million ads a day (and yes, online ads.. I sold my soul) with just a hand full of servers, running a Ruby application. The team I'm working with consists of only smart an experienced people, which makes it even better. I'm learning every day.

My Swedish isn't really great yet. I've signed up for a course, but I have to wait some months to start, since it's fully booked. In the mean time I'm doing some Rosetta Stone, but not so much. I hear a lot of Swedish at work, but everybody speaks English, so there is no real need to speak Swedish. Reading is not so hard, since there are a lot of commonalities with Dutch, but speaking is a lot more difficult.  However, I had a small breakthrough yesterday, when I went to the hairdresser. She doesn't speak English, so I was forced to speak Swedish and I had an actual (and pretty decent) conversation with her! I guess it's all about not being shy and just try.

Swedes are friendly people and very modest. Business is somewhat different than in the Netherlands. For example, meetings are different. In the Netherlands, often people go into a meeting with a goal. Decisions should be made. A general consensus is OK, but there should be a result after a meeting. Here it's different. Meetings are for listening to people and talk about the stuff that needs attention, but decisions are made at the coffee machine. I read some articles about this and some said that Swedish mentality is closer to the way people do business in Belgium, but I'm not sure. Belgians seem far more hierarchical; one person is the boss. Here, it's more about compromises and talking.

Some people asked me if Sweden is expensive. I guess it can be, but apart from the alcohol, it feels a bit like the Netherlands. Of course my salary is in Swedish Kronor, so it's harder to compare, but I don't really have the feeling that products are that more expensive here. I must admit that I'm not very price aware, so it's more a gut feeling. VAT is higher; 25% and 12% instead of 19% and 6%, but I guess you get a lot back from that. Alcohol is excessively expensive though. Half a liter (note to Wayne: "I'm sorry sir, I don't know what a liter is") of beer costs roughly Euro 6,50 in a bar and a Mojito is roughly Euro 13,-. Next to that, you can't buy wine or (normal) beer in the supermarket. There is only one company that sells alcohol and it's run by the government and there a bottle of beer is about Euro 1,50. In the supermarket you can buy 3.5% vol. beer, but that just tastes like water.

But you get a lot back for the stuff you pay. Public transport is good. I think it's better than in the Netherlands. Metro runs pretty much on time (in the Dutch meaning of the word) and often. Swedes tend to have a thing for time. If the metro is 3 minutes late, it's late. 13 past the hour isn't a quarter past, it's 13 past. Interesting difference.

Another thing you get is health care. Paying for that is done through taxes, directly. Not through strange systems with insurance companies. If you need health care, you will get it. Everybody is equal. The first 900 kr (90 euro's) per year, you'll have to pay by yourself, and then it's free for 365 days.

How do I know this? Experience. 3 weeks ago I broke my right ring finger. I went snow boarding with a friend and at the beginning of the first run, I touched the ground with my hand while trying to keep balance. It hurt, but I thought that it would be just a contusion. It wasn't even a cool crash or anything and I had been standing on my board for only 5 seconds. I went to the first aid at the slope to have it checked, but the guy there said it was probably only a contusion and that it was just a little bit swollen. so, I continued boarding for a couple of hours. My finger turned blue and purple the next day, but when the swelling disappeared after a couple of days. I still couldn't move it and it still hurt, so I decided to see a doctor and after some x-rays it turned out to be broken. At the hospital, I got a cast that is coming off today. I'm actually sitting at the hospital while writing this piece. I just had x-rays and I have a doctors appointment in an hour. Hope it healed a bit and that they'll take off the cast, because it's really uncomfortable and typing is pretty hard.

I'll try to make some more time in the future to blog and I hope I can get a mobile subscription soon, so I'm able to be online a bit more. But I'm alright!

Published at February 5, 2010 in english, life and sweden. 2 Comments