Archive for the 'work' Category

Ruby’s garbage collector and caching

My current job at Delta Projects is great in terms of working with high volumes. The fact that we serve around 50+ million ads a day creates the need for different approaches in terms of storing and retrieving data. Our ad servers for example are completely independent from the backend. Business logic and models are exported as code in our backend systems and pushed to our array of ad servers. Content itself is put into a git repository and is updated regularly by the ad servers. The backend system is unaware of which ad servers there are. Raw data is pushed back from the ad servers and processed later.

Because we want to serve ads as quickly as possible (and creating the least possible delay on the page that uses ads that are served by us), we cache ads on the ad servers. Some weeks ago however, we noticed that CPU usage was increasing at a very high rate over time. Memory usage was also increasing, but not as quickly as the CPU usage. The memory increase was caused by the fact that we stored ad meta data in memory inside our application. Since we run around 10 Unicorn processes, an ad would be in memory 10 times, but that wasn't such a big deal, since we run with a lot of RAM. The CPU consumption was more worrying. After some investigation we found out that when all ads are loaded in memory, we had a lot of string objects in memory (around 1.2 million) that were retained, since a global cache array would keep a reference to them. In other words, the objects are never garbage collected. But, since ruby 1.8 has a non-generational GC, all objects are inspected by the GC and having 10 processes performing a GC run over 1.2 million objects every now and then, caused a lot of CPU load.

So, we needed a better way of caching. Memcached was our first idea, but having yet another process on which we depend didn't feel like such a good idea. Since it's a local cache, my colleague Kalle came up with the idea of storing our cache data on tmpfs. Our application takes care of filling the cache (since it serializes ad meta data) and reading from it. Invalidating cache items is now done through a git hook, that simply removes a file that has been updated or deleted from tmpfs.

This al lead to a tenth of the memory consumption and  a lot less and constant cpu usage.

Published at February 20, 2010 in english, ruby and work. 0 Comments

A new date – moving early

In my last post, I wrote about my plans for moving to Stockholm, Sweden. I forgot to mention the exact date, but later, I edited the post and said that it would be the 1st of February 2010. Over the last few weeks I've been speaking to people to find a job and I already had a job interview in Stockholm with a company that seems very interesting. Because I sold my shares in Jewel Labs and staying in the Netherlands will only cost me money, I decided (after discussing this with my colleagues, my family and my girl) to move to Stockholm early. The planning now is to move before the 1st of December. Of course, I still need to rent my apartment to someone (contacted some companies that rent stuff to expats) and find a job, but I know that everything will work out.

Published at September 15, 2009 in life, sweden and work. 0 Comments

Big news: I’m migrating

Let me just cut to the chase: I'm moving to Stockholm.

For real. Not just for a holiday, or something else. I'm moving there to start a new life. Some months ago, I met an incredible girl and fell in love. She's living in Stockholm, Sweden and since I've had a long distance relationship before, I know that being apart for too long is not going to work for me. We spoke about how a relationship would be possible and the only way seems to be close to each other. Because of circumstances, I would be the one that should move and after some consideration, I decided to take the plunge an just go.

Of course, there will be stuff and people that I leave behind. Family and friends, but Stockholm isn't so far (cheap 1.5 hour flights from and to Eindhoven), my house (I will rent it to someone), my company (this was actually the hardest part of the decision) and swimming with my visually impaired friends. I told most of the people close to me about my plans already (I'm really sorry for the people who I didn't tell yet. It's not something personal).

Since I want me and her to be boyfriend and girlfriend like other people, I will rent my own place at first. Next to that, I'm currently looking for a job as a Ruby developer in Stockholm, so if you happen to have connections there or you are looking for a developer in Stockholm, let me know. I deliberately decided to quit my current job, since I also want to build social life. I know it sucks for my current colleagues, but this is something I need to do. Some time ago (before I met her actually), I figured that I don't want to regret stuff I didn't do. Some things I did, I regret, but I learned from my mistakes, but not doing things that I could have done.. It's a whole different story.

My house, I will rent to someone. Preferably to a friend or an acquaintance, so if you would like live in Utrecht, 5 minutes from the central station, in the middle of the great neighborhood of Lombok, let me know. It's a two story house (about 85 m2) with a large living room, two bedrooms, kitchen, bathroom (with bath) and a small back yard. I'm considering renting it furnished, but this depends who is going to rent my house (I might do it through a rental company if I don't find someone I know).

Of course there will be things that I'm going to miss (think peanut butter, stroopwafels and other typical Dutch things), but most of all, I'll miss my friends and family. But since flying is cheap, I will still be able to see all of them. Because of the Internet, everybody is only 30ms away. The only thing is that if someone will visit me there, he or she needs to take alcohol, since in Sweden alcohol is very expensive. Swimming every Wednesday, I will miss. If there's anyone out there that wants to do some voluntary work and drive 2 or 3 people from Utrecht to Amersfoort once a week and swim for an hour, let me know. I've been doing this for over 2 years now and I still like it a lot.

Luckily people speak English very well in Sweden, but I've already started learning the Swedish language. I bought Rosetta Stone and I really like it. Swedish isn't so hard for Dutch people; there are a lot of similarities and the grammar is not so difficult (no cases, no conjugation of verbs), so I'll manage.

As said, I already told most people that are really close and I got great support from all of them. Since I've never done this before, I'm very open for advice from anyone. Over the coming months, I'll keep all of you updated on the progress I make on my blog and probably also on Twitter.

All in all, this feels like the right decision and I'm looking forward to living there, in a beautiful country, with a beautiful girl.

And oh, my cats will be moving too ;)

Edit: I forgot to mention the date. I'm moving before the 1st of February 2010

Published at September 7, 2009 in life, sweden, travel and work. 12 Comments

Jewel Labs

Over the last couple of months I've been starting new business with two friends. With one pal, I've been working on and off over the last couple of years and last autumn we decided to join forces in software development. We started developing a system for festival management (initially inspired by the lack of good systems for managing complex film festivals) and over the last months our product has taken shape. We're currently doing all the legal stuff, but we've already launched our website at http://www.jewellabs.net. Work will be done on the site over the coming weeks (some people requested screenshots of our product so I guess we'll have to make those), but I'm pretty pleased for now.

If you have any questions/comments/remarks/business proposals, let me know!

Published at April 6, 2009 in work. 0 Comments

Rails will_paginate plugin

I know, it has been a long time since I've blogged and actually people have complained about the fact that I didn't do this for a long time. Well, in short, I'm fine. A lot has happened since the last time I've blogged, but I'll write about that when I make the time somewhere in the upcoming weeks. Since I've been coding a lot with Rails lately, I just wanted to share some geek stuff with the rest of the world today.

In one of my projects, I'm using the will_paginate plug-in, which is pretty cool. Without a lot of stuff, it will give you pagination in your views. It also works for generic Arrays. Anyway, there is one problem. The project I'm working with can contain a lot of records per table. Since will_paginate doesn't act like a named scope (or at least not in this case, since we're not directly calling paginate() on an AssociationProxy, but on an Array) and requires the whole set returned. This is, because it needs the size of the array as well, so it will calculate the total amount of pages and records in the collection. Obviously, this can be quite expensive if you have a lot of records in your table. It will retrieve all records, create objects in the Array and finally cut of just a very small part of it to display. Because of this, we came up with the following solution:

# The amount of objects per page we want to show
limit = $OBJECTS_PER_PAGE

# If the page was set to 0 (shouldn't happen), or nil (yeah, nil.to_i == 0),
# set the offset to 0
offset_page = (page.to_i > 0) ? page.to_i - 1 : 0
offset = limit * offset_page

# Create args hash with which we count
count_args = args.dup

# Add the limit and the offset to the arguments hash
args[:limit] = limit unless args[:limit]
args[:offset] = offset unless args[:offset]

# Count all object that would be there
total_count = objects.count(:all, count_args)

# Find all objects (this is limited)
objects = find(:all, args)

# Create an array filled with nil values, the size of the total collection
objects_array = Array.new(total_count, nil)

# Insert the found objects at the right place in the array
objects_array[(offset)..(offset + limit - 1)] = objects

# Call paginate() on the array with limit and page   
return objects_array.paginate(:per_page => limit, :page => page)

What this does is only getting the required records, using :limit and :o ffset in find() and an additional count() without :limit and :o ffset. To make sure that will_paginate gets a collection the size of the total collection, we create an Array filled with nil values, the size of count(). Then we insert the found records in this array.

The only problem is that the whole operation isn't atomic. The count could differ from the real objects array.

I haven't run any benchmarks, but the Rails logs tell me that only a very small subset of the complete table is selected.

Update: Pointed out by Habbie, using SQL OFFSET is very slow. Back to the drawing board.

Published at February 18, 2009 in rails, ruby and work. 2 Comments

Office in a day

Since some time, I've been working on some software with a friend of mine, but since things accelerated over the last couple of months, we decided that it was time for a real office. We're both freelance geeks but we decided to join forces and start a new business. I'm currently working on an awesome application for a customer and the development for this is coming on very well, but since a freelancer can work more than 40 hours a week, I have plenty of time to do extra stuff. Most of the time, I work at home, with the ocassional meeting with my customers, but I sometimes notice that at home I'm distracted; doing some laundry, feeding the cats, getting some groceries, etc. And, the room I initially planned as my office is still full of stuff I haven't found a place for. I figured I needed a place that is a better working environment and to really get things going with my friend, we need to sit face-to-face a lot to discuss the design and implementation of our soon-to-be application. So, in the last months, we looked for office space and found it. Currently we're renting a place in Diemen for very little money.

Last week we got the keys and we wanted to get the whole thing up and running as soon as possible. Since my sister is an interior decorator and has her own company, we decided to give her carte blance on the interior. We're both male geeks and don't have a lot of knowledge on interior stuff, having my sister do the interior was a very good choice. Since she was thinking about advertising a new service called "office in a day", this was a perfect test case for her. Last week I made some photos of the place and sent them, together with a very basic map to her. On Sunday, she made a design and showed us Monday morning. We both were very excited about the design and the concept behind it, so we rushed to various stores to buy furnitures and office supplies. Since my sister has a big network of people in her business, we were able to get flooring tiles in two hours. We drove to the factory to pick them up and got the whole floor done in an hour or two. We put the furniture together and put the stuff in and the result is amazing! Test case succesful and customer happy! Here are the results:

Of course, I wouldn't be me if I didn't make a shameless plug, so, if you need an awesome interior designer, especially for your office needs, call my sister. You can find more information on her website at http://www.insida.nl

Published at October 29, 2008 in work. 1 Comment

Life update

Just a quick update on life. Currently, I'm very busy finishing a project for one of my customers, which will be done by the 14th of October. A new project is already in the pipeline, for which I'm very eager to start with. The project will be about implementing a brand new Ruby on Rails application and I really can't wait to do some hardcore coding again. Being a freelance developer allows me to pick the projects I really like and after 6 months of managing people and talking a lot in meetings, it's time to dive into some code again. Next to that, there is another upcoming project that I started with a good friend of mine last year, but is now getting very serious. The planning is to rent some office space and start coding to finish the product before January.

For the rest, things are going well. Last weekend I travelled to St. Petersburg again (for obvious reasons). The weekend was so great. The only thing we did was visit a birthday party of some friends and the rest was just spending time together. I decided to fly from Dusseldorf this time, since it was a lot cheaper than from Amsterdam and the scheduled times were also much better (Friday in the afternoon and back on late Sunday evening). I flew with BlueWings, a German budget company, and the service was nice.

The day after returning, I suffered my usual post-Russia depression, but now I'm doing fine. I know that this was not the last time.

Published at September 10, 2008 in life, russia, travel and work. 0 Comments

The return of the Suit

When I was working at McNolia, suits were part of the business culture. When I started there as techie, I went to work wearing jeans, a hooded sweater and Vans and often unshaved. After a while I started doing some project management in more corporate environments and it was expected that I wore a suit. For a friends wedding I once bought a suit, but it soon turned out that one suit wasn't enough, so I bought several. At the time, still trying to fight the corporate image, I didn't wear a tie. Since I had to deal with increasingly "important" people, the necktie came into my life and I learned how to make a good knot. On one hand, I didn't really like it, since the whole suit and tie thing seemed very hypocrite, but on the other hand, I made the suit look good. I started to understand that some people only take you seriously if you have the corporate image. Sad, but true. Business just works this way sometimes and I was in that particular business.

When I left McNolia, I started at NoXa as a Ruby on Rails developer. Since NoXa was part of a bigger recruitment company that had a very hip and young image, the suit went back into the closet. Since the average age was around 24 or so, the suit didn't really fit in, especially not for technical staff. Although I'm not really the fashion type, I did go to the hairdresser to get a funky haircut, to blend in a bit. The jeans and Vans were back.

A couple of months ago, I started my own business and to me, image is something to take care of, just like a logo, website and letterhead. I decided that I would continue the jeans and Vans image, combined with the funky haircut, and not to wear a suit, unless there was no other option. Last week, I signed a contract for a project at a big Dutch bank (obviously suit material) and took the suit out of the closet again. The company's image who hired me is somewhere in between the jeans and the suit (casual is the term, I guess), but when visiting our mutual customer, it's expected to dress in a representative way. The first meeting, I wore the suit with tie and all, but apparently this was a bit overdone, so today, the tie stayed home. Combined with the haircut and my designer-like glasses, I felt pretty good wearing the suit again. When visiting my customer, the jeans-vans-combo is used, when it's the bank, it's the suit-no-tie-thing. At home, it's jeans-vans all the way. But I guess the suit has returned.

Published at February 6, 2008 in life and work. 0 Comments

tuber


tuber

Na lang zoeken, had ik em dan eindelijk; een naam voor mijn eigen bedrijfje: Tuber. Het belangrijkste was natuurlijk dat het .nl domein nog niet weg, dus die meteen geregistreerd. Tuber is naast een synoniem voor 'root' ("bow before me, for I am root") ook "someone who works with tubes" en dat is precies wat ik doe, aangezien het internet niet meer is dan "a series of tubes". Inschrijving bij de KvK is gedaan, nu nog een logo, briefpapier, visitekaartjes, de hele rambam, maar dat komt goed!

Published at November 27, 2007 in life, techno stuff and work. 0 Comments

Zelfstandig ondernemer!

Vandaag markeert het officiële begin van een nieuwe stap in mijn carrière. Ik heb besloten om niet meer bij NoXa te blijven, maar als zelfstandig ondernemer verder te gaan. Omdat ik natuurlijk nog wel wat werk hier heb om af te maken en over te dragen en omdat er nog het een en ander geregeld moet worden (met name klanten :) ) zal ik me de komende 2 maanden hier op focussen. Omdat mijn huidige werkgever erg flexibel en ook enthousiast voor me is hoef ik niet mijn gehele opzeg termijn in dienst te blijven, maar kan ik (na overdracht) weg wanneer ik een klus heb. Erg ok dus. Mochten er dus lezers zijn die projectjes hebben, dan hoor ik dat natuurlijk graag. Werk aan de winkel dus!

Meer info over mijn skills is natuurlijk te vinden in mijn CV

Published at November 8, 2007 in life and work. 0 Comments
« Previous Entries