Masterzen’s Blog

Puppet Memory Leaks… Or not…

3 minute read

From time to time we get some complaints about so-called Puppet memory leaks either on #puppet, on the puppet-user list or in the Puppet redmine.

I tried hard to reproduce the issue on the Days of Wonder servers (mostly up-to-date debian), but never could. Starting from there I tried to gather from the various people I talked to on various channels what could be the cause, if they solved it and how.

You also can be sure there are no memory leaks in the Puppet source code. All of the identified memory leaks are either not memory leaks per-se or are caused by an out of puppet control code base (ruby itself or a library).

Watch your Ruby

It is known that there are some ruby versions (around 1.8.5 and 1.8.6) exhibiting some leaks of some sort. This is especially true for RHEL 4 and 5 versions (and some Fedora ones too), as I found with the help of one Puppet user, or as others found.

Upgrading Ruby to 1.8.7-pl72 either from source or any repositories is usually enough to fix it.

Storeconfigs and MySQL

I also encountered some people that told me that storeconfigs with MySQL but without the real ruby-mysql gem, lead to some increasing memory footprint for their puppetmaster.

Storeconfigs and Rails < 2.1

It seems also to be a common advice to use Rails 2.1 if you use storeconfigs. I don’t know if Puppet uses this, but it seems that nested includes leaks in rails 2.0.

Is it really a leak?

The previous items I outlined above are real leaks. Some people (including myself) encountered a different issue: the puppetmaster is consuming lots of memory while doing file transfer to the clients.

In fact, up to Puppet 0.25 (not yet released at this time), Puppet is using XMLRPC as its communication protocol. Unfortunately this is not a transfer protocol, it is a Remote Procedure Call protocol. It means that to transfer binary files, Puppet has to load the whole file in memory, and then it escapes its content (same escaping as URL, which means every byte outside of 32-127 will take 3 bytes). Usually that means the master has to allocate roughly 2.5 times the size of the current transferred file.

Puppet 0.25 will use REST (so native HTTP) to transfer files, which will bring speed and streaming to file serving. Hopefully, if the Garbage Collector has a chance to trigger (because your ruby interpreter is not too much loaded), it will de-allocate all these memory used for files.

If you are not so lucky, the ruby interpreter don’t have time to run a full garbage cycle, and the memory usage grows. Some people running high-load puppetmaster have separated their file serving puppetmaster from their config serving puppetmaster to alleviate this issue.

Also, if like me you are using file recursive copy, you might encounter Bug #1469 File recursion with a remote source should not recurse locally.

I still have a leak you didn’t explain

Here is how you can find leaks in a ruby application:

• On DTrace enabled platform, and with the DTrace Toolkit Ruby Script you can have a better view of the generated/freed objects.
• Using GDB to inspect a live Ruby process
• Using Ruby Bleak House, this is a ruby gem which builds a specially patched ruby interpreter that can print leaked objects

I tried the three aforementioned techniques, and found that the GDB trick is the easier one to use and setup.

Another Ruby?

There’s also something that I think hasn’t been tried yet: running Puppet under a different Ruby interpreter (we’d say Virtual Machine in this case). For instance JRuby is running on top of the Java Virtual Machine which has more than 14 years of Garbage Collection development behind it. You also can be sure than a different Ruby interpreter won’t have the same bug or memory leak as the regular one (the so called Matz Ruby interpreter from the name of his author).

There are some nice Ruby VM under development right now, and I’m sure I’ll blog about using Puppet on some of them soon :-)

How do you like your Mocks served?

2 minute read

I like them refreshing, of course:

Mockito is the new Java mock library on the block, with lots of interesting features. It replaced JMock in almost all my Java projects, mainly because:- the syntax produces_ clear and readable test code_ (see below for an example), because it doesn’t abuse of anonymous class and methods are really methods.

• stub and verification happens logically, and at different place
• no replay or framework control methods ala EasyMock
• fully integrated to Junit (using @RunWith for instance)
• helpful annotations to create mock automagically
• it promotes simple tests by nature (and that’s essential to my eyes)

Basically, you can only do two things with Mockito:

• stub, or
• verify :-)

Enough discussion, let’s focus on an example:

@Test
public void itShouldComputeAndSetThePlayerRank() 
{  
  // creating a mock from an interface  
  // is as easy as that:  
  Player p = mock(Player.class)  
  
  // stub a method  
  when(p.getScore()).thenReturn(5);  
  
  // our imaginary SUT  
  ELOCalculator.computeRank(p);  
  
  // let's verify our rank has been computed  
  verify(p).setRank(12);
}

Due to its use of Generics and Java 5 autoboxing, the syntax is very clean, clear and readable. But that’s not all, Mockito provide a Junit 4 runner that simplifies mock creation with the help of annotations:

@RunWith(MockitoJUnit44Runner.class)
public class OurImaginaryTestCase
{  
  @Mock
  private Player player;  
  
  @Test  
  public void playerShouldBeRanked()  
  {     
    // we can use player directly here,     
    // it is mocked to the Player Interface  
  }
}

Of course during the verification phase of the test you can check for

• the number of calls (or check for no calls at all)
• the arguments (Mockito defines lots of useful arguments matcher, and you can plug any Hamcrest matchers),
• the call order,
• and for stubbing, you can also throw exception, return values, or define callbacks that will be called when a return value is needed.

In a word it’s really powerful. It is also possible to spy on concrete objects however as the manual says this is not partial mocking:

so you can’t use this method to check that the method under test calls other methods of the same object.Here’s an example of what I mean (the following test passes):

public class RealObject 
{
  public int a() 
  { 
    return 10; 
  } 
  
  public int b() 
  { 
    return 20 + a(); 
  }
}

@Test
public final void test1()
{ 
  RealObject real = new RealObject(); 
  RealObject spy = spy(real);
  when(spy.a()).thenReturn(12); 
  
  // notice the 30 here 
  assertThat(spy.b(), equalTo(30));
}

See Mockito author’s last blog post about the subject or this mockito mailing list post.

Basically the code should be refactored or we could use a subclass to overcome this.

There is also a debate about stubbing and verifying (the same call). Usually you don’t want to do that. Stubbing should be enough, if your code succeed then the call was implicitly verified. So usually if you stub there is no need to verify, and if you verify you don’t need to stub (except if you need to return something critical to the rest of the code, in which case you don’t need verification). Once again, Mockito’s author has a great post on the_stubbing or verifying debate_.

Of course if you are an Eclipse user, do not forget to add to the list of Favorites all Mockito static import, so that Content Assist knows all the matchers.

Happy unit testing with Mockito :-)

Net-Snmp doesn’t detect interface speed on Linux

less than 1 minute read

Have you ever wondered why net-snmp doesn’t report a ccomments: true orrect interface speed on Linux?

I was also wondering, until this morning…

I tried to run net-snmp as root, and miracle, the right interface speed was detected for my interfaces. In fact net-snmp uses the SIOCETHTOOL ioctl to access this information. Unfortunately the get settings variant of this ioctl needs to have the CAP_NET_ADMINenabled.

Of course root has this capability set, but when net-snmp drops its privileges to an unprivileged user, this capability is lost and the ioctl fails with EPERM.

That’s too bad because getting this information is at most harmless and shouldn’t require special privileges to succeed.

Someone even posted a Linux Kernel patch to remove CAP_NET_ADMIN check for SIOCETHTOOL which doesn’t seem to have been merged.

The fix could also be on the snmpd side before dropping privileges.

The workaround is to tell net-snmp how the interface are looking:

interface eth0 6 10000000
interface eth1 6 100000000

Here I defined eth0 as a 100mbit/s FastEthernet interface, and eth1 as a GigabitEthernet interface.