Making OpenDaylight Helium successfully run on OSv

As discussed in a previous blog post, CloudRouter provides an OpenDaylight OSv image. The OSv project maintains a repository of make files and glue code needed to run various applications on OSv. This included OpenDaylight Hydrogen, but not the current OpenDaylight Helium release. We thought we’d submit a quick patch upstream to upgrade to Helium, but it turned out to be a little more tricky than we first thought.


First, we should take a quick look at capstan, the tool used to simplify building and running OSv application images. The process of building the application is driven by a Makefile, using standard Makefile syntax. Here is a snapshot of the current OpenDaylight Helium Makefile for OSv:

.PHONY: module

VERSION := 0.2.3-Helium-SR3
URL :=$(VERSION)/distribution-karaf-$(VERSION).zip

module: ROOTFS

        wget $(URL)

opendaylight: distribution-karaf-$(VERSION).zip
        unzip -x distribution-karaf-$(VERSION).zip

ROOTFS/opendaylight: opendaylight
        mkdir -p ROOTFS/opendaylight
        cp -a distribution-karaf-$(VERSION)/bin distribution-karaf-$(VERSION)/configuration distribution-karaf-$(VERSION)/data distribution-karaf-$(VERSION)/deploy distribution-karaf-$(VERSION)/etc distribution-karaf-$(VERSION)/externalapps distribution-karaf-$(VERSION)/lib distribution-karaf-$(VERSION)/ distribution-karaf-$(VERSION)/system ROOTFS/opendaylight

ROOTFS: ROOTFS/opendaylight

        rm -rf distribution-karaf-$(VERSION) ROOTFS

Since OpenDaylight is a distributed as a pre-compiled Java application, the Makefile just needs to download and extract an OpenDaylight zip distribution. The process of running the application is driven by a Capstanfile, using a syntax similar to Dockerfiles. Here is a snapshot of the current OpenDaylight Helium Capstanfile:

base: cloudius/osv-openjdk

cmdline: >
  -classpath /opendaylight/lib/karaf.branding-1.0.3-Helium-SR3.jar:/opendaylight/lib/karaf-jaas-boot.jar:/opendaylight/lib/karaf.jar:/opendaylight/lib/karaf-jmx-boot.jar:/opendaylight/lib/karaf-org.osgi.core.jar

build: make

The base element specifies that this image should be built on top of the existing osv-openjdk image. This automatically provides the JDK environment necessary to run OpenDaylight. The cmdline element specifies the command to be executed in order to run OpenDaylight.

The problem: ClassNotFoundException

After modifying the Makefile and Capstanfile to upgrade to OpenDaylight Helium, the capstan build process worked fine, but the image would throw a fatal exception at runtime:

$ capstan run opendaylight
Created instance: opendaylight
OSv v0.16
chdir: Quota exceeded
OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack location - find_vma failed
Error occurred during initialization of VM
java.lang.Error: java.lang.
ClassNotFoundException: io.osv.OsvSystemClassLoader
    at java.lang.ClassLoader.initSystemClassLoader(
    at java.lang.ClassLoader.getSystemClassLoader(
Caused by: java.lang.ClassNotFoundException: io.osv.OsvSystemClassLoader
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(
    at Method)
    at java.lang.ClassLoader.initSystemClassLoader(
    at java.lang.ClassLoader.getSystemClassLoader(

Note that the “chdir: Quota exceeded” error was transient and unpredictable, but the ClassNotFoundException was consistent. Usually a ClassNotFoundException is due to a dependency missing on the classpath. After verifying that the classpath contained all the necessary classes, we were left unsure what was causing the exception. We initially thought it could be a known issue related to the sequence of arguments provided, but that was not the culprit.

The solution: java.ext.dirs

After much head scratching, we realized that io.osv.OsvSystemClassLoader is loaded as a Java extension. This means it is not resolved via the classpath, but via the path specified by the java.ext.dirs property. The initial OpenDaylight Helium Capstanfile we wrote contained:


By specifying a new java.ext.dirs value, the default value of /usr/lib/jvm/java/jre/lib/ext:/usr/java/packages/lib/ext was being overriden. The fix was to add these paths to the value specified in the Capstanfile:


With this fix in place, a patch was submitted upstream to complete the upgrade to OpenDaylight Helium. The CloudRouter project now provides a distribution of OpenDaylight Helium running on OSv. For more information, see the CloudRouter wiki.