Modules, JARs, Class Loaders, and the "Class Path" - NetBeans API Javadoc 5.5.0

Introduction
The Class Loader Hierarchy
The System Class Loader, Thread Context Class Loading, and nbres:
Package Overlaps and "Parallel" Libraries
Examining Your "Class Path" and Preresolving Classes
Common Problems and Solutions
Why is This So Darn Complicated Anyway?

Introduction

NetBeans, as a large Java application with a sophisticated module system and a strong framework for maintaining inter-module compatibility, has a specialized infrastructure for physically loading the classes that produce the application. It uses class loaders to manage the interactions between modules.

While developers who are accustomed to working with application servers or other large componentized applications might already be familiar with the purposes and basic mechanisms of class loader partitioning, many Java developers only have experience with monolithic applications loaded entirely from the system class path (the -classpath parameter to the Java VM launcher). All developers seeking to write a NetBeans module should have some practical understanding of how NetBeans manages class loading.

This document attempts to explain the basic system by which classes are loaded into NetBeans; and how you as a module author can take advantage of the power this system offers, without becoming a victim to some common misperceptions and mistakes.

Everything written here is nonnormative, which is to say it is not a formal part of the Modules API specification. For precise guarantees as to what you can and cannot do, as well as details on several points of manifest syntax, please read the Modules API.

The Class Loader Hierarchy

The basic thing you need to understand about how modules control class loading is this:

If module B has a declared dependency on module A, then classes in B can refer to classes in A (but A cannot refer to B). If B does not have a declared dependency on A, it cannot refer to A. Furthermore, dependencies are not considered transitive for purposes of classloading: if C has a declared dependency on B, it can refer to classes in B, but not to A (unless it also declares an explicit dependency on A).

Remember: refer to here means refer to statically: compile and link against. Source code for B should be passed the JAR for A in its classpath when compiling. Module A can certainly use objects created by module B, if assigned to a suitably generic interface; it just cannot link against these specific classes in Java source code. The same system applies to classpath-oriented calls such as Class.getResource(String path).

This constraint is enforced using the parent-child relationship of class loaders. Each module has its own dedicated class loader. A NetBeans module classloader delegates only to the classloaders of modules which the module author asked to depend on.

This kind of relationship - a hierarchy of classloaders - is common in componentized Java applications. NetBeans takes the concept a little bit further: since a module can (and very often will) depend on several other modules, the classloader hierarchy is a directed acyclic graph, rather than a tree. This could be called multiple inheritance of classloaders.

There are a number of reasons for using such a system, rather than dumping everything in one big classpath. The main reason, though, is safety. If NetBeans let you use any class from any other module, you might begin relying on another module without even realizing it. This could cause sudden errors if that other module were missing from a user's installation. Declaring a module dependency prevents such a situation from arising.

Here is a trivial summary of the situation:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.a

public abstract class A {/* ... */}
// ...
public static void useSomeA(A a) {/* ... */}

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.b
OpenIDE-Module-Module-Dependencies: org.netbeans.modules.a

public class B extends A {/* ... */}
// ...
useSomeA(new B());

This basic rule covers the bulk of the situations you will encounter. There are some further reasons why you might not be able to access certain classes, but you can probably stop reading now unless you actually run into problems.

Extensions using `Class-Path`

Commonly a NetBeans module will serve as a wrapper for an independent library, or will use one or more separate libraries as part of its implementation or API. Forcing all classes and resources used by a module to be packed together into the module JAR is generally undesirable for reasons of maintenance and clarity. Sometimes it may even violate license terms of a library to modify it by repackaging it into a module JAR.

NetBeans handles this situation by letting you use the Class-Path manifest attribute defined by the Java Extension Mechanism. The value of the attribute consists of one or more relative paths to library JARs, in the same directory as the module or (normally) beneath it. Conventionally, such extensions are placed in a directory named ext/ beneath the directory where the module is. It is a good idea to name extension JARs precisely (i.e., include the version number).

For example, a module $NBHOME/modules/module.jar would refer to the extensions $NBHOME/modules/ext/library-a-1.0.jar and $NBHOME/modules/ext/library-b-1.1.jar this way:

Class-Path: ext/library-a-1.0.jar ext/library-b-1.1.jar

When you do this, the libraries are loaded from the same classloader as the main module JAR, just as if they had been unpacked and physically added to it. Classes from any of these JARs can refer to classes in the others. As a matter of style, however, you should generally refer to the libraries from the module but not refer to the module from the libraries. Modules dependent on a module with extensions can refer to its extensions too.

It is common for a module to just be a wrapper which has no classes itself, but merely refers to a library using Class-Path. Creating such a wrapper module blesses the library with the flexible deployment and maintenance qualities of a module. If there is no legal reason not to, you can also just insert NetBeans-specific manifest attributes in any JAR and use it as a module, without interfering with its use outside NetBeans.

Overuse of Class-Path is a common bad habit. Use module dependencies to express relationships between independent blocks of code; Class-Path should be used only when a JAR is a wholly owned part of a module.

Never refer to the same JAR using Class-Path from two different modules. (It will get loaded twice and could cause strange problems.) If you think you need to, generally this just means you have not factored out your module dependencies thoroughly: split off a third module to hold the library.

Never try to use Class-Path to refer to other module JARs, or to JARs in the NetBeans lib/ directory, or generally to any JAR you did not specifically bundle alongside your module and your module only (for example, in an NBM package for Auto Update).

Do not be alarmed by all the "never"s; these are good guidelines to protect you from common design mistakes, but the basic behavior of Class-Path is pretty simple:

If a module A has a Class-Path attribute listing some relative JAR paths, the effect is exactly the same as if the contents of those JARs were just copied into A's main JAR.

Startup Libraries

In a given NetBeans release, there are several libraries placed in the lib/ and lib/ext/ subdirectories of the NetBeans installation. These are referred to as startup libraries. They fall into a few categories in regard to how they may be used by modules.

API Libraries

lib/openide.jar holds the NetBeans Open APIs. All modules may use public classes in this JAR without doing anything special. The manifest tag OpenIDE-Module-IDE-Dependencies permits a module to request a specific version of the APIs and should always be used and kept current (i.e. request the oldest version of the APIs with which your module can run).

The Core

lib/core*.jar and lib/ext/boot.jar contain various pieces of the NetBeans core implementation. These are not available for use by modules. A handful of NetBeans modules have special reasons to interact directly with the core - for example, Auto Update sometimes needs to cooperate with it in order to install new modules - but the mechanisms for gaining access to core classes are not supported (subject to change without notice).

lib/updater.jar is used only by the Auto Update module, and is not accessible to other modules.

Patches and Localizations

In addition to use of Class-Path, there are a couple of other ways in which additional JARs might be added into the classloader for a module.

Patches are JARs which may replace classes and resources in a module classloader. If a module has the code name base a.b.c, and there is a subdirectory named patches/a-b-c/ beneath the directory in which the module JAR resides, then this patch area will be scanned for JAR files to be added to the "front" of the module classloader - anything found in them will override corresponding classes or resources in the module and its extensions.

This patching facility is occasionally useful for testing modifications to installed modules, or making emergency fixes in the field, without needing to touch the master copy of the module. Modules should not be shipped with any patches, however.

Localization and branding may also affect the module classloader. A module may load resources (such as property bundles, images, XML layers, and so on) using the recommended internationalized lookup techniques - NbBundle methods, the nbresloc: URL protocol, and so on. Actual localizations of these resources are best placed in separate JAR files, which must be named according to the locale and placed in a subdirectory named locale/ beneath the directory in which the module JAR resides. For example, a Japanese localization of a module $NBHOME/modules/module.jar would be named $NBHOME/modules/locale/module_ja.jar.

Analogously, a module may be branded to adjust it to use in a specific product, using essentially the same mechanism. The module just mentioned could have a few text changes made for a product branding named myapp by creating a file named $NBHOME/modules/locale/module_myapp.jar. Locales and brandings are orthogonal, so a Belgian French translation of text strings specific to MyApp Community Edition might be kept in $NBHOME/modules/locale/module_myapp_ce_fr_BE.jar.

All applicable locale and branding variants of module JARs are placed in the same classloader as the module itself. Only variant JARs which actually apply at runtime (according to the current locale and branding) will be loaded.

Eager and Autoload Modules, Test-Mode Modules, and Manually Installed Modules

While in many cases, modules live in the $NBHOME/modules/ directory, or the matching $NBUSER/modules/ part of the user directory, this is not always true, and it is worth mentioning how the location of a module defines the locations of associated JARs.

Extensions using Class-Path, patches, and locale and branding variants of a module are always located relative to the actual position of the module JAR, wherever that might be.

The user of a NetBeans installation has the option of installing a module JAR from any location on disk. Therefore, any extensions or locale variants must be found in corresponding locations, not inside the NetBeans installation.

Developers of NetBeans modules can additionally install them in test mode, where they are loaded in such a way that it is safe to reload them after making changes. (Currently this involves making a temporary copy of the JAR which is loaded in its stead.) Again, any extensions must be found relative to the actual position of the JAR, not placed in the NetBeans installation. Note: reloading a test module will generally not reload its extensions too. If you need to test changes to a module's extensions as well as the module itself, it is recommended that the test-mode JAR be built with all extension classes and resources directly included; the extensions can still be kept in separate JARs at production time.

Provide-Require Dependencies

In addition to regular dependencies using OpenIDE-Module-Module-Dependencies, modules can also depend on the existence of other modules using provide-require dependencies. Here, one or more modules provide some arbitrary token (often the name of an API class for which they supply implementations in Lookup); and other modules may request that this token be available. A requesting modules can be enabled only when at least one (perhaps more) providing module is enabled.

Provide-require dependencies have no effect on classloading. If module A provides token T, and module B requires token T, B may not refer to classes in A (unless it additionally declares a regular dependency on A).

Dependencies and Package Restrictions

By default when a module B declares a direct dependency on another module A, then B can access any public (or, as appropriate, protected) class in A - i.e. all of A is "in its classpath". However it is often the case that such broad access is undesirable. Module A may provide a formal API in certain packages, and use other packages for its private implementation. In this case it would not want other modules freely using its undocumented implementation classes.

For this reason, a special manifest attribute OpenIDE-Module-Public-Packages can be specified in A:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.a
OpenIDE-Module-Specification-Version: 1.0
OpenIDE-Module-Implementation-Version: 1.0-alpha-2
OpenIDE-Module-Public-Packages: org.netbeans.api.a.**, org.netbeans.spi.a.**

This attribute tells the NetBeans module system to limit which packages from A are considered part of its API. If B declares a regular dependency on A according to its public API specification version:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.b
OpenIDE-Module-Module-Dependencies: org.netbeans.modules.a > 1.0

then B can only use some packages from A: here, org.netbeans.api.a (and any subpackages it may have) and org.netbeans.spi.a (and subpackages). Module B will not be permitted to use other packages from A, such as org.netbeans.modules.a; attempts to do so will just result in a NoClassDefFoundError when loading code from B.

This manifest goes further and prevents any packages from being available to other modules:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.a
OpenIDE-Module-Public-Packages: -

You may still declare a dependency on such a module, in order to ensure that it is installed (perhaps it provides some non-Java-level service you need), but you may not import any classes from it.

There is a limited "back door" to this package restriction: if module B declares that it knows about module A's internal implementation, and is prepared to track arbitrary changes in A, then it can use any public classes from A regardless of the public package declaration. This is done using an implementation dependency:

Manifest-Version: 1.0
OpenIDE-Module: org.netbeans.modules.b
OpenIDE-Module-Module-Dependencies: org.netbeans.modules.a = 1.0-alpha-2

Here module B declares that it is written to match details of A's implementation classes at that point in time. This version of B may not use any earlier or later versions of A, since there is no telling what changes to A's implementation classes there might have been. Therefore, such implementation dependencies are usually used among "friend modules" which are updated and published in clusters by a single developer or team.

Again, dependencies are not considered transitive for purposes of class loading; so package visibility from module A to module C (where C depends on B) is entirely independent of visibility from A to B - it can be computed entirely from the manifests of A and C.

Public package declarations also apply to packages contained in Class-Path extensions - just as if those classes were in the main module JAR.

In summary:

If module A declares that it has a particular list of public packages, and module B declares a direct dependency on A, B can always use classes from A's public packages (if any). B may only use undeclared packages from A (implementation classes) if it declares an implementation dependency on the exact version of A.

The System Class Loader, Thread Context Class Loading, and `nbres:`

There is a special class loader in NetBeans referred to as the system class loader. This loader can load classes from any enabled module, as well as the JRE/JDK and all startup libraries (APIs and core). Therefore it is useful as a default class loader to be used by any code which tries to find a class by name without specific knowledge of where it may be located.

Finding the system class loader is easy using lookup:

ClassLoader syscl = (ClassLoader)Lookup.getDefault().lookup(ClassLoader.class);

This class loader is also the context class loader for every thread in the NetBeans VM, unless that thread (or a parent) explicitly set some other context class loader. Since many libraries which are independent of NetBeans (including in the JRE) are written to assume that all relevant classes can be loaded by name from the current thread's context loader, it is very useful for this loader to be the system class loader - you can specify any class in your module by name.

(More helpful information: Understanding Class.forName())

ClassLoader l = Thread.currentThread().getContextClassLoader();
Class c = l.loadClass("some.module.Class");

Any module class can be accessed from this loader, regardless of any OpenIDE-Module-Public-Packages declarations.

The system loader always represents the contents of the enabled modules in NetBeans. If a module is newly enabled at runtime, its classes are effectively added to the namespace of the same system ClassLoader instance. (Note that this means that a call to Class.forName which fails before the module is enabled may succeed afterwards.) However, if a module is disabled at runtime, the system class loader is reset to a new loader which does not have access to the old module. This is necessary because it is impossible to remove classes from a loader once they have been loaded. After a module is disabled and the loader reset, a saved Lookup.Result query on ClassLoader will fire a lookup result change, and all threads will be updated to get the new context class loader too.

It is often useful to be able to refer to resources other than classes in the system class loader. This is easy to do because NetBeans defines a special URL protocol handler for just this purpose. URLs with the nbres protocol refer to resources in the system loader, and thus can refer to resources present in any module JAR. This is very useful when some declarative syntax requires a URL that should look in a module; for example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN"
                            "http://www.netbeans.org/dtds/filesystem-1_1.dtd">
<!-- Register one DTD in the system entity catalog. -->
<filesystem>
    <folder name="xml">
        <folder name="entities">
            <folder name="NetBeans">
                <file name="DTD_Foo_1_0"
                    url="nbres:/org/netbeans/modules/foo/resources/foo-1.0.dtd">
                    <attr name="hint.originalPublicID"
                          stringvalue="-//NetBeans//DTD Foo 1.0//EN"/>
                </file>
            </folder>
        </folder>
    </folder>
</filesystem>

There is also a related protocol nbresloc which loads from the system class loader but additionally performs automatic localization and branding of the resource you specify. Various suffixes are inserted between the base name and the extension of the resource (beginning with the last dot in the path, if it is in the last path component), according to the current locale and branding. For example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE filesystem PUBLIC "-//NetBeans//DTD Filesystem 1.1//EN"
                            "http://www.netbeans.org/dtds/filesystem-1_1.dtd">
<!-- Add a URL to the Help menu with a localized name and icon. -->
<filesystem>
    <folder name="Menu">
        <folder name="Help">
            <file name="netbeans-web-link.url" url="netbeans-web-link.url">
                <attr name="SystemFileSystem.localizingBundle"
                      stringvalue="org.netbeans.modules.url.Bundle"/>
                <attr name="SystemFileSystem.icon"
                     urlvalue="nbresloc:/org/netbeans/modules/url/webLink.gif"/>
            </file>
        </folder>
    </folder>
</filesystem>

Here NetBeans will look for the most specific applicable icon; it might for example find org/netbeans/modules/url/webLink_ja.gif in modules/locale/utilities_ja.jar, if Japanese localizers decided the default icon was not intuitive for Japanese users and made a replacement. Whenever you are asked for a URL to a displayable resource, consider using nbresloc in place of nbres.

Package Overlaps and "Parallel" Libraries

XXX

Examining Your "Class Path" and Preresolving Classes

XXX - also org.netbeans.core.modules=0, printing class loaders, code source, ...

Common Problems and Solutions

This section is an attempt to gather a number of the problems, questions, and misperceptions that people using the NetBeans module system have run across. In some cases an error is caused by a simple mistake that can be corrected just as simply; in other cases, you may need to consider the design of your module and how it can work smoothly in NetBeans.

I am confused by the difference between the module "class path" and the Filesystems tab in the Explorer

XXX

My module seems to run, but there is a warning message in the log file

XXX

My module compiles, but throws `NoClassDefFoundError` when I run it

XXX

I get a `ClassCastException`, but the "incorrect class" is actually the one I cast to

XXX

My module stopped working after I ran an obfuscator on it

XXX

I can only load some classes in a certain package that I have in two modules

XXX

I am getting warnings when trying to access resources from the default (root, unnamed) package

Loading classes from the default (root) package is disabled. You can load resources from that package but a warning is printed. It is strongly discouraged to use the default package. Please see the Java Language Specification for more details. You can suppress the warning using -J-Dorg.netbeans.do_not_warn_default_package=true command line switch.

I tried to override a common library with my own version but I get the wrong copy of classes

XXX

I have a dependency on another module but still cannot access any/some classes in it

XXX

Patching someone else's module using `lib/patches/` gave me errors

XXX

I am trying to use internal execution and I get a strange warning in the status bar

XXX

I used `Class-Path` from my autoload module but it has no effect

XXX

`Class-Path` works in my module if I install it normally, but not in test mode

XXX

I want two or more of my modules to be able to refer to each other, but they cannot both do so

XXX

A little bit of one of my modules depends on another, but I do not want a dependency just for this

XXX

I need to supply two different versions of the same library

XXX

I made my own `ClassLoader` and now I just get `SecurityException`s when I try to use it

XXX

Users ought to be able to add JARs to my module's class path to extend it

XXX

Users should be able to compile and/or run their own apps with some of my JARs in the class path

XXX

I would like to use `%CLASSPATH%` / `$CLASSPATH`

XXX

Using `lib/ext/` or `-cp` sounds much easier but someone told me not to do it

XXX

I need to install a JDBC driver, JNDI provider, Logging implementation, Look & Feel, XML parser, XSLT processor, ...

XXX

Some of my code uses JNI-based native libraries but I get `UnsatisfiedLinkError`s

XXX

I had problems reloading a module using JNI

XXX

I want a clean architecture to separate my API/SPI, one or more implementations, a GUI, and clients

XXX

I need to add a library JAR from outside the NetBeans installation

Q: Can my module add a library JAR to the classpath from outside the IDE installation? For example, I have an application called Etch-a-Kvetch for modeling customer call response systems, and I want to build a module to integrate it into the IDE. The user may already have Etch-a-Kvetch installed on their disk, and I want to reuse the eak.jar main library JAR in my module. It is not present in the IDE's installation directory. Can I add it to the IDE's classpath so my module can use it?

A: Not easily. You have a few options:

Add an entry to ide.cfg. For example:
```
-cp:a c:\eak\lib\eak.jar
```
This startup file provides the ability to add classpath entries to the IDE's Java invocation.

Pros: Adds the library as desired. Cons: Very fragile. Assumes that the ide.cfg is actually read, which is not guaranteed for all platforms. Easy to clobber existing user customizations. Dependent on the exact structure of the IDE's bin/ directory, which has changed quite a bit in the past and could change again. Places library in startup classpath, whereas it is preferred to have it in the module classloader only. Must be done before the IDE starts, requiring some kind of manual installation step and making updating your module very difficult.
Duplicate eak.jar in modules/ext/. That is, ship a copy of the required JAR file inside the IDE installation, and continue to refer to dynamic resources in the desired external location.

Pros: Simple to implement and should be reliable. Version of the library shipped can be controlled to match that which the module was compiled against, avoiding potential version skew. Cons: Impractical when the library is physically very large, or there are licensing issues involved in redistributing it. Bugfix upgrades to the master library may not be reflected in the IDE's copy.
Partition your module and use a new classloader. In other words: logically divide your module into two halves. The first half will form a compilation unit depending on the Open APIs and will contain all of the classes referred to directly in the module manifest (the installer, data loaders, or whatever you have). It should contain one or more interfaces or similar constructs which describe what it expects the second half to implement. The second half will form a separate compilation unit, depending on the first half, possibly the Open APIs, and also eak.jar. It should contain implementations of the interfaces specified in the first half. The first half, at install/restore time (or lazily when any functionality is needed) should create a new URLClassLoader whose parent should be the first half's classloader, and including as URLs the locations of both the second half as a JAR and the library JAR; using this classloader, look up the implementation classes by name; create new instances of them; cast them to the interface types; and begin using them. The second half should probably be placed in a separate JAR file; if not, it will be necessary to subclass the dynamic classloader to not delegate class loads for the implementation packages to its parent. Either way, it is strongly recommended that the build process enforce that the first half compile without reference to the second.

Pros: Compliant with the Open APIs and reliable. The library JAR may be located anywhere at runtime and could even be moved or replaced at runtime. Cons: Potentially complex to implement. Some use of reflection required, though it should be safe. More complex build and test procedure for the module. The partition between the two halves must be carefully chosen, especially for large modules, to minimize the complexity of the interfaces and push as much implementation as possible to one side or the other.

Oyetunde Fadele has kindly posted a detailed explanation of how to use this technique in practice: see his message long ago on dev@openide.netbeans.org.

Why is This So Darn Complicated Anyway?

XXX

Modules, JARs, Class Loaders, and the "Class Path" - NetBeans API Javadoc 5.5.0

Contents

I am confused by the difference between the module "class path" and the Filesystems tab in the Explorer

My module seems to run, but there is a warning message in the log file

My module compiles, but throws NoClassDefFoundError when I run it

I get a ClassCastException, but the "incorrect class" is actually the one I cast to

My module stopped working after I ran an obfuscator on it

I can only load some classes in a certain package that I have in two modules

I tried to override a common library with my own version but I get the wrong copy of classes

I have a dependency on another module but still cannot access any/some classes in it

Patching someone else's module using lib/patches/ gave me errors

I am trying to use internal execution and I get a strange warning in the status bar

I used Class-Path from my autoload module but it has no effect

Class-Path works in my module if I install it normally, but not in test mode

I want two or more of my modules to be able to refer to each other, but they cannot both do so

A little bit of one of my modules depends on another, but I do not want a dependency just for this

I need to supply two different versions of the same library

I made my own ClassLoader and now I just get SecurityExceptions when I try to use it

Users ought to be able to add JARs to my module's class path to extend it

Users should be able to compile and/or run their own apps with some of my JARs in the class path

I would like to use %CLASSPATH% / $CLASSPATH

Using lib/ext/ or -cp sounds much easier but someone told me not to do it

I need to install a JDBC driver, JNDI provider, Logging implementation, Look & Feel, XML parser, XSLT processor, ...

Some of my code uses JNI-based native libraries but I get UnsatisfiedLinkErrors

I had problems reloading a module using JNI

I want a clean architecture to separate my API/SPI, one or more implementations, a GUI, and clients

I need to add a library JAR from outside the NetBeans installation

My module compiles, but throws `NoClassDefFoundError` when I run it

I get a `ClassCastException`, but the "incorrect class" is actually the one I cast to

Patching someone else's module using `lib/patches/` gave me errors

I used `Class-Path` from my autoload module but it has no effect

`Class-Path` works in my module if I install it normally, but not in test mode

I made my own `ClassLoader` and now I just get `SecurityException`s when I try to use it

I would like to use `%CLASSPATH%` / `$CLASSPATH`

Using `lib/ext/` or `-cp` sounds much easier but someone told me not to do it

Some of my code uses JNI-based native libraries but I get `UnsatisfiedLinkError`s