Objectives

Recently, two paradigms have evolved that are especially aimed at this type of problems: Plugins[1] and Components. But, in order to understand why the need for this technology arose, we can compile, from the above problems, a list of objectives we would like to fulfil. It is easy to see, looking at the nature of the problems and the following list, why these new paradigms have become so popular.

Modularity

Modularity is one of the keywords. How to divide an utility in different parts that obey the rules of modularity (independence, encapsulation) and interact with each other in a desired way. A special case would be to divide an utility in its main part (management) and its several branches, thus, separating functionality.

Dynamic functionality independent of the main part

In other words, to be able to add/change/remove functionality of an utility without touching the main part. As an add-on, to be able to do that on runtime. This implies that the application shouldn't be rebuild and, further more, compiled, each time these changes take place.

Independent development of parts

The development of each part of a project should be independent from the other ones. This obliges the communication between parts to be transparent and standard. Also, each part should be able to cope with the others being in different platforms and locations. Basically, each part shouldn't have to worry about how others are implemented and located.

Communication between applications

This means being able to embed an application A that provides a desired functionality inside an application B thus reusing code and providing a lighter and simpler application B.

These are, of course, very different objectives which rarely are assembled together. This is not intended to find a unique solution that covers all of them.

Using plugins and components enables us to fulfil this objectives in a clever and transparent way (when used correctly). How this is done is the subject of the next chapters. But first it is necessary to have a look at shared libraries.

Basic concepts

Basically, a library is a set of functions and/or structures to be used by applications. For example, if a programmer wishes to use more complex mathematical functions in C like sin he would have to use the library libm where this and other functions are defined. To do so, he can link the library in the compilation stage, using the -l option with the name of the library (without the lib prefix and the extension: in this case, -lm).

A shared library opposes to a static one in the sense that the library is only loaded at runtime and not in the compilation stage. So, while in static libraries the program carries the library code, when linked to a shared one the library code is not present and is only loaded into memory when needed.

One of the named advantages of shared libraries, and also another characteristic that differs them from static ones, is the fact that the system has, at most, one copy of the library's code per library. When loading, the system first checks if there is already a copy of the library on memory. The code is only copied if there is none, while as in static the code is included in every compiled program to which the library is linked.

So, if the program doesn't carry the library's code, when is it loaded? There are two alternatives:

• automatically at the start of the process and, thus, only unloaded at the end: the library must be linked at compilation stage;

• by request of the program itself (using, for example, dlopen()): the code can be loaded and unloaded in the middle of the process or not even loaded at all; furthermore, the library is not linked at compilation stage;

A small example at the end of this section will illustrate these concepts.

Advantages and disadvantages

But why use a shared library instead of a static one? And why use demand loading instead of automatic loading or the other way around? Shared libraries bring a number of advantages...

• libraries are only loaded at run time (thus, the possibility to change the library without having to recompile the end application);

• lighter applications and saving of space (since the application doesn't carry the library's code);

In particular, demand loading of a shared library can be very useful

• as stated before it is the application itself which requests the loading: the code can be loaded and unloaded at any given stage or not loaded at all; this is particularly useful when the program may use resources from several libraries but not necessarily at that moment, thus, avoiding that dozens of libraries that are not being used would be wasting memory;

• since the library is not linked at compilation stage the application doesn't have to know it beforehand: for example, all the libraries can be put in a certain directory where the application will fetch them; this enables that resources can be added or changed without touching the application itself;

But they have a cost. Also, some problems may arise and should be taken in account.

• First, errors only show at runtime. This makes the debugging more difficult but also imposes some care on moving libraries and changing them;

• Creating a shared library is more complex and requires a few number of flags and details. However, this can be easily overcome (see Libtool);

• The dynamic loading obliges the code to be Platform Independent Code(PIC). This makes the library heavier so, although applications are lighter, space is only actually saved if there is more than one application linked or there is demand loading;

• The resources that demand loading enables (see above) also bring dangerous situations:

  • What happens if the program doesn't find the library expected? The error can occur at anytime of the process life and not at the beginning as in automatic loading;

  • How does the program know which interface the library provides? If it is defined beforehand, how to ally versioning and heterogeneity of libraries with this prior knowledge without having to touch the application?

Developing a shared library

Developing a shared library, as is, is generally more difficult than developing a static one. The complexity increases in a great amount when trying to make it platform independent and easily built (a good user interface). There are, however, some very powerful utilities that not only simplify this procedure but are also greatly recommended since they produce a standard, compliant and robust final product. Their use is quite straightforward and apply to static ones as well.

    $ ./configure
    $ make
    $ make install

A small example

To illustrate, here is the development of a very small library, libgarfield.so, from the files eating.c and sleeping.c with the header file garfield.h. An executable called garfield is created by linking main.c to libgarfield.la (which is a wrapper to libgarfield.so).

Autoconf and automake are not used here, only libtool. It is presented the steps for the most common actions; see the libtool manual for the explanation of these steps and other possible actions that can be performed (see Links and further information).

Creating object files. Note that besides the normal object files it is also created .lo files which are PIC object files needed when creating the shared library.

    $ gcc -g -O -c main.c 
    $ libtool gcc -g -O -c eating.c 
    mkdir .libs
    gcc -g -O -c eating.c  -fPIC -DPIC -o .libs/eating.lo
  5 gcc -g -O -c eating.c -o eating.o >/dev/null 2>&1
    mv -f .libs/eating.lo eating.lo
    $ libtool gcc -g -O -c sleeping.c 
    rm -f .libs/sleeping.lo
    gcc -g -O -c sleeping.c  -fPIC -DPIC -o .libs/sleeping.lo
 10 gcc -g -O -c sleeping.c -o sleeping.o >/dev/null 2>&1
    mv -f .libs/sleeping.lo sleeping.lo

Linking libraries. The libraries are put in the .libs/ directory and libgarfield.la is created in place which, as was said, is a wrapper for the shared library.

    $ libtool gcc -g -O -o libgarfield.la eating.lo sleeping.lo \
    	 -rpath /usr/local/lib
    rm -fr .libs/libgarfield.la .libs/libgarfield.* .libs/libgarfield.*
    gcc -shared  eating.lo sleeping.lo  -lc  -Wl,-soname -Wl,libgarfield.so.0 -o .libs/libgarfield.so.0.0.0
  5 (cd .libs && rm -f libgarfield.so.0 && ln -s libgarfield.so.0.0.0 libgarfield.so.0)
    (cd .libs && rm -f libgarfield.so && ln -s libgarfield.so.0.0.0 libgarfield.so)
    ar cru .libs/libgarfield.a  eating.o sleeping.o 
    ranlib .libs/libgarfield.a
    creating libgarfield.la
 10 (cd .libs && rm -f libgarfield.la && ln -s ../libgarfield.la libgarfield.la)

Linking the executable. The executable is put inside .libs/ and a wrapper script is created in place.

    $ libtool gcc -g -O -o garfield main.o libgarfield.la 
    gcc -g -O -o .libs/garfield main.o .libs/libgarfield.so -Wl,--rpath -Wl,/usr/local/lib
    creating garfield

Installing libraries. libgarfield.la is also installed with the static and shared libraries. All the necessary operations are usually done (like, for example, calling ldconfig) but running libtool -n --finish libdir will give further hints on what to do (in this example, the installation command already gives this information).

    $ libtool install -c libgarfield.la /usr/local/lib/libgarfield.la
    install -c .libs/libgarfield.so.0.0.0 /usr/local/lib/libgarfield.so.0.0.0
    (cd /usr/local/lib && rm -f libgarfield.so.0 && ln -s libgarfield.so.0.0.0 libgarfield.so.0)
    (cd /usr/local/lib && rm -f libgarfield.so && ln -s libgarfield.so.0.0.0 libgarfield.so)
  5 install -c .libs/libgarfield.lai /usr/local/lib/libgarfield.la
    install -c .libs/libgarfield.a /usr/local/lib/libgarfield.a
    ranlib /usr/local/lib/libgarfield.a
    chmod 644 /usr/local/lib/libgarfield.a
    PATH="$PATH:/sbin" ldconfig -n /usr/local/lib
 10 ----------------------------------------------------------------------
    Libraries have been installed in:
       /usr/local/lib
    
    If you ever happen to want to link against installed libraries
 15 in a given directory, LIBDIR, you must either use libtool, and
    specify the full pathname of the library, or use `-LLIBDIR'
    flag during linking and do at least one of the following:
       - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
         during execution
 20    - add LIBDIR to the `LD_RUN_PATH' environment variable
         during linking
       - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
       - have your system administrator add LIBDIR to `/etc/ld.so.conf'
    
 25 See any operating system documentation about shared libraries for
    more information, such as the ld(1) and ld.so(8) manual pages.
    ----------------------------------------------------------------------

Other actions like debugging are also easily done and take advantage of the wrapper scripts.

Links and further information

Autoconf and it's manual can be found at http://www.gnu.org/software/autoconf/autoconf.html.

Automake and it's manual can be found at http://www.gnu.org/software/automake/automake.html.

Libtool's homepage is located at http://www.gnu.org/software/libtool/libtool.html.

The man pages of dlopen , ld, ldconfig also provide useful information.

General description and implementation information on shared libraries can be found in Building and Using Static and Shared "C" Libraries and Shared Libraries (Section 4).

In the GLib Reference Manual, there is a section describing the gmodule library, Dynamic Loading of Modules.

Concept and examples

The following image tries to illustrate this concept: an independent application where units of software, intended to complement the functionality, can be "plugged" to.

Using plugins is becoming very popular, especially in programs that process heterogeneous data and would, otherwise be less flexible and heavier. Examples are:

• The use of the Mime-type methodology in Netscape and other web browsers: this enables that special data like audio or video can be present on a web page and viewed accordingly;

• Manipulation of images and scripting in GIMP (GNOME);

• Translation from other formats (e.g.: Excel) and extension of the functionality in Gnumeric (GIMP), e.g., adding a statistic function not included;

• Acrobat, Photoshop, AutoCad...

Pros and cons

Plugins are a good way of adding functionality to an independent application: they can be loaded and unloaded during runtime and there is no need to recompile the main application when adding a new plugin. This way we can have a main application lighter and focused on the "essentials" which also brings easier development: instead of an heavy, complex, mixed application we have a main application with the fundamental parts and a variety of plugins which extend the functionality.

Plugins are usually implemented as demand-loaded shared libraries since these libraries can be loaded on runtime (as opposed to static) which is necessary for the "plug" concept characteristic of plugins. Thus, they inherit their characteristics.

Note that plugins are not shared libraries. A plugin is a paradigm based on the idea of "plugging" functionality into an application, demand-loaded shared library is a way to implement this paradigm.

But the demand loading raises some questions:

• Should the application know beforehand the plugin? "Knowing the plugin" translates to knowing the name of the library as well as it's interface and what it does. If it does, then what to do if

  • there is a change in the name or version of the plugin?

  • there is a change in the functionality (extension or alteration of the interface)?

  • there is a wish to add, remove or substitute plugins?

  On the other hand, if the application doesn't have this information

  • how should it acquire this information?

  • the knowledge on what to expect from the plugins is practically none

  • and so it is dangerous for the application to depend on a plugin

• Nevertheless, it has to be defined an interface that a plugin must implement and must be common to all ( in order to use the plugin, the application has to call, at least, one function; the reason why this should be common to all is related with the last topic and, in particular, comes clear if we consider we want to be able to add new plugins without modifying the application). But the question now is: how to define an interface that is not to restrict nor to loose?

• How much of the application could the plugin see and vice-versa? The view of the plugin by the application falls, in part, on the item above; in particular, if the application has no prior knowledge of the plugin, since some information must be given, using the plugin will be more difficult if this information is complex and big.

  On the other hand, the plugin has also to have some knowledge on the application. Since the plugin is developed with a particular application in mind the only problem that arises in carrying this knowledge is if there are alterations on the application.

In What to choose further discussion is done mainly comparing the different strategies presented here. In the next section, baring these problems in mind, it's presented a list of requirements that are believed to be essential in a plugin-based technology.

On developing plugins

We can specify the following list of requirements as a list that should be met in order to have a "true" plugin system. This list was determined essentially from the various plugin-based technologies and personal experience and, of course, some may not agree with it.

1. The application must be independent from the plugins.

2. There should be a "black-box" view in both directions.

3. The application must not "know" the plugin beforehand; the plugin must introduce itself to the application.

4. The application must be loosely coupled with the plugins; a common interface to be provided by the plugins (whether total or partial) must be defined precisely.

5. The application should define what services are provided to the plugin via interface definitions.

6. A plugin must be self-contained.

7. A plugin should create a plugin instance to handle each task.

The first point has to do with the fact that the application must run by itself, in other words, it must couple with not having a plugin adequate for a particular task ( even if it means, for example, discarding the task).

Most of the points before have to do with what was discussed before. The "black-box" view and the well defined interfaces in both direction enable both parts to be implementation independent of the other. So, if there is a change in the version of one of the parts, the risk of affecting the other part is lesser (less need, if not none at all, to recompile the other part). Also this accounts for a clear and standard communication between parts.

As was noted, so that there is more flexibility in "plugging" the application shouldn't known the plugins in advance. This also accounts for the independence between both parts (as an example, imagine adding a new plugin or changing by another functionally similar).

But then how does the application acquire the needed information? First, by expecting that the plugin obeys a certain defined interface common to all the plugins of that application. This interface must provide a function that returns the description of the plugin, in particular, what type of task it handles. The other important aspect that the application needs to know is what function to call so that this particular task is done: this can be achieved by either passing this information at the same time that the information above is passed, or, by requiring this function on the interface common to all (something of the type do_work()).

So, although this interface varies between technologies (as can be seen in the examples below), the plugin should provide:

• information on the type of task it can handle

• a function to be called to handle that type of task (do_work())

• an instance-type strategy including a structure to store all the data of a particular task and a way to know if there aren't any tasks (instances) still working so that the plugin can be unloaded

• a function to initialize the plugin

Whether the application only accepts the common interface or whether the plugin can provide an extension to this interface (a total interface opposed to a partial one) it's the developers decision but must also be defined beforehand so that the application can couple with that. If it's a partial one, the extra should also be declared when the plugin presents itself.

The last point comes mainly from the fact that a plugin is loaded at runtime and the separation between it and the application; if the plugin has dependencies, it can be very difficult to check these dependencies in runtime. This point is desirable but not always achievable.

Not quite a requirement but more of a principle: a plugin is application-dependent in the sense that it was developed with an application in mind. Even if it can be used outside it still has to obey an interface imposed by the application.

Looking at some plugin-based technologies

We will know look at some of the examples referred at the beginning of the section and see how the plugin paradigm is implemented in them: Netscape, Gnumeric and GIMP.

Example: a Gnumeric plugin

Although Gnumeric has some flaws it provides most of the core ideas of this paradigm.

The following plugin adds the famous factorial function (MyFactorial()) to Gnumeric. Here is the source code (plugin-myfact.c ).

    #include <gnome.h>
    #include <glib.h>
    
    #include "gnumeric.h"
  5 #include "func.h"
    #include "plugin.h"
    
    static Value *
    func_fact (FunctionEvalInfo *ei, Value *argv [])
 10 {
      int num = value_get_as_int (argv[0]);
      int res = 1;
    
      while(num > 1) {
 15     res *= num;
        num--;
      }
      return value_new_int (res);
    }
 20 
    static int
    can_unload (PluginData *pd)
    {
    	Symbol *sym;
 25 
    	sym = symbol_lookup (global_symbol_table, "MyFactorial");
    	return sym->ref_count <= 1;
    }
    
 30 static void
    cleanup_plugin (PluginData *pd)
    {
    	Symbol *sym;
    
 35 	g_free (pd->title);
    	sym = symbol_lookup (global_symbol_table, "MyFactorial");
    	if (sym)
    		symbol_unref (sym);
    }
 40 
    PluginInitResult
    init_plugin (CommandContext *context, PluginData *pd)
    {
    	FunctionCategory *cat;
 45 
    /* The next line checks if the Gnumeric version being used corresponds
       to the required by the plugin */
    	if (plugin_version_mismatch  (context, pd, GNUMERIC_VERSION))
    		return PLUGIN_QUIET_ERROR;
 50 
    	cat = function_get_category (_("Fact PlugIn"));
    	function_add_args (cat, "MyFactorial", "f", "number", NULL, func_fact);
    
    	pd->can_unload     = can_unload;
 55 	pd->cleanup_plugin = cleanup_plugin;
    	pd->title = g_strdup ("My PlugIn Plugin");
    
    	return PLUGIN_OK;
    }

The main function here is the init_plugin() function. Line 51 establishes the category of the plugin; line 52 adds the function func_fact() to Gnumeric with the name MyFactorial and requiring one argument. Next lines fill the remain positions of the PluginData variable.

Links and further information

Netscape has a web page on plugins: http://home.netscape.com/plugins/

Gnumeric can be found at http://www.gnome.org/projects/gnumeric/.

Gimp can be found at http://www.gimp.org.

No clear documentation of plugins was found. However Mozilla, Gnumeric and Gimp provide the source code so I recomend looking at the files where the loading and management of plugins are handled. Many of these applications provide sample/test plugins with the source code. It is also interesting to look at these samples for their structure[2].

Concept and examples

There isn't much consensus in what constitutes a component. A good yet still incomplete definition is Clemens Szyperski's list of characteristic properties of a component:

This definition may seem too general but basically constitutes the core on what consists a component and has several implications. This list will be complemented in the next section.

Discussing and complementing the definition

Middleware and CORBA

An example: BONOBO

"BONOBO is the GNOME architecture for creating reusable software components and compound documents."

GNOME implemented BONOBO in order to comply with interoperability between different applications in the desktop (e.g., embed a graphics object in a word processor document).

BONOBO is basically a set of CORBA interfaces and an implementation in GNOME/GTK+ of these interfaces that are designed to be simple to use and hide CORBA and between-components protocol used. An application uses these interfaces to export its functionality or integrate other components functionality making it possible to edit a graphic or even play GnoMines in a spreadsheet.

BONOBO is targeted both at the developer and the user.

Links and further information

Cetus links is a site devoted to providing links and information on Objects and Components. At the time of this writing, they stated having 18873 links related with these subjects. Among this were the following:

• Components and Objects Together by Clemens Szperski who also wrote a popular book on components.

• Components: Where Are They? by Dion Hinchcliffe and Michael J. Gaffney that presents a very concrete study on components and from where the requirements and characteristics presented in this document where most based.

• CDBi FORUM stands for Component-Based Development and Integration Forum and provides several interesting and useful informations about components.

Lesson 134: Middleware by Jonathan Angel provides a good overview of middleware. It focuses on the evolution of this software starting from the appearance of distributed computing to distributed objects with CORBA, JavaBeans and DCOM. It also makes a short reference to XML.

As for CORBA, there is a good number of documentation on CORBA. A good place to start is the OMG homepage.

The aim of these paradigms

There are always problems that are better solved with one strategy than another one. A script language is not very good to write Data Mining applications or complex mathematical algorithms or an application like a spreadsheet program. It is good for system management, automatic jobs and setting configurations.

Plugins are good for applications that have to do tasks from a big and heterogeneous set. It is even more powerful because it assumes that this set is not known before hand. This enables further tasks to be assigned to the application as long as the functionality to do that task is provided. This brings several advantages:

• a lighter application since all the complexity of the tasks is divided through the plugins

• the code to handle a task is only loaded when needed. If the set is big and not all the tasks are being done this means less space used in memory.

• flexibility in the functionality added to the application by plugins

At CERN this type of applications are found very frequently. The data generated is heterogeneous and is common to find: "if this is a particle a do A, a particle b do B, ..., a particle z do Z". Having enormous case columns is space consuming, complicated and provides few flexibility. Plugins may be a good way to handle this. There are also other situations similar where plugins can be a good solution.

Components are better aimed at complex, distributed projects but can also apply to projects that usually sit in only one computer but whose architecture OS clearly divided in several separate parts that interact with each other. It provides, among other things (see Key aspects and raised questions in Components):

• standardized parts and services

• few need of contact between teams designing components

• independence of implementation and location

• a good way to design the architecture and abstract from the implementations details

At CERN it is common to find big and complex distributed projects. Some of these projects even have teams in other places in the world developing parts of the project. With the importance of these projects and the volume of actions that the final product will have to perform, a good design and separation between parts is probably the most important aspect of the project. If this is well done, changes can even be made to one part without affecting the other parts.

But

Care must be taken before choosing to use them. Each has its advantages and disadvantages. If it is the right choice to use one of them or which one to use depends on the project.

Why comply with the sets of rules proposed?

When developing either plugins or components, there is a good list of requirements that must be met and that were enumerated in the previous sections.

It may seem, however, that some of this requirements can be dispensed in some occasions. For example, why use strategies like CORBA when it doesn't seem to be necessary? For the same reasons used at the beginning of this section to justify the need to establish them.

If we want to make use of the benefits that this technology enables, not complying with just one of the requirements will make the plugin/component-based strategy less robust and flexible. Regarding the example of the use of CORBA-like strategies, if later on there would be a need to have different platforms, for example, wrappers would have to be written to do the work this technology is responsible for.

Furthermore, Informatics Technology has evolved so much that the enumerated requirements can be handled with much less effort for the developer. The power that CORBA gives to application overcomes, in a good amount, the effort to use it.

Plugins and components: two popular and recent technologies

Plugins and components, although they are very recent, are now very popular. A search on the web will present thousands of references to it.

Most of the concepts used are not new but only recently technology provided the means for them to evolve. Plugins and components would not be what they are without Middleware and shared libraries and many other facilities available today.

But although they are so recent, products like Netscape, AutoCAD, the Windows and Linux desktops and many others, are using them as an important part of the product.

Although they are combined in this document, plugins and components are very different and each has its particular advantages and disadvantages as also having different scopes. Nevertheless there are come common keywords that pop out in both.

Finally

It was the purpose of this document to be a tutorial to this interesting and popular technology. There was an effort that this tutorial would have both a good and complete theoretic part with precise and correct definitions and a complete characterization of the paradigms, and a useful practical part by focusing on important implementation aspects and providing simple examples and references to useful tools and products. There was also an effort in presenting the information in a clear way that would be both complete and motivating.

Although it was the purpose of this tutorial to present a complete theory, this document is still an introduction. Please refer to the links and references provided throughout the document for a deeper understanding.