Headless Eclipse RCP builds with Buckminster and Hudson

After visiting the “From source to automated builds with Buckminster and p2” presentation at Eclipse Summit Europe 2009 I had to give building RCP applications with Buckminster a try. And I wasn’t disappointed - despite from some minor file permission issues which took some time to debug, everything worked like a charm.

So I decided to extend my training materials for my Eclipse RCP training course (the next one is in March in Frankfurt/Main) to use Buckminster instead of PDE Build. I wrote the following tutorial as preparation for the course.

The tutorial will show you how to set up a headless build for the Mail sample application using Eclipse Buckminster and the Hudson build server. I assume you’re familiar with OSGi and Eclipse RCP development. You will learn:

• How to define and share a target platform
• How to export a p2 site for the application using Buckminster
• How to install the product from the p2 repository manually
• How to automate installing the product
• How to automate the product build using Hudson

Preparation

• Create a new RCP plug-in com.example.mail. Create a rich client application and choose RCP mail as template.

• Create a new feature com.example.mail.feature. Include the com.example.mail bundle in the feature. Add org.eclipse.rcp as Included Feature.

• Create a new product mail.product in the feature com.example.mail.feature. Configure the product: Specify a name for the product and change it to be based on features. Add the com.example.mail.feature (so the product contains the feature that hosts the product file). Configure a launcher name like mail under Launching.

  

• Run the product (just to be sure).

• Install Buckminster from the Eclipse release software site into your Eclipse IDE:

  

• Create a new feature com.example.mail.site. This feature will host all files for the Buckminster build.

Defining and sharing the target platform

Eclipse RCP applications are developed with a target platform containing all the platform features and bundles which are required for the application. This is configured in Window > Preferences > Plug-in Development > Target Platform. Create a new, empty target definition. Add Eclipse RCP SDK and Eclipse Platform Launchers from the Eclipse Software Site (you have to uncheck Group by category to see the feature). Uncheck Include Required Software so that you can check Include all environments (otherwise the target is only suitable for building on your own platform):

The target definition should look like this:

Click Move to move the target definition to a file rcp.target in the feature com.example.mail.site. This persists the target platform as file. That way, the target definition can be shared with other developers and can be used by the build.

Exporting a p2 site for the application using Buckminster

The first step is to use Buckminster from the IDE to build the contents of the feature com.example.mail.site and to create a software site for the results.

• Add the com.example.mail.feature as included feature to com.example.mail.site.

• You can export this feature as p2 site by invoking a Buckminster action. To do this, right-click the com.example.mail.site feature project and click Buckminster > Invoke action. You get a list of actions which can be invoked on the feature. Choose site.p2 to create a p2 site. You also need to specify some properties for the Buckminster build. So create a temporary buckminster.properties file somewhere containing:

  # Where all the output should go
  buckminster.output.root=${user.home}/tmp/mail
  # Where the temp files should go
  buckminster.temp.root=${user.home}/tmp/buildtmp
  # How .qualifier in versions should be replaced
  qualifier.replacement.*=generator:lastRevision
  
  target.os=*
  target.ws=*
  target.arch=*

  This tells Buckminster where to output the results and which platforms are to be included in the p2 site.

• Go to the exported site. Under com.example.mail.site_1.0.0-eclipse.feature/site.p2 you will find the exported p2 site containing your features and bundles for all platforms.

Installing the product from the p2 repository manually

The exported p2 site contains all the features and bundles of the product for all platforms. You could install the mail product from this software site now. The task of going to a p2 repository and installing an application locally is performed by the p2 director. If you have not used the director before, you should try to do this manually now to learn how it works. Otherwise you can skip to the next section.

• Download the standalone director from the Buckminster download page, look for director_latest.zip.

• Execute the director replacing the placeholders with the actual paths on your system:

  <path/to/director>/director
  	-consolelog
  	-r file://<path/to/p2/site>
  	-d <path/to/destination/folder>
  	-i com.example.mail.product

• This installs the given product (or the given installable unit, as it is called by p2) from the repository into the destination folder. You could specify a platform using the command line arguments -p2.os, -p2.ws, -p2.arch. If you don’t specify them, you install the product for the platform you’re running on. This should show you something like:

  Installing com.example.mail.product 1.0.0.qualifier.
  Operation completed in 7096 ms.

• Try to start the application from the destination folder.

How to automate installing the product

Usually users don’t want to install products by calling the director. Instead, a full installation is zipped together and distributed. You can automate this using Buckminster as well, but it is not supported out of the box. You need to add a custom buckminster task utilizing ant.

• Create a new folder build inside com.example.mail.site and copy the file product.ant in there. You can get this file from the Buckminster examples: product.ant.

• Use File > New > Other > Buckminster > Component Specification Extension File to create a new buckminster.cspex inside com.example.mail.site. This allows to define new Buckminster actions. The contents of this file should look like this:

  <?xml version="1.0" encoding="UTF-8"?>
  <cspecExtension xmlns:com="http://www.eclipse.org/buckminster/Common-1.0"
                      xmlns="http://www.eclipse.org/buckminster/CSpec-1.0">
      <actions>
          <public name="create.product" actor="ant">
              <actorProperties>
                  <property key="buildFile" value="build/product.ant" />
                  <property key="targets" value="create.product" />
              </actorProperties>
              <properties>
                  <property key="profile" value="MailProfile" />
                  <property key="iu" value="com.example.mail.product" />
              </properties>
              <prerequisites alias="repository">
                  <attribute name="site.p2" />
              </prerequisites>
              <products alias="destination" base="${buckminster.output}">
                  <path path="mail.${target.ws}.${target.os}.${target.arch}/" />
              </products>
          </public>
          <public name="create.product.zip" actor="ant">
              <actorProperties>
                  <property key="buildFileId" value="buckminster.pdetasks" />
                  <property key="targets" value="create.zip" />
              </actorProperties>
              <prerequisites alias="action.requirements">
                  <attribute name="create.product" />
              </prerequisites>
              <products alias="action.output" base="${buckminster.output}">
                  <path path="mail.${target.ws}.${target.os}.${target.arch}.zip" />
              </products>
          </public>
      </actions>
  </cspecExtension>
  

• This adds two new actions create.product and create.product.zip which basically call the p2 director using ant. Please note that the name of the destination folder/zip and the product id are specified in the cspex actions.

• Click Buckminster > Invoke action again and choose create.product. Please have a look in the buckminster.properties file you used to create the p2 site before. Creating the product would not work with * for os/ws/arch, because you can create the product only for a specific platform. So create a second file buckminster_product.properties to specify a concrete platform, f.e.:

  # Where all the output should go
  buckminster.output.root=${user.home}/tmp/mail
  # Where the temp files should go
  buckminster.temp.root=${user.home}/tmp/buildtmp
  # How .qualifier in versions should be replaced
  qualifier.replacement.*=generator:lastRevision
  
  target.os=win32
  target.ws=win32
  target.arch=x86

• Check that the product was correctly exported for the specified os/ws/arch combination by running it on that platform.

Automating the build using Hudson

One very helpful aspect about building with Buckminster is that you can automate builds using the Hudson build system. Before we can do that, we need to define how Buckminster should obtain our projects.

• To tell Buckminster to fetch our feature / bundle projects, we have to use a component query (.cquery). Create a new file site.cquery inside com.example.mail.site:

  <?xml version="1.0" encoding="UTF-8"?>
  <cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0"
                     resourceMap="site.rmap">
      <cq:rootRequest name="com.example.mail.site" componentType="eclipse.feature"/>
  </cq:componentQuery>
  

• As Hudson will check out the files and provide them inside a workspace folder for us, we need to tell Buckminster to get the features and bundles from this folder. For this, create a new resource map site.rmap inside com.example.mail.site:

  <?xml version="1.0" encoding="UTF-8"?>
  <rmap
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://www.eclipse.org/buckminster/RMap-1.0"
      xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
      xmlns:mp="http://www.eclipse.org/buckminster/MavenProvider-1.0"
      xmlns:pp="http://www.eclipse.org/buckminster/PDEMapProvider-1.0">
  
      <searchPath name="resources">
          <provider readerType="local" componentTypes="osgi.bundle,eclipse.feature"
                    mutable="true" source="true">
              <uri format="file:///{0}/{1}/">
                  <bc:propertyRef key="workspace.root" />
                  <bc:propertyRef key="buckminster.component" />
              </uri>
          </provider>
      </searchPath>
  
      <locator searchPathRef="resources"/>
  </rmap>
  

• Install the Hudson build server (Installing Hudson, Hudson on Tomcat, Latest Hudson WAR).

• Go to Hudson > Manage Plug-ins and install the Hudson Buckminster plug-in.

• So far, we have been running Buckminster from the Eclipse IDE. For running Buckminster from Hudson, we need to get a copy of ‘Buckminster Headless’. But you can’t download that. Do you remember how we installed the mail application from the p2 repository using the p2 director? That’s how you obtain Buckminster Headless:

  ./director
  	-r http://download.eclipse.org/tools/buckminster/headless-3.5/
  	-d /path/to/buckminster-headless/
  	-p Buckminster
  	-i org.eclipse.buckminster.cmdline.product

  It’s recommended to use an absolute path as destination path (I did not encounter this, but some people got error messages in the next step when they used a relative path as destination - thanks to Henno Vermeulen for reporting this).

• Once you have Buckminster headless, you can install additional features. As we are going to build products and features we need to install the Core and PDE features:

  cd /path/to/buckminster-headless/
  ./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.5/
  	org.eclipse.buckminster.core.headless.feature
  ./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.5/
  	org.eclipse.buckminster.pde.headless.feature

• Configure the path to your buckminster headless installation in Hudson > Manage Hudson > Configure System.

• Create a new free-style job in Hudson.

• Hudson expects to check out the files from some version control system like CVS, SVN or git. To continue, you need to have your files in some version control system. Configure the job to check out the projects from your source code repository.

• Create a new build step Run Buckminster that 1. imports the target definition and the projects, 2. builds the project and 3. creates product installations for all platforms:

  importtargetdefinition -A '${WORKSPACE}/com.example.mail.site/rcp.target'
  import '${WORKSPACE}/com.example.mail.site/site.cquery'
  build
  perform -D target.os=* -D target.ws=* -D target.arch=* com.example.mail.site#site.p2
  perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86 com.example.mail.site#create.product.zip
  perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86_64 com.example.mail.site#create.product.zip
  perform -D target.os=linux -D target.ws=gtk -D target.arch=x86 com.example.mail.site#create.product.zip
  perform -D target.os=linux -D target.ws=gtk -D target.arch=x86_64 com.example.mail.site#create.product.zip
  perform -D target.os=macosx -D target.ws=cocoa -D target.arch=x86 com.example.mail.site#create.product.zip
  perform -D target.os=macosx -D target.ws=cocoa -D target.arch=x86_64 com.example.mail.site#create.product.zip

• Check Archive the artifacts and specify these files (ignore the warning which might be shown):

  buckminster.output/com.example.mail.site_*-eclipse.feature/mail*.zip

• Run the build job:

  

Example code

I created a github project for the mail example: com.example.mail.buckminster

There is also example code in the Buckminster SVN (see org.eclipse.buckminster.tutorial.mailapp.releng).

Possible improvements

This tutorial leaves some room for improvement for the reader. I plan to extend this tutorial, but for the moment you’ll have to do some work on your own. If you find solutions for one of the following problems, please post a comment about it!

• Clear workspace before build: See How to delete Hudson workspace before build?, Add option to clean workspace before each build.
• Faster builds: The build executes the site.p2 action for every installation, taking 2-3 minutes for building the Mail app for 6 platforms. This is a bug fixed in Buckminster 1.2 / Eclipse 3.6, see here: Repeatedly executed action site.p2
• Manual target fetching: The target platform is fetched for every single job run. You can also fetch the target only once and store it to make the build faster. See here: Eclipse Community Forums - Mini-Tutorial: Fetching, archiving and re-using a target platform with Buckminster/Hudson
• Running unit tests and code coverage: See Execute JUnit tests on the bundles in the current workspace.
• Reports about compiler errors and warnings in Hudson.
• Creating native Windows installers, native OS X .apps and Debian packages.
• Uploading the build results to a web-site.
• Simplifying the whole process, see Bug 294142 - Convenient builds for Eclipse RCP applications (Vote/CC if you’re interested)
• Building WAR archives for Eclipse RAP/Equinox-Server based applications, see Eclipse Community Forums - Building a RAP web-app with buckminster
• Building and updating a software site that can be used to update the application using p2, not possible at the moment, see Using headless Buckminster to create p2 reporsitory with multiple versions

More information

• Buckminster Documentation: Eclipse Buckminster, The Definitive Guide by Henrik Lindberg
• Building an RCP application with hudson (Buckminster)
• Hudson Buckminster Plug-in

Thanks

Thanks to Henrik and Thomas for showing and explaining Buckminster at their ESE talk “From source to automated builds with Buckminster and p2”. Also big thanks to all the people providing all these pieces of useful information about Hudson and Buckminster in the Eclipse newsgroups and wiki which I took as starting point for this tutorial - especially to Johannes Utzig for the existing RCP+Buckminster tutorial.

Updates

• 2009-11-26: There was a problem related to the {workspace} variable when building on Windows. Make sure you use at least version 0.9.2 of the Hudson plug-in for Buckminster. Thanks to Tobias Wagner and Johannes Utzig for finding, reporting and fixing this issue! (See eclipse.tools.buckminster for discussion about it). Also, when calling the director, spaces in path parameters might cause problems, so better avoid such paths altogether.

• 2010-01-18: Updated the tutorial to use a .target definition file and to utilize a single free-style job for building multiple platforms.

Schulungen

• Eclipse RCP (20.09.2010 - 24.09.2010, Hamburg)
• git (07.10.2010 - 08.10.2010, Essen)

Eclipse RCP

• Do bug bounties foster open source collaboration?
• Building OSGi based applications
• Privately building SWTBot on Hudson for OS X, Windows and Linux
• Git In Action - Which features do you want to see in my screencast?
• The test-driven Eclipse RCP course
• Eclipse RCP examples
• Iterate over a JFace selection in a for each loop
• Headless Eclipse RCP builds with Buckminster and Hudson
• Developing Eclipse RCP applications following the MVC pattern
• Tutorial: p2 updates for Eclipse RCP applications
• Contributing commands to context menus
• Eclipse Riena Tutorial
• Building tables made easy: TableViewerBuilder
• Tutorial: Spring OSGi + Eclipse RCP
• Adding a command with a key binding for an editor
• Showing menu contributions depending on active perspective
• Creating an Eclipse RCP 3.5 target platform
• Toggle Handler for Eclipse Commands
• SWT/XML project started
• Eclipse RCP recipes: Automatically switch to a preferred perspective