Headless Eclipse RCP builds with Buckminster and Hudson

After visiting the “From source to automated builds with Buckminster and p2” presentation at Eclipse Summit Europe 2009 I had to give building RCP applications with Buckminster a try. And I wasn’t disappointed - despite from some minor file permission issues which took some time to debug, everything worked like a charm.

So I decided to extend my training materials for my Eclipse RCP training course (the next one is in March in Frankfurt/Main) to use Buckminster instead of PDE Build. I wrote the following tutorial as preparation for the course.

The tutorial will show you how to set up a headless build for the Mail sample application using Eclipse Buckminster and the Hudson build server. I assume you’re familiar with OSGi and Eclipse RCP development. You will learn:

How to define and share a target platform
How to export a p2 site for the application using Buckminster
How to install the product from the p2 repository manually
How to automate installing the product
How to automate the product build using Hudson

Preparation

Create a new RCP plug-in com.example.mail. Create a rich client application and choose RCP mail as template.
Create a new feature com.example.mail.feature. Include the com.example.mail bundle in the feature. Add org.eclipse.rcp as Included Feature.
Create a new product mail.product in the feature com.example.mail.feature. Configure the product: Specify a name for the product and change it to be based on features. Add the com.example.mail.feature (so the product contains the feature that hosts the product file). Configure a launcher name like mail under Launching.
Run the product (just to be sure).
Install Buckminster from the Eclipse release software site into your Eclipse IDE:
Create a new feature com.example.mail.site. This feature will host all files for the Buckminster build.

Defining and sharing the target platform

Eclipse RCP applications are developed with a target platform containing all the platform features and bundles which are required for the application. This is configured in Window > Preferences > Plug-in Development > Target Platform. Create a new, empty target definition. Add Eclipse RCP SDK and Eclipse Platform Launchers from the Eclipse Software Site (you have to uncheck Group by category to see the feature). Uncheck Include Required Software so that you can check Include all environments (otherwise the target is only suitable for building on your own platform):

Add Eclipse RCP feature

The target definition should look like this:

Eclipse RCP Target Definition

Click Move to move the target definition to a file rcp.target in the feature com.example.mail.site. This persists the target platform as file. That way, the target definition can be shared with other developers and can be used by the build.

Exporting a p2 site for the application using Buckminster

The first step is to use Buckminster from the IDE to build the contents of the feature com.example.mail.site and to create a software site for the results.

Add the com.example.mail.feature as included feature to com.example.mail.site.
You can export this feature as p2 site by invoking a Buckminster action. To do this, right-click the com.example.mail.site feature project and click Buckminster > Invoke action. You get a list of actions which can be invoked on the feature. Choose site.p2 to create a p2 site. You also need to specify some properties for the Buckminster build. So create a temporary buckminster.properties file somewhere containing:
```
# Where all the output should go
buckminster.output.root=${user.home}/tmp/mail
# Where the temp files should go
buckminster.temp.root=${user.home}/tmp/buildtmp
# How .qualifier in versions should be replaced
qualifier.replacement.*=generator:lastRevision

target.os=*
target.ws=*
target.arch=*
```
This tells Buckminster where to output the results and which platforms are to be included in the p2 site.
Go to the exported site. Under com.example.mail.site_1.0.0-eclipse.feature/site.p2 you will find the exported p2 site containing your features and bundles for all platforms.

Installing the product from the p2 repository manually

The exported p2 site contains all the features and bundles of the product for all platforms. You could install the mail product from this software site now. The task of going to a p2 repository and installing an application locally is performed by the p2 director. If you have not used the director before, you should try to do this manually now to learn how it works. Otherwise you can skip to the next section.

Download the standalone director from the Buckminster download page, look for director_latest.zip.

Execute the director replacing the placeholders with the actual paths on your system:

<path/to/director>/director
	-consolelog
	-r file://<path/to/p2/site>
	-d <path/to/destination/folder>
	-i com.example.mail.product

This installs the given product (or the given installable unit, as it is called by p2) from the repository into the destination folder. You could specify a platform using the command line arguments -p2.os, -p2.ws, -p2.arch. If you don’t specify them, you install the product for the platform you’re running on. This should show you something like:
```
Installing com.example.mail.product 1.0.0.qualifier.
Operation completed in 7096 ms.
```
Try to start the application from the destination folder.

How to automate installing the product

Usually users don’t want to install products by calling the director. Instead, a full installation is zipped together and distributed. You can automate this using Buckminster as well, but it is not supported out of the box. You need to add a custom buckminster task utilizing ant.

Create a new folder build inside com.example.mail.site and copy the file product.ant in there. You can get this file from the Buckminster examples: product.ant.

Use File > New > Other > Buckminster > Component Specification Extension File to create a new buckminster.cspex inside com.example.mail.site. This allows to define new Buckminster actions. The contents of this file should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<cspecExtension xmlns:com="http://www.eclipse.org/buckminster/Common-1.0"
                    xmlns="http://www.eclipse.org/buckminster/CSpec-1.0">
    <actions>
        <public name="create.product" actor="ant">
            <actorProperties>
                <property key="buildFile" value="build/product.ant" />
                <property key="targets" value="create.product" />
            </actorProperties>
            <properties>
                <property key="profile" value="MailProfile" />
                <property key="iu" value="com.example.mail.product" />
            </properties>
            <prerequisites alias="repository">
                <attribute name="site.p2" />
            </prerequisites>
            <products alias="destination" base="${buckminster.output}">
                <path path="mail.${target.ws}.${target.os}.${target.arch}/" />
            </products>
        </public>
        <public name="create.product.zip" actor="ant">
            <actorProperties>
                <property key="buildFileId" value="buckminster.pdetasks" />
                <property key="targets" value="create.zip" />
            </actorProperties>
            <prerequisites alias="action.requirements">
                <attribute name="create.product" />
            </prerequisites>
            <products alias="action.output" base="${buckminster.output}">
                <path path="mail.${target.ws}.${target.os}.${target.arch}.zip" />
            </products>
        </public>
    </actions>
</cspecExtension>

This adds two new actions create.product and create.product.zip which basically call the p2 director using ant. Please note that the name of the destination folder/zip and the product id are specified in the cspex actions.
Click Buckminster > Invoke action again and choose create.product. Please have a look in the buckminster.properties file you used to create the p2 site before. Creating the product would not work with * for os/ws/arch, because you can create the product only for a specific platform. So create a second file buckminster_product.properties to specify a concrete platform, f.e.:
```
# Where all the output should go
buckminster.output.root=${user.home}/tmp/mail
# Where the temp files should go
buckminster.temp.root=${user.home}/tmp/buildtmp
# How .qualifier in versions should be replaced
qualifier.replacement.*=generator:lastRevision

target.os=win32
target.ws=win32
target.arch=x86
```
Check that the product was correctly exported for the specified os/ws/arch combination by running it on that platform.

Automating the build using Hudson

One very helpful aspect about building with Buckminster is that you can automate builds using the Hudson build system. Before we can do that, we need to define how Buckminster should obtain our projects.

To tell Buckminster to fetch our feature / bundle projects, we have to use a component query (.cquery). Create a new file site.cquery inside com.example.mail.site:

<?xml version="1.0" encoding="UTF-8"?>
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0"
                   resourceMap="site.rmap">
    <cq:rootRequest name="com.example.mail.site" componentType="eclipse.feature"/>
</cq:componentQuery>

As Hudson will check out the files and provide them inside a workspace folder for us, we need to tell Buckminster to get the features and bundles from this folder. For this, create a new resource map site.rmap inside com.example.mail.site:

<?xml version="1.0" encoding="UTF-8"?>
<rmap
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="http://www.eclipse.org/buckminster/RMap-1.0"
    xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
    xmlns:mp="http://www.eclipse.org/buckminster/MavenProvider-1.0"
    xmlns:pp="http://www.eclipse.org/buckminster/PDEMapProvider-1.0">

    <searchPath name="resources">
        <provider readerType="local" componentTypes="osgi.bundle,eclipse.feature"
                  mutable="true" source="true">
            <uri format="file:///{0}/{1}/">
                <bc:propertyRef key="workspace.root" />
                <bc:propertyRef key="buckminster.component" />
            </uri>
        </provider>
    </searchPath>

    <locator searchPathRef="resources"/>
</rmap>

Install the Hudson build server (Installation guide for Hudson server, Hudson on Tomcat, Latest Hudson WAR).
Go to Hudson > Manage Plug-ins and install the Hudson Buckminster plug-in.
So far, we have been running Buckminster from the Eclipse IDE. For running Buckminster from Hudson, we need to get a copy of ‘Buckminster Headless’. But you can’t download that. Do you remember how we installed the mail application from the p2 repository using the p2 director? That’s how you obtain Buckminster Headless:
```
./director
	-r http://download.eclipse.org/tools/buckminster/headless-3.5/
	-d /path/to/buckminster-headless/
	-p Buckminster
	-i org.eclipse.buckminster.cmdline.product
```
It’s recommended to use an absolute path as destination path (I did not encounter this, but some people got error messages in the next step when they used a relative path as destination - thanks to Henno Vermeulen for reporting this).

Once you have Buckminster headless, you can install additional features. As we are going to build products and features we need to install the Core and PDE features:

cd /path/to/buckminster-headless/
./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.5/
	org.eclipse.buckminster.core.headless.feature
./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.5/
	org.eclipse.buckminster.pde.headless.feature

Configure the path to your buckminster headless installation in Hudson > Manage Hudson > Configure System.
Create a new free-style job in Hudson.
Hudson expects to check out the files from some version control system like CVS, SVN or git. To continue, you need to have your files in some version control system. Configure the job to check out the projects from your source code repository.

Create a new build step Run Buckminster that 1. imports the target definition and the projects, 2. builds the project and 3. creates product installations for all platforms:

importtargetdefinition -A '${WORKSPACE}/com.example.mail.site/rcp.target'
import '${WORKSPACE}/com.example.mail.site/site.cquery'
build
perform -D target.os=* -D target.ws=* -D target.arch=* com.example.mail.site#site.p2
perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86 com.example.mail.site#create.product.zip
perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86_64 com.example.mail.site#create.product.zip
perform -D target.os=linux -D target.ws=gtk -D target.arch=x86 com.example.mail.site#create.product.zip
perform -D target.os=linux -D target.ws=gtk -D target.arch=x86_64 com.example.mail.site#create.product.zip
perform -D target.os=macosx -D target.ws=cocoa -D target.arch=x86 com.example.mail.site#create.product.zip
perform -D target.os=macosx -D target.ws=cocoa -D target.arch=x86_64 com.example.mail.site#create.product.zip

Check Archive the artifacts and specify these files (ignore the warning which might be shown):
```
buckminster.output/com.example.mail.site_*-eclipse.feature/mail*.zip
```
Run the build job:

Example code

I created a github project for the mail example: com.example.mail.buckminster

There is also example code in the Buckminster SVN (see org.eclipse.buckminster.tutorial.mailapp.releng).

Possible improvements

This tutorial leaves some room for improvement for the reader. I plan to extend this tutorial, but for the moment you’ll have to do some work on your own. If you find solutions for one of the following problems, please post a comment about it!

Clear workspace before build: See How to delete Hudson workspace before build?, Add option to clean workspace before each build.
Faster builds: The build executes the site.p2 action for every installation, taking 2-3 minutes for building the Mail app for 6 platforms. This probably is a bug, see here: Repeatedly executed action site.p2
Manual target fetching: The target platform is fetched for every single job run. You can also fetch the target only once and store it to make the build faster. See here: Eclipse Community Forums - Archiving and publishing an imported target
Running unit tests and code coverage: See Execute JUnit tests on the bundles in the current workspace.
Reports about compiler errors and warnings in Hudson.
Creating native Windows installers, native OS X .apps and Debian packages.
Uploading the build results to a web-site.
Simplifying the whole process, see Bug 294142 - Convenient builds for Eclipse RCP applications (Vote/CC if you’re interested)

More information

Thanks

Thanks to Henrik and Thomas for showing and explaining Buckminster at their ESE talk “From source to automated builds with Buckminster and p2”. Also big thanks to all the people providing all these pieces of useful information about Hudson and Buckminster in the Eclipse newsgroups and wiki which I took as starting point for this tutorial - especially to Johannes Utzig for the existing RCP+Buckminster tutorial.

Updates

2009-11-26: There was a problem related to the {workspace} variable when building on Windows. Make sure you use at least version 0.9.2 of the Hudson plug-in for Buckminster. Thanks to Tobias Wagner and Johannes Utzig for finding, reporting and fixing this issue! (See eclipse.tools.buckminster for discussion about it). Also, when calling the director, spaces in path parameters might cause problems, so better avoid such paths altogether.
2010-01-18: Updated the tutorial to use a .target definition file and to utilize a single free-style job for building multiple platforms.

Martin Dilger, 05. November, 21:01 Uhr

hi once again,
really nice tutorial..
I read about Buckminster before, but I never tried it myself.

Thanks for this.

Karsten Thoms, 06. November, 07:06 Uhr

Hi Ralf!

Really good article! I was also in the ESE talk and decided to finally use Buckminster and started to work through the docs. Was not as easy as expected, but your article really helped to get things done!

~Karsten

Christian, 06. November, 13:42 Uhr

great tutorial, thank you very much!

Cheney, 12. November, 07:14 Uhr

Cool! Thanks!
But I have some problem that building the com.example.mail.buckminster project source with hudson.

First, in Automating the target platform build using Hudson step 8: #

Add a build step "Run Buckminster". Configure it to import the target.mspec using:

import ${WORKSPACE}/com.example.mail.target/target.mspec

the correct one should be: import ${WORKSPACE}/com.example.target/target.mspec, and then the rcp_target platform build successful.

Second, I get step by step from the Automating the product build using Hudson instruction, but it always builds failure, below is the console output from hudson:

[...]
/data/ssis/.hudson/jobs/MailAppExample/workspace/arch/x86_64/os/macosx/ws/cocoa/com.example.mail.feature/build/product.ant:22: Java returned: 13
[...]

Can you please give me some suggestion how to fix the build product problem?

Ralf Ebert, 17. November, 10:23 Uhr

@Cheney, thanks for the hint. Sorry, I can't see any cause of the problem you encountered from the stack trace. You might have more luck asking in the 'eclipse.tools.buckminster' newsgroup.

Tobias Wagner, 19. November, 15:18 Uhr

Hi,

nice tutorial but I´m having problems in step:
Exporting a p2 site for the application using Buckminster (2)

If I right-click of my com.eclipse.mail.site feature and say : Buckminster -> invoke actions an error message appears saying:

CSpec
org.eclipse.rcp:feature$3.5.1.R35x_v20090811-9SA0FxVFqE70OL1ARMrfcO6e7BA6 has no action, group, or local artifact named source.feature.jars

Does anyone has a solution for that problem?

TW

Tobias Wagner, 19. November, 15:49 Uhr

Gave it a second try with same error.
I´m quite sure that I followed your tutorial and have no idea what could be the problem

Ralf Ebert, 19. November, 18:11 Uhr

I assume you tried restarting Eclipse already? I remember that Buckminster wouldn't show actions directly after I created the project. Otherwise, you might have more luck asking in the 'eclipse.tools.buckminster' newsgroup.

Tobias Wagner, 19. November, 19:13 Uhr

Hi,
I´ve addet a cquery file to my feature and deleted it and now I´m able to add actions. Very strange think its a bug.

Tobias Wagner, 20. November, 06:49 Uhr

Hi, I´ve fixed the error.
I added a cquery file to my mail.site feature. After adding it i deleted it instantly. Since then I was able to add an action like you described in your tutorial. I don´t know if this has anything to do with my system configuration but it seems like a bug to me.

Tobias Wagner, 23. November, 14:20 Uhr

Hi,

I´m having other problems now. I´ve set up a hudson server with buckminster plugin. I downloaded your project to ensure that I have not made any errors during the eclipse part of your tutorial. I´ve downloaded a Buckminster-headless and extractet my target Platform. I´ve also entered all Paths in hudson. When I try to build my project I get the following message:

Gestartet durch Benutzer anonymous
Commandline:
java -Dbuckminster.output.root=/C:/hudson/jobs/rcp_target/workspace//buckminster.output -Dbuckminster.temp.root=/C:/hudson/jobs/rcp_target/workspace//buckminster.temp -DtargetPlatformPath=C:/buckminsterTargetPlatform -jar plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar -application org.eclipse.buckminster.cmdline.headless -data /C:/hudson/jobs/rcp_target/workspace/ --loglevel info -S C:\hudson\jobs\rcp_target\builds\2009-11-23_15-13-24/commands.txt INFO: import 'file:///D:/workspaces/buckminster/com.example.target/target.mspec'
ERROR [0001] : No suitable provider for component com.example.target:eclipse.feature was found in resourceMap file:/D:/workspaces/buckminster/com.example.target/target-resources.rmap
ERROR [0001] : No suitable provider for component com.example.target:eclipse.feature was found in searchPath default
ERROR [0001] : Resolution attempt ended with exception: Provider local(file:///C:/hudson/jobs/rcp_target/workspace/com.example.target/): Missing CSpec source required by component type eclipse.feature
ERROR Provider local(file:///C:/hudson/jobs/rcp_target/workspace/com.example.target/): Missing CSpec source required by component type eclipse.feature
INFO: TAG-ID 0001 = Query for com.example.target:eclipse.feature

Archiviere Artefakte
Finished: FAILURE

Maybe you know a solution, because I saw a post from you with a similar problem (http://www.eclipse.org/forums/index.php?S=f9510e808d1e85762010ee0629119c26&t=msg&th=156976)

Ralf Ebert, 24. November, 00:16 Uhr

Tobias, glad to hear it works in the IDE now. Finding the reason for these "Missing CSpec source required by component type eclipse.feature" messages is cumbersome indeed. For me the problem was that Buckminster couldn't write in its own folder, which meant that it couldn't store downloaded artifacts. Sorry, I can't see any clue about what's going wrong from the error message. Maybe ask in the 'eclipse.tools.buckminster' newsgroup about this for some more tips where to look for potential problems...

Tobias Wagner, 24. November, 10:03 Uhr

Hi Ralf,
I did already ask for help in the eclipse newsgroup and now it works.
I did not enter any SVN/CVN Connection and thought that hudson would use the files in my eclipse workspace. Now I know that this is wrong . As soon as I added a svn connection everything worked, because hudson downloaded the workspace to its workspace location and was able to build the project successfully.
So the whole point was a big misunderstanding ^^

Thanks or your support

Henno Vermeulen, 27. Januar, 13:04 Uhr

Thank you for the tutorial.

I followed the tutorial until before the Hudson part, but there I run into problems with the http://download.eclipse.org/tools/buckminster/headless-3.5/ site. When I try to download buckminster-headless (using the same command as on the official "Buckminster Downloads" page) I get

director -r http://download.eclipse.org/tools/buckminster/headless-3.5/ -d temp-buckminster-headless/ -p Buckminster -i
org.eclipse.buckminster.cmdline.product
Unable to load repositories.
Application failed, log file location: C:\Users\h.vermeulen\Oblomowdev\director\
configuration\1264596913052.log

So I downloaded the entire headless site using the link on the Buckminster Downloads page and running the command on the local site works perfectly.
However when I try to install the needed features I get this:

buckminster install http://download.eclipse.org/tools/buckminster/headless-3.5/ org.eclipse.buckminster.core.headless.feature
No suitable feature/version found that matches org.eclipse.buckminster.core.headless.feature.feature.group

Problems with the eclipse site? It's a real showstopper for us to set up our continuous integration...
Should I register to bugzilla and file a bug report? Is there a way to manually download and install the features? Or am I doing something wrong?

Henno Vermeulen, 27. Januar, 13:42 Uhr

Because this issue is not specific to your tutorial, I decided to post to the Buckminster forum, my post can be found here:

http://www.eclipse.org/forums/index.php?t=msg&goto=510414&#msg_510414

Henno Vermeulen, 28. Januar, 13:17 Uhr

Thank you for the tutorial I got it all working now! The 3.5 update site is online again by the way. I also had an issue on windows when installing the second feature into the headless Buckminster, it turned out that I needed to specify an absolute path when installing it with the Director, after that installing the features works flawlessly.

It is still a bit of black magic to me how everything works together and how I can customize it for us. But I suppose reading the Buckminster book might help a lot with that.

erdal, 03. Februar, 14:24 Uhr

create.product.zip creates a zip that contains an empty root dir.
The p2 site is generated, the product and the executable as well.
Any hints?

erdal, 03. Februar, 15:19 Uhr

OK, I forgot the trailing slash in:

<products alias="destination" base="${buckminster.output}">
<path path="mail.${target.ws}.${target.os}.${target.arch}/" />
</products>

ViK, 04. Februar, 14:53 Uhr

Hi Ralf,

I encountered myself with a problem related with platform specific fragments. Just wanted to add here my feedback in case someone faces the same problem. You can read the discussion at Buckminster's newsgroup:

http://www.eclipse.org/forums/index.php?t=msg&th=162028&start=0&

It seems buckminster, while importing the component query, was locating bundles from the target platform, but (silently) rejecting because it did not match the platform filters. That led to several errors, and unsuccessful builds. Thomas Hallgren already issued a bugzilla and immediately fixed it.

So, considering this is a tutorial for a multiple-platform build, I think it would make sense to add that the following values to the component query example:

<cq:property key="target.arch" value="*"/>
<cq:property key="target.os" value="*"/>
<cq:property key="target.ws" value="*"/>

Maybe I missed something in your tutorial. I know you suggested to specify those through a properties file when making a buckminster build within eclipse. Would it make sense to include those values in the cquery file of the tutorial?

Thanks for this great tutorial :D

ViK

Ralf Ebert, 04. Februar, 21:40 Uhr

@ViK, thanks for investigating this and posting a comment about it. I did the same thing in a different way, I configured the .target to include all environments. This is also mentioned in the tutorial: "Uncheck Include Required Software so that you can check Include all environments (otherwise the target is only suitable for building on your own platform)". Probably I should make this more eye-catching.

I'm looking forward to your comments: