Saturday 2 November 2013

Deploying Additional Packages to the Sailfish Emulator and SDK

Latest update is 24 Dec 2013 11:40 UK time.

One of the areas that raises the most questions on the Sailfish Developers' mailing list is the sticky topic of Installing additional packages to the Sailfish Emulator and SDK. Many developers, including myself have found installation very confusing.

The reasons for the confusion are clear: there is no official documentation on this topic yet, thought it is promised; there are a number of different ways that a package can be installed; both the Emulator and SDK may required installation; each package is available in a number of flavours; and then there is the challenge of identifying the correct package in the first page, and determining the correct name of that package. And if that was not enough, due to the refactoring of Qt Mobility from Qt 4 to Qt 5, some of our favourite components are now offered by quite different packages.

This post should give you a broad understanding of the different installation techniques. To those who live and breath Linux, who first compiled the kernel before they went to school, much of this may be obvious. To the rest of us from different development backgrounds it is less so, hence this post

We will look at zypper, PkCon, ScratchBox2, yaml and spectacle. We will find out how to install "by hand*, and automatically on building and running.



If your app does anything interesting at all, then fairly soon you will want to use functionality not included in the packages pre-installed on the Sailfish Alpha Emulator / SDK. I am sure that the number of default packages will increase as Sailfish moves on to Beta and to real physical devices, but even then you app may need "something extra"; and one of the great things about the Sailfish world is that being based on Qt, Nemo and Meego there is a lot of stuff out there. Unfortunately some of these super packages will make you application ineligible for Jolla Harbour inclusion, but that is a problem for another day. That notwithstanding you can "easily" install these packages to your device.

Except that the "easily" turns out not to be quite that "easy" until you have spent some time understanding how installation works.

Installation Targets

The first stumbling block is that there are two targets you may potentially need to install stuff to. For want of a better word I will use the term "Installation Target". In the first evolutions of the Sailfish Alpha this was more obvious then it is now, as there were two visible virtual machines running, the Emulator, and the SDK build machine. As of the end of October 2013 the second of these is still very much there, and just as important as ever, but it now runs in stealth "headless" mode. 

The Emulator is the bit that looks like a Sailfish phone, and is the thing that (for the time-being) our apps will run on. Later when we get real Sailfish hardware in our grubby paws, then we will be able to substitute "Sailfish phone" for Emulator in this post. Packages that you install to the Emulator are packages that your application requires to run.

The second Installation Target is the SDK Build Engine Virtual Machine. This is a Nemo Linux installation in its own right. On this runs Scratchbox2 (SB2) with the ARM and i486 Targets to which we will need to install packages in order to build and run our code. Note that the SB2 Targets should not be confused with the underlying operating system / build engine: it is also possible to install stuff to the OS, but for development purposes we need to install to SB2 Target(s) chain, not the OS. More on this later. Packages that you install to SB2 Targets are packages that your application requires to build, and for QtCreator auto-code-completion to work properly.

Package Types

Somehow or other you have come across a package that you want to use. But as you look into the detail of the package, you find that it is available in different flavours. You will come across versions with devel in the name or with import-declarative, or even debug. All clearly related, but which do you need, and where?

Not all package developers follow the rules, but many do follow these rules of thumb.

The first thing to ascertain is the Qt version of the Package. As we are developing for Sailfish, we will want Qt5 packages, and normally the Qt5 version of a package will have Qt5 in the name, so plumb for that one first.

Packages with devel in the name are required for C++ development, and should be installed to the SKD Build Engine, but not to the Emulator / Real Device. Installing a devel package should normally automatically also install the main package of the same name.

The main package (i.e. the flavour without devel, debug, declarative etc.) in the name should be installed to both the SDK Build Engine and the Emulator / Real Device.

Packages with import-declarative in the name are required for QML development and should be installed to both the SDK Build Engine and the Emulator / Real Device. If they are not present on the Emulator, then if you app imports and uses QML components offered by the package, then your app will not run. It is possible that you may be able to build your app if the import-declarative package is not present on the Build Engine, but Qt Creator will not recognise the import statement or the imported QML component, and auto-code-completion won't hep you, so programming with the components won't be much fun.

Packages with debug in the name are used for debugging.

Installation Routes

Thee are 2 principal routes to installing additional packages: "By Hand", and "Automatically on Build via Spectacle". While the results are identical, we will see later that installing automatically on build is the preferred option, yet installing by hand is still useful, an important part of the learning curve, and some of the tools are even more useful for locating packages.

A Refresher: Connecting to the Emulator and SDK via SSH

In the next sections we will learn how to install by hand. To do so we will need to connect via SSH to the SDK and Emulator. I have posted articles in the past focussing on this, but her is a refresher of the commands used by the latest release of the SDK.

//Connect to SDK

ssh -p 2222 -i ~/SailfishOS/vmshare/ssh/private_keys/engine/mersdk mersdk@localhost

 

ssh -p 2222 -i ~/SailfishOS/vmshare/ssh/private_keys/engine/root root@localhost

 

//Connect to Emulator

ssh -p 2223 -i ~/SailfishOS/vmshare/ssh/private_keys/SailfishOS_Emulator/root root@localhost

 

ssh -p 2223 -i ~/SailfishOS/vmshare/ssh/private_keys/SailfishOS_Emulator/nemo nemo@localhost

Installing By Hand to the Emulator via PkCon and zypper

To install packages "by hand", we'll connect by SSH and use the RPM package management tools PkCon and zypper.

Why 2 tools?

Early version of the SDK used zypper as the default package manager on the Emulator, and it is still used on the SDK Build Engine / SB2. However with the current version of the SDK , the Emulator now uses PkCon.

Both tools are similar in scope, so which you use is a matter of personal taste and experience. I still prefer zypper, but am gradually getting the hang of PkCon.

You will find plenty of advice on the internet on zypper - look out for  "zypper cheat sheet", so I won't go into more detail here.

Update 20.12.2013: The latest SDK release no longer includes zypper by default on the Emulator. But you can easily install it by running: 

pkcon install zypper

Once zypper is installed the first useful call is:

zypper se

This searches the Emulator and repositories to which it is connected for all installed and available packages. Packages marked with "i" are installed, those without are not installed (yet). 

If you only have packages with "i", then you should  try "zypper refresh" to refresh the available packages.

"zypper se" should present you with a very long list of candidate packages, too long to be useful, so lets narrow down our search a bit. Let's say that we want GPS functionality, and we have heard that this is offered by QtLocation. In which case we could try this search:

zypper se location

This will present us all packages available with location in the name. On my Emulator this is:

S | Name                                        | Zusammenfassung                                                 | Typ
--+---------------------------------------------+-----------------------------------------------------------------+-----------
|   libdeclarative-location                     | Qt Mobility Location QML plugin                                 | Paket
|   libqtlocation1                              | Qt Mobility Location module                                     | Paket
|   qt5-qtdeclarative-import-location           | QtDeclarative location import                                   | Paket
|   qt5-qtdeclarative-import-location-debuginfo | Debug information for package qt5-qtdeclarative-import-location | Paket
|   qt5-qtlocation                              | The QtLocation library                                          | Paket
|   qt5-qtlocation-debuginfo                    | Debug information for package qt5-qtlocation                    | Paket
|   qt5-qtlocation-devel                        | Development files for QtLocation                                | Paket
|   qt5-qtlocation-source                       | Cross-platform application and UI framework                     | Quellpaket
|   qt5-qtlocation-source-debugsource           | Debug sources for package qt5-qtlocation-source                 | Paket
|   statefs-provider-inout-location             | Statefs inout provider: location information                    | Paket

So if we want to install qt5-qtlocation, we do the following:

zypper in qt5-qtlocation

Of course we can also remove packages with zypper rm, so real free to play around, and add and remove things.

The PkCon equivalent calls are:

pkcon search name location

 

pkcon install qt5-location

Installing By Hand to the SDK Build Engine via the Control Center

We could use zypper to install to SB2 Targets (and we look at how later), but it much easier to use the "SDK Control Center" to do so, as this is conveniently found in Qt Creator. Press the Sailfish Icon on the lefthand edge of QtCreator and chose the Targets tab.

We now have a choice of 2 targets. At a later date, when we finally get real hardware we will be able to use the SailfishOS-armv7hl target, but for the moment we will take the  we will chose SailfishOS-i486-x86 to test with the emulator. So click the "manage" button next to the SailfishOS-i486-x86 build target.

Just like zypper or PkCon, we can use the Control Center to search for packages using part of the name  The control center will start to suggest packages as soon as you start typing, so type slowly, and no enter is required!  Some packages will be listed twice, as source packages are also listed. Once you have found a candidate package, you can install or deinstall by the buttons next to the package name. So using our location example, here we might chose to install qt5-qtlocation-devel which should automatically also install qt5-qtlocation.

Having installed manually via zypper to the Emulator, and via the Control Center to the SDK Build Engine your app should now build and run - assuming of course that you have identified and installed the correct package "flavours" to the appropriate Installation Target.

Even though I now tend to install via the applications yaml file (more of which later), I find the control centre a good way of checking what is currently installed, and of searching for available packages. I note the name found in Control Center and add that to the yaml file.

Installing by Hand to the SDK Build Engine using zypper.

Connect to the SDK VM as user mersdk, and search for all Qt5 packages already installed with:

zypper se qt5 | grep "i |"

Then search for Qt5 packages in the SDK Control Center.

Notice anything?

The list of installed Qt5 packages found by zypper will be much shorter than that found by the Control Center.

So what's up?

To prove a point I have deliberately guided you to look in the wrong place! Remember I said that we should not confuse the SDK Operating System with the Build Engine?. Well using zypper directly tells us what is installed to the OS. It tells us noting about the packages available to the Build Engine. Installing a missing package here with zypper will have no effect on your build. Instead we need to enter the world of the build engine by typing:

sb2 -t SailfishOS-i486-x86

and then repeat zypper se qt5 | grep "i |"

This should give us a very different result to before; and even more crucially the same result as a search from the SDK Control Center.

Alternatively you can combine both commands in one:

sb2 -t SailfishOS-i486-x86 zypper se qt | grep "i |"

This calls the zypper search from within sb2, then exits.

If we want to install something to the build engine, then taking the example of qt5-qtlocation-devel, we would execute the following.

sb2 -t SailfishOS-i486-x86 -m sdk-install -R zypper in qt5-qtlocation-devel

While this technique does work, you still have to visit the SDK Control Center to press the "sync" button. This is to ensure that Qt Creator is fully synced with your newly installed packages.

What is ScratchBox2 (SB2)?

To build for the Emulator, and for a real device, the build engine needs to cross-compile: i.e. compile code for a different chip architecture. SB2 is the bit that achieves this magic, and allows us to compile for an ARM chip on a PC with a i486-x86 architecture. It also offers the advantage that we can use the power of the host CPU, even though it may be completely different from that of the target.

 

In the next section we will look at installing automatically as part of the build / run process. While I think that is the preferred technique, I still like to have SSH sessions open against the Emualtor and SDK, and use zypper to search for packages, and to identify the full name.

Installing Automatically as Part of a Build using Spectacle.

While installing packages manually can be fun in a nerdy sort of way, it does have one strong downside. Namely anyone else who tries to build and run you app will have to work out what are the required modules, and install those too. Yes you could add a list of required packages in your README file, but such documentation has a habit of going out-fo-date, So it would be great if there was a way of programatically declaring required packages, and have them installed automatically if not already present.

Such a system already exists. It's called spectacle, and is configured by the application's .yaml file, located in the RPM directory.

Yaml refers to format of the file, rather than to what it is does, but for want of anything better I will call it the .yaml file for the moment. In the same directory as the .yaml file is the spectacle file. You should not edit the spectacle file, as it is auto generated based on the config you perform in the .yaml file. Any changes you make to the .spec file will end up being over-written. The Spec file does have sections that are safe to tune. These are marked with >> and <<. If you change anything outside those areas and later modify the .yaml your changes in spec will be lost.

In the .yaml file are three config sections of interest to install packages: Requires:,  PkgBR: , and PkgConfigBR:.

Let's look at these in reverse order:

Requires: causes packages to be installed to the Emulator: i.e. packages required to run. The format of the package names it accepts is the same as we used with zypper, so using Requires: is functionally identical to installing manually with zypper, except that it happens automatically when you (or someone else) builds your application.

PkgBR: stands for Package Build Requires, and is used to configure packages that are required to build your app: i.e. any package configured here will automatically be installed to the SDK Build Engine, just as though we had installed it via the SDK Control Center, or via SB2 zypper. The format of the package names is the same as those used by the control centre.

PkgConfigBR: is in someways a functional duplication of PkgBR: in that it also allows the configuration of packages to be installed that the SDK requires to build your application. However it uses a different format of package name, namely those required by pkgconfig. These names are "published" in .pc files which are normally found within devel packages. You can list all such packages already installed with:

pkg-config --list-all

As PkgBR and PkgConfigBR do much the same thing, a particular package need only be specified in one of the two, and as far as I can assertion, it is mainly devel packages that have pkgconfig names.

Then when we build and run our application the specified packages will be installed to the SDK and Emulator if not already present.

Note 1:  This requires the application to be run with the option "Deploy as RPM Package". It will not work with the option "Deploy as By Copying Binaries.

Note 2:  You will require a network connection to access the repository hosting the missing package(s). If you have no network connection you will get an error like the one below:

 Retrieving: geoclue-0.12.99.2-1.3.1.i486.rpm [error]

 Download (curl) error for 'http://releases.sailfishos.org/sdk/latest/jolla/i486/mw/i486/geoclue-0.12.99.2-1.3.1.i486.rpm':

 Error code: Unrecognized error

 Error message: Empty reply from server

Note 3: After installing packages this way, you may need to visit the SDK Control Center and press the Sync button in order for Qt Creator to become aware of the changes. You may also need to select Tools /  QML/JS / Reset Code Model.

 

Examples to install Qt5Location and Qt5Positioning


Let us take a hypothetical app using GPS functionality as an example. In Qt5 GPS functionality is divided across 2 packages - Location and Positioning. Let us assume that we need both, and that we want to use both C++ and QML.


This means we need to end up with the following packages installed (if not already present):


On the SDK Build Engine (SB2)


 qt5-qtdeclarative-import-location  // for mL


 qt5-qtdeclarative-import-positioning // for qml


 qt5-qtlocation-devel // for C++


 qt5-qtpositioning-devel // for C++


 qt5-qtlocation // required by qt5-qtlocation-devel and qt5-qtdeclarative-import-location


 qt5-qtpositioning // required by qt5-qtpositioning-devel and qt5-qtdeclarative-import-positioning

 

On the Emulator / Real Device


 qt5-qtdeclarative-import-location // for qml


 qt5-qtdeclarative-import-positioning // for qml


 qt5-qtlocation // required by qt5-qtdeclarative-import-location


 qt5-qtpositioning // required by qt5-qtdeclarative-import-positioning


To achieve this our .yaml file could have the following entries:


 PkgConfigBR:

 Qt5Core

 Qt5Qml

 Qt5Quick

 sailfishapp

 Requires:

 sailfishsilica-qt5

 qt5-qtdeclarative-import-location

 qt5-qtdeclarative-import-positioning

 PkgBR:

 qt5-qtdeclarative-import-location

 qt5-qtdeclarative-import-positioning

 qt5-qtlocation-devel

 qt5-qtpositioning-devel

Note that for both the Emulator (Requires:) and the SDK Build Engine (PkgBR:) I have not explicitly added entries for qt5-qtlocation. This is because it will be installed by the dependant packages qt5-qtdeclarative-import-location and qt5-qtlocation-devel. The same applies to qt5-qtpositioning.

References


Packaging in general


https://sailfishos.org/develop-packaging-apps.html under Tips and Tricks


Scratchbox2:


http://www.scratchbox.org/documentation/user/scratchbox-1.0/html/tutorial.html


PkCon:

 

https://lists.sailfishos.org/pipermail/devel/2013-October/000845.html

http://www.mail-archive.com/devel@lists.sailfishos.org/msg00863.html

http://www.packagekit.org/pk-intro.html

http://www.packagekit.org/pk-using.html

 

Thanks

This post draws on a number of threads on the Sailfish Developers' mailing list. Thanks go to: Thomas Perl, Jarko Vihriala, Roberto Collstete, bolek, Dmitry, David Greaves and others who contributed to these threads, or have given feedback to earlier versions of this post.

 

 

1 comment:

  1. Thanks a lot for this post! This was exactly the information I was looking for :)

    ReplyDelete