An update server and package distribution system made (easy) with jBackend

Since version 1.6 Joomla has an Update System that allows you to update installed extensions automatically. And since January 10th 2017 supporting this system is almost mandatory for each extension.

So we (finally) decided to add it to our extensions (named jBackend, ReDJ and Tag Meta). On the "extension" side this is really simple. You just need to add an update server to the manifest, something like this (let's use jBackend as practical example):

<updateservers>
    <server type="extension" priority="1" name="jBackend Updates">http://www.selfget.com/updates/jbackend.xml</server>
</updateservers>

Things on the "server" side are quite simple too. The jbackend.xml file should countain a release notice like the following:

<updates>
    <update>
        <name>jBackend 3.7.0</name>
        <description>To update the package you must have a valid subscription</description>
        <element>pkg_jbackend</element>
        <type>package</type>
        <version>3.7.0</version>
        <client>0</client>
        <folder></folder>
        <infourl title="selfget.com website">https://www.selfget.com</infourl>
        <downloads>
            <downloadurl type="full" format="zip">https://www.selfget.com/packages/get/release/download/jback_370</downloadurl>
        </downloads>
        <tags>
            <tag>stable</tag>
        </tags>
        <maintainer>selfget.com</maintainer>
        <maintainerurl>https://www.selfget.com</maintainerurl>
        <targetplatform name="joomla" version="3.[012345678]" />
    </update>
</updates>

Where the downloadurl tag could just be the direct link to the download package. That's all!

Simple, right? Well, not exactly in our case...

The challenge

We immediately faced with two main problems:

  • For free extensions we DON'T have a direct link to download. Each download requires the user to accept the GPL license to proceed, so direct package download is disabled.
  • For paid extensions we need to check if the user is authorized (i.e. has a valid subscription) to download the package. Subscriptions are managed (and this is very common) through assignment of specific groups to the user (e.g. if the user belongs to the group "jBackend" it is allowed to download it, once the group is removed from the user, downloads are automatically forbidden).

What kind of support Joomla provides for paid extensions? Well, there is a field named extra_query in the table update_sites that can be used to store some query params (e.g. download_id=123456). Those params are automatically appended to the download url wheneven Joomla makes a request to download a package (i.e. it would be https://www.selfget.com/packages/get/release/download/jback_370?download_id=123456).

The extension itself must update the field extra_query with the enabling information (e.g. a download token), and there are some examples about this here and here.

Now all we need is a service to generate and distribute the download keys to the users, as example, sending the key by email once the user has provided a granted email (i.e. with a valid subscription for the required package).

Our solution

So we could create some services to deal with license keys generation and package downloads, and we started seriously to think about it and... hey, we already have a simple way to build and expose services, thanks jBackend.

How about to create a jBackend plugin that implements a full Update Server for Joomla? Two days and less than 700 lines of code after, jBackend Release System was ready to use.

A product description with a list of features is available here, while the API documentation is here.

A real use case

What we want to describe here is how it basically works, showing step-by-step how to setup an Update Server for Joomla. In our example we can refer to ReDJ, which is available in two distributions, ReDJ Community (free) and ReDJ Enteprise (paid).

So there are two zip files in two different folders:

(free) => JPATH_ROOT/files/public/joomla/redj/ReDJ1.9.0Community_J3.zip
(paid) => JPATH_ROOT/files/private/joomla/redj/ReDJ1.9.0Enterprise_J3.zip

First we need to install and enable the Release System plugin, and create an endpoint (is a Joomla menu of type jBackend => Request) to publish this module only (we want an endpoint dedicated only to package distribution). The base path for this endpoint is https://www.selfget.com/packages.

jBackend Release System endpoint

Setup jBackend Release System plugin

Next step is to configure the jBackend Release System plugin settings. The field Folder must contain the base folder that contains the packages, related to JPATH_ROOT. In our case we have a files folder that is common for both free (folder public) and paid (folder private) extensions, so we can put this in the Folder option.

For free packages (i.e. ReDJ Community) it is enough to put this in the Packages field:

redj_com_190;/public/joomla/redj/ReDJ1.9.0Community_J3.zip
package id;package path[;package group]

package id is an unique identifier for the package release (i.e. redj_com_190)
package path is the physical path of the package file, relative to the packages distribution folder (i.e. /public/joomla/redj/ReDJ1.9.0Community_J3.zip)
package group is the group identifier and is related to the Package groups option. Is optional and not required for free downloads (i.e. we left it blank for ReDJ Community).

At this point we can already download this free package calling the URL /get/release/download/. In our example:

https://www.selfget.com/packages/get/release/download/redj_com_190

Support for paid extensions

For paid extension we need first to define a package group entity:

redj_enterprise;9,12;HASH123KEY456;ReDJ Enterprise
package group;groups[;hash;package name]

package group is an unique identifier (i.e. redj_enterprise)
groups is a list separated by commas of user groups IDs allowed to download the packages in this group (i.e. 9,12)
hash is an optional random string used as seed for the generation of license keys related to this package (i.e. HASH123KEY456, ...of course this is not the real one)
package name is the human readable name of the package used in the license mails (i.e. ReDJ Enterprise)

Once the package group is defined, we can add the package release to the list of packages:

redj_ent_190;/private/joomla/redj/ReDJ1.9.0Enterprise_J3.zip;redj_enterprise

jBackend Release System settings

In this case the package group is specified so the free download is not possible. If we just call the download URL:

https://www.selfget.com/packages/get/release/download/redj_ent_190

The service response in this case is not the binary package, but a JSON:

{
    "status": "ko",
    "error_code": "REL_ENS",
    "error_description": "Email not specified"
}

The reason is that for non-free packages the request MUST CONTAIN two mandatory params, email and key (e.g. https://www.selfget.com/packages/get/release/download/redj_ent_190?email=This email address is being protected from spambots. You need JavaScript enabled to view it.&key=123456). This means that only authorized users can download the package, and they must provide their email (the email used to register on joomla), and the license key. The only missing step is "how the user/extension can get the license key for the package?". There's a service for this...

The license keys distribution

The great news is that generation and distribution of those license keys is completely AUTOMATIC, zero maintenance for the package owner. If the user (identified by his email address on joomla) belongs to an authorized user group, he can get a valid license key to download the package, using one of the available services. Simple and clever... great, no?

There are two services for getting license keys. This first one requires just the user email as input parameter:

{{basepath}}/post/release/keybymail/redj_enterprise?email=This email address is being protected from spambots. You need JavaScript enabled to view it.

The response is a simple:

{
    "status": "ok"
}

And the license key is sent by email to the email address itself, after all the required verifications (e.g. the user is valid and belongs to an authorized group). This process requires that the user must enter manually the key in the extension license settings. There is a second service available to get the license key in the service response, but this of course requires the user authentication:

{{basepath}}/post/release/keybyauth/redj_enterprise?username=user123&password=pwd123

In this case the response is:

{
    "status": "ok",
    "email": "This email address is being protected from spambots. You need JavaScript enabled to view it.",
    "key": "123456"
}

It contains all information required to make a download request (email and key params). Of course for security reasons it is better to pass credentials with a POST payload instead of pass them as GET params (as in our example).

Conclusions

We described how we implemented our real extension update system for Joomla. Of course there is still the client side piece to implement, but this part is really easy (i.e. get the license information and save in the extra_query field) and there are a lot of examples out there.

With this service based approach we tried to simplify the server side duties for package distribution. The Release System has been created to work (easily) with the Joomla Update System, but we quickly realized that it is general enough to be used in any package distribution scenario. And all this is provided by a (very) small plugin for jBackend.