TL;DR

This article explains Stream Events and Companion Screens based on a showcase for maxdome presented at the IFA. In order to develop an HbbTV application you need to setup a live service and take care of following things:

• Create a MPEG-TS Stream for testing with stitched events
• Upload and serve this file via Sofia Backstage
• Connect to your HbbTV application hosted on a web server
• Setup correct MIME types
• Initialise and create the Application Manager
• Connect your mobile phone with your SmartTV with the Companion Screen Manager
• Develop UI and bind to key events

Finally many use cases are possible with HbbTV but development is not always easy. Every documentation is helpful and that is the motivation for this blog article.

Definition

HbbTV (Hyper broadcast broadband) was created in 2009 by the merger of several partners and is an open and international standard that combines traditional television with the possibilities of non-linear content, i.e. the Internet. When switching to a television channel, in principle any hosted web application can be run in the background.

Application Areas

A typical HbbTV example is the display of an overlay on the lower right edge of the screen and the associated additional information that appears by pressing an arbitrary button on the remote control (RC). However, the so-called “Red Button” is recognised by the user as the preferred way of opening applications. This button has been memorised by users, but technically every button press trigger corresponding events and other views. For example, the ZDF media library showed users of HbbTV supporting devices a 360º Livestream of the Biathlon World Cup 2019 [ZDF].

Enhancing sports events like the upcoming Olympics in Tokyo 2020 with additional information like live statistics, highlights and parallel games might also be an interesting use case. This leads to an improved user experience for the otherwise static, linear television.

Enhancing the linear experience

An use case for HbbTV could be to link additional information about actors, the scene or the music etc. in an overlay with the currently displayed content during a film or series. Amazon’s well-known example is the so-called “X-Ray”, which represents exactly this information. In the world of linear signals, this can also be done by devices supporting HbbTV.

A well-known example is the Program Guide or EPG, i.e. the display of current and future programs of all existing channels. Here it is possible to switch between the channels within the EPG and to select a new program. The interfaces for controlling the video signal that HbbTV makes available make it an interesting standard.

Subtitles and context information

Many of today’s streaming providers, such as Netflix, offer subtitles in different languages. Of course, this is also possible with HbbTV. Subtitles for film or song texts for music can be displayed. This enables accessibility for many viewers regardless of a monthly subscription and video-on-demand (VOD) providers. Stream events can help to display at exact time.

Personalised advertisement

However, the most interesting argument seems to be personalised and automated advertising. HbbTV enables users and viewers to be addressed more individually. During an ad break the location data can be used to play different ads over the linear content. Past purchases from a registered user can be taken into the decision to play the suitable ad for him/her. It would also be conceivable to introduce a cost-per-click pricing model for ads that are viewed and found to be interesting. This opens possibilities to automate and control advertising on television.

With HbbTV, it is no longer needed to spread out advertisement but rather use them in a targeted manner. Viewers become buyers, directly in the cozy living room — an interesting use case for Amazon. The idea is to not only refer to products from e.g. the series Hunters on Amazon, but to encourage viewers to buy on linear TV. These and many other possibilities result from the connection of television and internet and have enormous potential.

With companion screens you can start the playback of a next episode on your mobile device from the SmartTV. For example, should your favorite series, for example jerks. come to an end, the pre-installed mobile application such as Joyn can be started by the remote control. A similar example was developed and presented as following showcase.

What is HbbTV and what are the differences to normal web technologies? This article shows typical areas of application and explains HbbTV features such as stream events and companion screens based on a showcase. We refer to the standard that was used at the time of implementation: HbbTV 2.0.1 Specification (2016–07–04) [HBBTV-2.0.1]. Since then the latest HbbTV Standard 2.0.2 (2018–09) [HBBTV-2.0.2] was published in September 2018 and has new features in the area of HDR / HFR video and audio. Besides that, a variety of scenarios can be represented by HbbTV.

Showcase

Our HbbTV Showcase was presented at the IFA in Berlin and at the dmexco in Cologne in 2016. The goal was to develop a prototype for ProSiebenSat.1 Media in cooperation with the IRT and to present it on the booth of “Deutsche TV-Plattform”. The prototype uses stream events and shows a spot overlay at regular intervals, which makes it possible to open a detail page from the SmartTV to a mobile phone or tablet.

Starting an HbbTV application

SmartTV and Internet are the best prerequisites for developing an HbbTV application. But how far does this differ from classic web technologies? The biggest difference is that HbbTV is only intended for broadcasting, i.e. linear services. How does this affect the viewer? Compared to the normal browser, the SmartTV loads the application when changing to a channel or service. The illustration shows the interaction between the hybrid terminal (TV), application server (Internet) and broadcast signal (DVB-S). In addition, the use of a second screen device is shown. When starting a service, the device loads all necessary files such as scripts (ES), stylesheets (CSS) and markup files (HTML).

Among other things the specification defines the start of an HbbTV application by calling createApplication(), which is possible by the Application Manager (see Section 6.2.2 Starting and Stopping Applications [HBBTV-2.0.1]):

• Directly by the end-user (e.g. by using dedicated buttons on the remote control or an equivalent menu provided by the terminal).
• In response to signaling in a broadcast service (e.g. automatically starting a broadcast-related autostart application).
• By an already running application (via the JavaScript method createApplication()).
• By a Companion Screen as described in clause 14.6.

Switching to a channel on which an HbbTV application is registered, it is loaded and executed by the server. The path to the server is in the AIT (Application Information Table) and thus part of the service or channel as an embedded signal. In addition to the URL, this table also contains further information.

Setup with Sofia Backstage

After a chain of logical steps, the application is loaded and executed based on the AIT information. In this example we use a simple HTTP server with the control code AUTOSTART, which executes the application immediately after starting a channel. Theoretically, several applications can run within one service and — depending on the size of the data — can also be transmitted within a broadcast signal (DSM-CC) instead of a server.

We use the Mini Playout Service Sofia® Backstage to transmit the signal and embed the AIT. This is a small box on which an operating system like Open Suse and an Apache server run. Connected to a multiplexer, this embeds our digital data into a DVB-T signal and sends it to the television using an antenna or satellite cable. As a video we use a 3-minute sequence from the series “jerks.”. MPEG-TS (Transport Stream) is for the transmission of video, audio and data and in our case contains the stitched events to which we will react later in the application. In the Playout settings, you can upload this transport stream as a file and create a new application out of it.

To create a new channel, we create a new service with an automatically created Logical Channel Number (LCN) and connect it to the previously uploaded Transport Stream (TS) and the application.

After a restart of the DVB-T signal and an automatic channel search, the 3-minute scene from jerks now runs on the created channel. The playout service is now ready for use and my SmartTV shows my favorite scene continuously in a loop.

Differences to web applications

In contrast to classic web applications, HbbTV comes with its peculiarities. For example, it uses a special Doctype Declaration and XML Namespace. It is therefore important to correctly configure the MIME Type on the server when delivering the HTML. Since the client asks for an application/vnd.hbbtv.xhtml+xml, it has to be set accordingly in the/etc/mime.types file. The normal browser will not be able to display but only to download them.

The application must have the corresponding Content Type

Different from the usual HTML

On the server side, MIME types can be set here

The HbbTV specification basically describes the general structure of the XHTML markup.

Making the application work

Our application is an Express (https://expressjs.com/) server that generates certain views with the help of Handlebars (http://handlebarsjs.com/) and delivers them to the client (this is an architectural decision and therefore not mandatory). Since there is a live stream running in the background, we don’t want to do much in our application except to only show a spot overlay in the lower right-hand margin, which always appears as soon as an event has been sent by the linear stream. The goal is to use events in the MPEG-2 transport stream. As soon as an event is triggered the app will display the overlay. By connecting the TV and the mobile phone in the same network, the user can press the Red button and open a detail page in the maxdome app.

To achieve this, the app must listen to stream events and connect both devices to the Companion Screen Manager. A Samsung Galaxy S8 was used for the showcase, which already pre-installed an Android application. The maxdome Android application opens a specific detail page through intents with the URI format maxdome://[series, movies, episodes]/:assetId.

Application Manager
HbbTV executable devices implement an API interface with which it is possible to initialise the Application Manager and the Companion Screen Manager. The Application Manager is used for the life cycle of an application. For example, the standard defines that a broadcast-dependent application should be started in the background and should only be visible when calling ApplicationManager.show(). The application can also be hidden (hide ()), created (createApplication()) and destroyed (destroyApplication()). We start our application in the OnLoad callback. During initial loading, the application is bootstrapped, keys are registered and the companion screen manager is initialized for later connection to the mobile phone.

In order to initialise the Application Manager a reference to an HTML Embedded Object Element is saved, which provides the interface to the HbbTV API. If the oipfObjectFactory already exists, the reference to the OwnerApplication is sufficient: oipfObjectFactory.createApplicationManagerObject().getOwnerApplication(document). With that it is possible to control the application as described above (ApplicationManager.show() === app.show()).

Binding keys
The individual keys can be registered through the reference to the KeySet. In our case the “Red Button” is particularly important.

Companion Screen
The CS (Companion Screen) Manager gives us the opportunity to connect to other devices in the same network. With the app2appRemoteBaseUrl a web socket connection can be established with a mobile device. However, this is only necessary for continuous App2App use and not for our use case, since HbbTV terminals or SmartTVs automatically find other companion devices in the same network.

This setup is required to get our HbbTV application up and running. In summary, it can be said that the following configurations and initialisations must be carried out:

• Setting MIME types
• Initialising Application Manager
• Connecting with Companion Screen Manager
• Registering key bindings and event handlers

Carousel
To show the spot overlays, we’ll listen to stream events in the next part, but first we have to understand what they are. Under Stream Events you can understand the so-called DSM-CC (Digital Storage Media — Command and Control), which is a toolkit for controlling transport streams and — among other things — represents their associated data packets. In the world of broadcasting, data is typically sent in one direction only. The transmitter can send data, however, unlike a classic client-server relationship, the receiver cannot request data. So another solution had to be found. The sender distributes and sends every file in the file system at regular intervals and the receiver waits for the specific file that it has requested. A typical example is teletext. Each page has a specific number and each page is sent in a circle. When the user requests a page, he/she has to wait until it is sent, decoded and displayed again. This solution is called a carousel.

Each so-called module rotates in a circle and the receiver has to wait until the certain time at which the module is sent again. There are two types of carousels — data and object carousel. When an event is received the carousel tells the device where to find the specific data packet. In our showcase the object contains the location in form of an URL which includes the XML for the stream event. The IRT had successfully supported us in this stitching and provided the stream events with their own software.

In the application we register an event listener on the event onStreamEvent and are able to determine from the status whether our module or data packet was triggered.

We can now see how our application regularly shows the spot overlay.

Launching companion screen device

At the beginning we successfully installed the Android Launcher APK on the maxdome mobile application (Samsung Galaxy S8). With this the HbbTV application on the Samsung SmartTV is able to connect to the mobile phone (Companion Screen Device) through the Companion Screen Manager (CS Manager) at startup.

In the next step, the CS Manager queries and outputs a list of all available devices or launcher within the same network. We implement the function csManager.discoverCSLaunchers(callback), which returns all found launcher, i.e. connected devices, as parameters in the callback. We can check on three properties:

As soon as an overlay is triggered and displayed, we can iterate through the entire list of available launchers and start either a native application or a website in the browser. To do this, we have to specify which type and which URI we want to open in the payload. In the callback we get enumId and a corresponding error code with the following values.

With the implementation of companion screens, data can be exchanged without a central server. In this case, applications can even be opened and installed. This applies to both directions, which means that apps on HbbTV can also be started by the mobile phone and, for example, comparable to Apple and Google, can playback on a remote device.

Result

In the meantime, our application continues to rotate and shows one overlay after the other. We can react to one of the spot overlays with the “Red Button”. One interaction is enough and the connected smartphone opens the corresponding detail page for our favourite series. Now we can continue watching on our handheld device.

Conclusion

The current distribution of connected TVs supporting HbbTV is growing. However, the growth of international HbbTV households and their users could be faster. More than 44 million devices are registered in 35 countries in Europe (as of 2018) [HBBTV-2018]. After all, 10 countries more than in the previous year. For most supporters of the standard, it should still go faster. My feeling is that many streaming service providers and manufacturers do not yet dare to completely set this standard, because despite agreement the implementation on the device is not without a doubt and some functions simply do not get the distribution on the TVs that one wanted. It is clear that many of the features can be implemented with web standards and that more questions than answers arise during SmartTV development because there is a lack of documentation and knowledge about. If I look naively at the development environment in which one should develop an HbbTV application, then I face challenges. The helpful FireHbbTV plugin, which could be used to simulate HbbTV applications, can no longer be installed for Firefox since version 52. Alternatives to such as the Vewd Simulator plugin from Opera can unfortunately not depict ApplicationManager or CSManager, as can the HybridTVViewer, which is rather disappointing. Nevertheless, HbbTV has potential, especially in the live stream environment. There it finds its use and opens up ideas for personalized advertising and second screen applications. If you ever have the opportunity to work on an HbbTV project, you need tolerance for the standard implementation and patience during development — but also some money for the playout service. If you skip the hurdle, you can gain a good understanding and then it is also fun to deal with the topic in more depth. I currently see a future for the standard itself, but it is fading due to the slow implementation and the extensive testing of released features.

Further Links

1. GitLab Repository: https://gitlab.com/goo.fi/hbbtv.goo.fi
2. Companion Screen: https://www.fokus.fraunhofer.de/go/hbbtv
3. Data Carousel: https://www.tvwithoutborders.com/tutorials/dtv_intro/how-to-become-an-expert-in-dsm-cc/data-carousels/
4. HbbTV.org: https://www.hbbtv.org/resource-library/specifications/
5. HbbTV 2.0.2 Explained: https://www.hbbtv.org/wp-content/uploads/2018/02/HbbTV-SPEC-00270-003-HbbTV-202-Explained.pdf
6. HbbTV 1.5.1 Spezifikation: https://www.etsi.org/deliver/etsi_ts/102700_102799/102796/01.05.01_60/ts_102796v010501p.pdf
7. Implementation Guidelines for Data Broadcasting: https://www.etsi.org/deliver/etsi_tr/101200_101299/101202/01.02.01_60/tr_101202v010201p.pdf
8. How to become an expert in DSM-CC: https://www.tvwithoutborders.com/tutorials/dtv_intro/how-to-become-an-expert-in-dsm-cc/
9. DSM-CC Multi-Protocol Encapsulation: https://www.tvwithoutborders.com/tutorials/dtv_intro/how-to-become-an-expert-in-dsm-cc/dsm-cc-multi-protocol-encapsulation/

References

[HBBTV-2.0.1] HbbTV 2.0.1 Specification
https://www.hbbtv.org/wp-content/uploads/2015/07/HbbTV-SPEC20-00023-001-HbbTV_2.0.1_specification_for_publication_clean.pdf

[HBBTV-2.0.2] HbbTV 2.0.2 Specification
https://www.hbbtv.org/wp-content/uploads/2018/02/HbbTV_v202_specification_2018_02_16.pdf

[HBBTV-2014] HbbTV Overview by Guiseppe Pascal TPAC 2014
https://www.w3.org/2014/10/tv-ig-meeting/TPAC%20-%20HbbTV%20Overview%20-%20Giuseppe%20Pascale.pdf

[HBBTV-2018] HbbTV Update by Chris Needham TPAC 2018
https://www.w3.org/2011/webtv/wiki/images/7/76/W3C_Media%26Ent_IG_HbbTV_Update_2018.pdf

[FAME] FAME Fraunhofer Application for Biathlon Worldcup live in 360°
https://www.fokus.fraunhofer.de/en/fame/biathlon360

[ZDF] ZDF Service and Help
https://www.zdf.de/service-und-hilfe/smarttv-100.html

[IEEE] Crinon, Regis & Bhat, Dinkar & Catapano, David & Thomas, Gomer & Loo, James & Bang, Gun. (2006). Data Broadcasting and Interactive Television. Proceedings of the IEEE. 94. 102–118. 10.1109/JPROC.2005.861020.