TRIANGLE Project

This is the very last newsletter from the TRIANGLE project on 5G Applications and Devices Benchmarking. We have learned a lot and enjoyed working together to build a testbed and help small companies and operators to test devices and applications in order to be 5G ready. We have also defined a framework for the testing of Quality of Experience of devices and applications using 5G networks. TRIANGLE contributed to the Global Certification Forum for 5G certification standards, to NGMN to define 5G Trial KPIs and contributed a prolific number of important scientific papers, presentations and demos at major annual events such as MWC, EuCNC and Globecomm to name a few.

We will see you at EuCNC2019 for a final hurrah and demonstration of the 5G testing capabilities.

Best wishes for a successful device and/or application on future 5G networks!

From the TRIANGLE Consortium

TRIANGLE has released a comprehensive document containing all the test specifications developed in the project. The document can be downloaded here.

The Signals Research Group (SRG) carried out an independent benchmark study of 3GPP-based IoT devices in collaboration with the H2020 EU TRIANGLE Project and the Connectivity Services Section at Aalborg University. In order to do this, the power consumption and energy requirements of Cat M1 and Cat NB1 modules and devices were analysed in a range of network conditions and usage scenarios. LTE Cat M1/NB1 is a cellular technology specifically designed for the needs of applications targeting the Internet of Things (IoT) or machine-to-machine (M2M) communications and part of the same 3GPP release 13 standard that defined narrowband IoT.

TRIANGLE performed the tests on behalf of SRG for this analysis, using the TRIANGLE testbed.
The voltage and current of various devices were measured with a high sampling rate. The testing allowed the battery life of the devices under test to be estimated with great accuracy. Additionally, the tests enabled the energy budget to be allocated to each IoT state (PSM, DRX, Synch, etc.). The results showed the battery life of devices exceeded or fell short of the 3GPP target threshold of 10 years. A preview or the full 3GPP IoT benchmark study is available.

The TRIANGLE Project funded three waves of Open Calls for experimenter. This helped developers and operators to get their devices and applications 5G ready, helped increase the capabilities of the Testbed and provided valuable feedback to the TRIANGLE consortium on the benefits of using the Testbed but also to identify issues and improvements to the Testbed for future users.

Here is a non-exhaustive list of the applicants who used the testbed for 5G testing purposes:

1. atSistemas adapted their application in order to assess the performance of the Augmented Reality mobile app 5MARCAS while serving High Quality audio-visual content in real-time.
2. D-Cube established the QoE characteristics they were looking to test and then assessed various visual quality levels and production conditions of an immersive game that allows players to compete in shared virtual arenas using their reconstructed realistic appearance as their in-game avatars.
3. The Health5G@Home experiment from ThinkIN was aimed at validating the ability of 5G technology to effectively enable the delivery of advanced home health and care services
4. The objective of Dual-RoC Experiment from Politecnico Milano was to experimentally observe and demonstrate the feasibility of the interplay between an all-analog C-RAN architecture based on LAN cables and Multi-RAT.
5. The 5G-Bot experiment by Infolysis was to test the behaviour of chatbot applications under different mobile reception conditions, network traffic, processing power schemes and battery utilisation plans in order to quantify and assess their impact on the QoE level as perceived by the user and to stress the reliability of the chatbot app under different network conditions.
6. The motivation of the PHYSCHED experiment by IS-Wireless, was to explore the design space for improvement and further implementation of a LTE eNB scheduling algorithm which would be part of Software-defined RAN for 4G and 5G.
7. Comensus used TRIANGLE Testbed to test the wireless link performance for different location and user density in different real-life network scenarios.
8. The CNIT experiment DiMoVis involved a distributed mobile video surveillance system with a huge number of IP cameras operating over SDN/NFV infrastructures, a Network Orchestrator for dynamic set-up and adaptation of data delivery paths in SDN/NFV networks, and a Virtual Network Function (VNF) that is able to identify mobile terminal handovers and sending this information to the Network Orchestrator to arrange paths accordingly.
9. The Inova experiment SPEEDY was to test a wearable device which measures the ECG with state-of-the-art embroidered sensors locally, and the processed ECG data is then sent to the servers directly from a built in 4G LTE modem.
10. Digiotouch VR is a virtual application developed by Digiotouch OU and the experiment tested the reliability, energy consumption, network resource usage and user experience of the application on several different 5G network scenarios with three VR videos.
11. Cuatroochenta wished to analyse the performance of their mobile application called Acércate, the main feature of which is to be able to identify the location of key people using the GPS location in different situations. The tests detected where the app suffers degradation on the performance and excessive consumption of resources.
12. Finally, Solbyte used TRIANGLE to test the performance of two mobile applications under different network conditions and to monitor the impact of network performance on User Experience and on battery life.

TRIANGLE have created a brochure for the project with a summary of the main achievements. The brochure can be downloaded here.

The final review of the TRIANGLE project takes place in UMA premises in Malaga on 4 – 5 of April 2019. There is an exciting line up of presentations and demonstrations from developers such as Inova, Digiotouch, CNIT and StreamOwl among others.

We are sadly coming to the end of a very successful three years in the TRIANGLE project. We have exceeded our projected goals including attracting many applicants from industry and institutes across Europe, testing the mobile devices and software applications of those applicants, and very proudly, defining a framework for the testing of Quality of Experience of devices and applications using 5G networks. We have contributed significantly to the furthering of testing and certifying 5G devices and applications for industry, in a very meaningful and practical way. Not least of which we have contributed to the Global Certification Forum and indeed had a meeting in San Diego this week.

We have had a prolific number of weighty scientific papers, presentations and demos at major annual events such as MWC, EuCNC and Globecomm to name a few. It has been a super consortium to work with and we have enjoyed every minute of it.

We will produce one final newsletter next year to finish up. Thanks for tuning in!

Happy Christmas to you and your families and a peaceful and prosperous New Year!

From the TRIANGLE Consortium

San Diego, 11 – 13 December 2018

TRIANGLE presented a contribution around 5G user experience measurement for device and application to the Global Certification Forum. The presentation took place during the Steering Group meeting #77 in San Diego, USA.

The TRIANGLE consortium has been working hard over the last 3 years to develop a framework which can be used for quantifying the user experience for device and application in 5G networks. This ground-breaking work has led to a large set of insights, technical documents and a series of experiments performed with leading edge companies for device and application development. The TRIANGLE consortium is happy to report that our work has been presented to leading bodies in the 5G certification area.

The Global Certification Forum (GCF) enables the global wireless industry to deliver a connected world, with mobile and IoT device certification program that ensure quality, connectivity and interoperability, whatever the technology. GCF today has over 315 dynamic members from major operators, manufacturers and the test industry, working collaboratively and in association with key industry partners on the successful delivery of 5G and IoT connectivity. For more information, please visit: https://www.globalcertificationforum.org

TRIANGLE also recommends interested parties to read the following document published by NGMN: Definition of the testing framework for the NGMN 5g pre-commercial networks trials dated from January 2018.

One of the key objectives of the Triangle project was to develop an automated process for evaluating the QoE (Quality of Experience) for the end user of an application or a device. The success of 5G (the fifth generation of mobile communications), and to some extent that of 4G, depends on its capability to seamlessly deliver applications and services with good QoE. Along with the user, QoE is important to network operators, product manufacturers (both hardware and software) and service providers. However, QoE as such, is not well defined for devices or applications and QoE measurements require intensive human experiments. It is therefore currerntly difficult to assess QoE at the service level.

The TRIANGLE project has been working on QoE aspects to advance the state of the art. The main contribution of the TRIANGLE project is the provision of a framework that generalizes QoE computation and enables the execution of extensive and repeatable test campaigns to obtain meaningful QoE scores. The TRIANGLE project has also defined a methodology, which is based on the transformation and aggregation of KPIs, its transformation into synthetic Mean Opinion Score (MOS) values and its aggregation over the different domains and use cases. The process produces a final TRIANGLE mark, a single quality score that could eventually be used to certify applications. The approach developed in TRIANGLE is a methodology flexible enough to generalize the computation of QoE for any application/service. The methodology has been validated testing the Dynamic Adaptive Streaming over HTTP (DASH) implementation in media player Exoplayer.

The complete work on the QoE evaluation framework has been released in TRIANGLE Deliverable D2.7 which can be requested from the consortium. Aside from the deliverable, the TRIANGLE consortium will also shortly publish a paper entitled “QoE Evaluation: The TRIANGLE Testbed Approach” in the special issue Testbeds for Future Wireless Networks of the Wireless Communications and Mobile Computing journal https://www.hindawi.com.

TRIANGLE was represented by DEKRA at the 6th User Conference on Advanced Automated Testing (UCAAT). The TRIANGLE presentation was on “Testing Solution for Virtual Reality Applications” and the speaker bio of DEKRA’s Carlos Cardenas can be found here. The ETSI User Conference on Advanced Automated Testing (UCAAT) took place in Paris between 16-18 October 2018 and gathered users, vendors and service providers around testing and automation.

The conference was organized by Testing Solutions and Services, and sponsored by Sogeti, Kalifast, Conformiq and Spirent. The event was supported by ETSI’s Technical Committee on Method for Testing and Specification.