PhyNetLab: An IoT-Based Warehouse Testbed

Future warehouses will be made of modular embedded entities with communication ability and energy aware operation attached to the traditional materials handling and warehousing objects. This advancement is mainly to fulfill the flexibility and scalability needs of the emerging warehouses. However, it leads to a new layer of complexity during development and evaluation of such systems due to the multidisciplinarity in logistics, embedded systems, and wireless communications. Although each discipline provides theoretical approaches and simulations for these tasks, many issues are often discovered in a real deployment of the full system. In this paper we introduce PhyNetLab as a real scale warehouse testbed made of cyber physical objects (PhyNodes) developed for this type of application. The presented platform provides a possibility to check the industrial requirement of an IoT-based warehouse in addition to the typical wireless sensor networks tests. We describe the hardware and software components of the nodes in addition to the overall structure of the testbed. Finally, we will demonstrate the advantages of the testbed by evaluating the performance of the ETSI compliant radio channel access procedure for an IoT warehouse.

Abstract-Future warehouses will be made of modular embedded entities with communication ability and energy aware operation attached to the traditional materials handling and warehousing objects. This advancement is mainly to fulfill the flexibility and scalability needs of the emerging warehouses. However, it leads to a new layer of complexity during development and evaluation of such systems due to the multidisciplinarity in logistics, embedded systems, and wireless communications. Although each discipline provides theoretical approaches and simulations for these tasks, many issues are often discovered in a real deployment of the full system. In this paper we introduce PhyNetLab as a real scale warehouse testbed made of cyber physical objects (PhyNodes) developed for this type of application. The presented platform provides a possibility to check the industrial requirement of an IoT-based warehouse in addition to the typical wireless sensor networks tests. We describe the hardware and software components of the nodes in addition to the overall structure of the testbed. Finally, we will demonstrate the advantages of the testbed by evaluating the performance of the ETSI compliant radio channel access procedure for an IoT warehouse.

I. INTRODUCTION
F LEXIBILITY of system, modularity and reliable throughput, in addition to the scalability play major roles in the quality of materials handling and warehousing systems [1]. The Integration of embedded devices in current systems is the first step into this direction that will evolve classic warehouses into decentralized modular systems with improved performance. Multiple research instances of logistics turned into implementing distributed Cyber Physical Systems (CPS) for modern production, transportation and distribution strategies [2]. But the full potential will be reached when these entities communicate with each other and fulfill their tasks autonomously without any central management units.
Smart connectivity to communicate with the available networks and using them for context-aware computation is an indispensable part of Internet of Things (IoT) [3]. Therefore, realization of these CPS in the field of materials handling is also a move in the overall direction of the Industry 4.0 revolution with the emerging concept of IoT [1], [2].
Although the term IoT was first made in 1999 by Kevin Ashton in the context of supply chain management, the definition has been expanded into a wide range of applications during the past decade due to its interdisciplinary nature [3]- [5]. In [6] the three main paradigms for the realization of IoT are defined as Internet oriented, Things oriented, and Semantic oriented.
Warehousing application in a materials handling facility is exactly such a field which intersects all of these IoT aspects.
For such systems, hardware and software development becomes a challenging process, since the code base for controlling a large swarm of devices needs to be maintainable, and the implementation of new features requires a well organized software milieu with a testbed to avoid disastrous mistakes.
Another challenge is imposed by radio communications. As bandwidth and range of radio connections are limited, a great mass of intelligent containers need to reduce and adapt their communication behavior accordingly. Furthermore, the channel access must be organized very efficiently to avoid collisions and the waste of scarce energy. As we will show in this paper, commonly used approaches lead to a catastrophic increase in energy consumption in large-scale scenarios.
To tackle these challenges and enable the development, evaluation and validation of a real-life deployments, we present PhyNetLab (cf. Fig. 1), a large scale IoT testbed for future logistic systems, and perform an exemplary evaluation of an established channel access scheme.

II. RELATED WORKS
Wireless Sensor Networks (WSN) have been a major research topic during the last few years [7] which spans a diverse research area, e.g. routing algorithms, energy harvesting, management, security and privacy. According to this versatile range of work, wide variation of WSN testbeds are developed. In addition, there are federations of WSNs combining multiple installations into a single wide spread platform. Some of the most famous platforms are: a) MoteLab: A wireless network developed by Harvard and published in 2005 [8]. It was one of the first fully developed WSNs. MoteLab is an open source tool that is accessible on the Internet. Its web access facilitates remote programming and user scheduling.
b) Future Internet of Things: FIT/IoT-Lab [9] is a heterogeneous large-scale research facility with a federation of more than 6 locations and more than 2000 nodes altogether [10]. To interact with nodes it has command line interfaces through user virtual machines. c) Indriya: is a low-cost 3D WSN testbed implemented at the National University of Singapore [11]. The 127 wireless nodes are connected with active USB cables. This USB infrastructure provides a back channel for remote programming and powering the sensor nodes. d) WISEBED: is an IoT research facility with a heterogeneous implementation where each partner maintains its own testbed [12]. There is also an overlay network giving access to all testbeds as one, large IoT implementation for analysis. Anyhow, each testbed can also be handled separately [10].
Beyond that, numerous other WSNs are shown in surveys, such as [4], [7], which categorize them into general testbeds, server-based testbeds, single PC-based testbeds, multiple site testbeds, in-band management traffic testbeds, and specialized testbeds.
According to this categorization, PhyNetLab is considered to be a specialized testbed because it is designed to be a wireless sensor network testbed, which is built very close to a real world scenario. The major research challenges are to develop communication protocols and energy-aware syntactics, which are required for a deep integration in industrial systems such as a materials handling facility.

III. SYSTEM STRUCTURE AND HARDWARE COMPONENTS
Here, we give an overview of PhyNetLab, which is shown in Fig. 1. It is located in a large logistics hall providing space for a continuously extending amount of containers. Each container carries a PhyNode at its front which enables communication capabilities with the infrastructure. They are attached to equally distributed Access Points (APs), which serve as gateways to the internet via an optimized Long Term Evolution (LTE) link [13]. This cellular structure reduces the necessary transmission power and increases the network capacity by reusing radio channels. A detailed description of the PhyNetLab and of its components is presented in [10]. Due to limited space, this paper only briefly introduces the PhyNode platform, which is most important for the presented experiment.
The PhyNode (cf. Fig. 2) consists of two parts, a master network board for management and the actual experiment platform, which is a Swappable Slave Board (SSB). The management platform spans a ZigBee backbone network over the testbed and can be used, e.g., for updating the slave board firmware. The heart of the SSB is a ferroelectric random access memory (FRAM)-based MCU MSP430FR5969. Comparing to Fig. 2. Photography of the PhyNode, which is attached to every container in the PhyNetLab. It is equipped with a solar cell, display, radio interface, numerous sensors and a rechargeable battery.
conventional flash RAM, the 64 kB FRAM in the MCU is very durable concerning memory access and highly energy efficient. The radio communication is performed by a low-power Sub-1 GHz transceiver TI CC1200 in the 868 MHz Band for Short Range Devices (SRD). In addition, the board includes sensors, which capture accelerations, temperature, color, infrared and ambient light, intended for energy harvesting research. Interaction with humans is possible using buttons and a small LCD.
The SSB can be programmed to work as an energy neutral device using energy harvesting. For this purpose, and also for generic battery management, we integrated an ultra low power harvester IC with boost charger and autonomous power multiplexer. It is designed to allow switching to an alternative energy source if the solar charged energy storage is depleted.

IV. SOFTWARE COMPONENTS
To enable a rapid development of IoT applications in ultra low-power systems, we developed an IoT end device operating system Kratos. It supports developers with a comprehensive set of C++ library functions that can be "tailored" to suit the needs of a distinct application and thus can save resources by only deploying the required system components.
To support the modularization that is necessary for keeping an operating system highly configurable, we used AspectC++, a set of language extensions to facilitate aspect-oriented programming concepts into C and C++ [14], [15].
Kratos delivers interfaces to different system architectures and features a set of useful system properties like interrupt synchronization, preemptive thread scheduling, peripheral driver interfaces and power management mechanisms.
Kratos was mainly developed with PhyNetLab in mind, and hence fully supports the PhyNode platform. Furthermore, PhyNetLab and Kratos allow mutual research on hardware and software design for large-scale and energy-aware IoT deployments. By that, PhyNetLab gives us insights on how to design energy and network-aware software components for an operating system under heavy resource constraints. In the other direction, software requirements for enabling maintainability and the deployment of applications for PhyNetLab influenced the choice of hardware components used in the PhyNode.
The main focus of Kratos is on power management. Here, we developed methods to incorporate detailed energy models of all components of a system into the system drivers [16]. These energy models can be derived automatically by using power measurement techniques in a loop. This enables us to track the power consumption of all nodes over time and drive application decisions on this knowledge. In this paper we utilize this capability to track the power consumption of the radio transceiver, since this is the main consumer of the system in the presented setup. We postpone the presentation of the comprehensive details about the novel design principles of Kratos to a future work. Additionally, the Kratos source-code will be freely available as open source as soon as we consider it mature enough for the use by external users.
On the other hand, the software of the AP transparently forwards messages between the PhyNodes and the enveloping application, which provides an external interface to the PhyNetLab. For this paper, the application emulates typical interactions with the PhyNodes in a typical warehousing scenario, e.g., keeping track of available PhyNodes via device discovery and dynamic addressing, altering the goods stored in particular containers, or performing queries for single goods in the warehouse.

V. COMMUNICATION
The radio interface of the PhyNode is highly configurable in terms of, e.g., modulation, power consumption, data rate, channel access and interrupt signals. In this paper, an exemplary radio driver implementation configures the interface as shown in Tab. I. The expected high number of devices in the warehouse requires a collision-avoiding channel access mechanism. Otherwise, in case of an uncontrolled random access, the throughput over the air interface would achieve at maximum 18 % because of collisions [17], [18]. Since collisions require (possibly multiple) retransmission, this would waste a large amount of scarce energy. In addition, multiple replies to a single broadcast (or multicast) message might always provoke a collision, in case of an equal message-processing time.
Therefore, we implemented a Listen Before Talk (LBT) channel assessment in our setup, which conforms to the ETSI SRD standard [19] and complies with the regulatory constraints of that band. Instead of just accessing the channel for the transmission of a new data packet, the transceiver first listens on the channel for a total back-off time t L = t F + t P S , where t F is fixed to 5 ms and t P S is a randomly chosen between 0 ms and 5 ms. An example of the described channel access procedure is shown in Fig. 3  pending transmission detects no radio activity throughout t F , it shall subsequently perform its transmission by resetting its t P S to zero. However, if the transceiver detects any radio activity during t F (DEVICE B), the back-off timer is halted and will be restarted, when the channel is free again. In this case the device must back-off for the full total period t L .
The response of broadcast messages requires a special handling: After receiving a broadcast message, multiple devices may attempt to reply simultaneously. Although those devices back-off for the period t F , their messages will collide if no other activity happens on the channel during this period. Therefore, an additional random back-off of 0 ms to 5 ms is added in advance to each broadcast reply.
In order to reduce the power consumption of the transceiver during idle times, we implemented a low power listening mode into the device driver where the transceiver is put into a low power sleep mode and wakes up in intervals of 4.7 ms for 0.2 ms. This reduces the idle power draw from 23 mA in continuous RX to 1.5 mA in low power listening mode, but requires the AP to transmit extended packet preambles. The low power listen mechanism is disabled during a pending transmission in order to keep track of the exact moment, where an active transmission on the channel ends. This ensures the conformance to the previously described ETSI clear channel assessment.

VI. APPLICATION EXAMPLE
In order to bring out the benefits of the proposed PhyNet-Lab, we present an evaluation of a typical application in future warehouses. Within such warehouses, incoming orders might autonomously send broadcast messages into the network to poll for particular goods. These messages will be replied by only those PhyNodes, which contain the requested product, and inform the inquirer about the contained quantity. He then selects a subset of PhyNodes, which fulfill the requested quantity, and requests to the network a delivery of those goods.
This use case arises questions about radio channel congestion, reliability and energy consumption of the PhyNodes when replying to those polls simultaneously. To answer them, we performed real field measurements in the PhyNetLab (cf. Fig. 1) and set up a network of 38 PhyNodes attached to an AP. The nodes were distributed in the PhyNetLab in such a manner, that each node can perceive the activity of any other PhyNode in the network. Therefore, the free channel assessment algorithm is not negatively influenced by hidden stations. Each measurement run consists of assigning the same product to a subset of the attached PhyNodes and ten polls for that product by the AP. As soon as a PhyNode receives a matching poll message for the assigned product, it replies the call with its address and the contained amount with respect to the clear channel assessment procedure described in Sec. V. The remaining PhyNodes perform no transmissions during the run. All participants in the network count the number of successfully received and sent messages on their side and transmit those statistics to the AP after each run. The statistics also include the accounted energy of the radio transceiver, as explained in Sec. IV. For comparability, each run comprises an interval of 11.75 s, hence does not depend on the number of exchanged messages. The interval is encapsulated by broadcast messages for starting and stopping a run. These messages are required, because statistics and setup messages of the PhyNodes are exchanged over the same link as the performance runs but must be excluded from statistics in this series of measurements. Therefore, each PhyNode resets its statistics and accounted energy when receiving a start message and freezes the values after receiving a stop message.

VII. EVALUATION AND RESULTS
In this section we evaluate the measurements from the application example in Sec. VI, which challenged the ETSI collision avoidance algorithm of the radio interface during product polls to the warehouse. These product polls, which are transmitted as broadcasts by the AP into the network, trigger instantly numerous PhyNodes to transmit a reply message. Fig. 4 shows the achieved packet throughput T during the measurements. It is defined as the ratio where N RX is the number of successfully received messages by the AP and N T X is the number of transmitted packets by each PhyNode from the active set A. The algorithm performs very efficient for low numbers of concurring devices and enables a throughput above 80 % in case of 8 or less devices. In the range of 1 to 17 devices the throughput behaves nearly linearly and undercuts 50 % in case of 17 devices. Higher numbers of simultaneous attempts have only a negligible effect on the throughput, which stays nearly constant at a level of 50 %.
While this degree of throughput is typically not acceptable in a common communication system, it is still reasonable in the addressed logistics scenario. Considering, i.e., an incoming order for a particular product stored in the warehouse, where numerous containers comprise many entities of the demanded product, it is not necessarily required to receive all replies. Instead, it might be sufficient that enough containers reply to satisfy the demanded amount. Otherwise the poll could be repeated and uncover further containers, which suffered of a collision in the first place.
More critical is the impact of congestions on the energy consumption of the PhyNodes, which is shown in Fig. 5. In case of a single replying device, which reflects the minimum required energy for a test run, the radio interface consumes in average 57 mJ. By adding one additional device, the average energy consumption of each device more than doubles to 123 mJ. This is caused by a significantly higher amount of time spent in active receive mode: A device needs to wait in receive mode until a preceding transmission finishes plus the required back-off interval. The other way round, a device that already transmitted its packet and falls back into lowpower listening mode will be waked up by the replies of the remaining devices. Although the transceiver's logic discards those packets very quickly due to a mismatching address, the transceiver still spends much more time in active receive mode.
With an increasing number of concurring replies, the energy consumption of all devices raises to 1030 mJ in average for 38 active devices, which is an increase by factor 18. In addition, the deviation of the energy consumption increases as well. This is caused by the varying time instant, where the device finally transmits its packet. If a device sends its packet very late by repeatedly hitting a large back-off interval during the contention phase, its transceiver continuously stays in an active receive mode to keep track of the ongoing transmissions. Conversely, if the transmission succeeds quickly, the devices can fall back into low power listening mode and save energy.
The results show, that the standardized ETSI collision avoidance algorithm may conform to the requirements of a logistics scenario in terms of sufficient throughput, but the large impact on the energy consumption of all devices in the network cannot be neglected for this use case. Hence, the distributed channel access brings great potential for optimizations. For example, devices could send their replies on a different channel and use a blind back-off mechanism, which shuts down the receiver during the back-off period. We will elaborate and evaluate those approaches in future works with the help of PhyNetLab.

VIII. CONCLUSION
In this paper we presented PhyNetLab, an IoT testbed which enables a real-life evaluation of future smart logistic approaches with wireless connected containers (PhyNodes). The scope of this paper focused on the evaluation of common channel access approaches for radio communications in such an industrial IoT deployment. For this purpose, we implemented a radio-interface driver, which conforms to the ETSI specification for clear channel assessment of Short Range Devices (SRD). The driver is integrated in a novel embedded operating system Kratos, which highly customisable and includes efficient mechanisms for energy management and energy accounting of distinct peripheral components.
On this basis we performed a throughput and energy analysis of a realistic logistics application in the presented testbed, which is polling for a particular product in a warehouse. We showed, that synchronous replies of many PhyNodes to a single poll challenge the channel access algorithm and the energy demand of the entire network. Although the packet loss rate reaches 50 % at high numbers of replying PhyNodes, it is still acceptable for this use case, as long as the number of replies satisfies the demanded amount of goods. But the resulting storm of numerous reply packages leads to a significant increase of energy consumption of every single PhyNode in the network. This is caused by frequent wake ups of the radio chip due to radio channel activity and a larger listening period while waiting for a clear channel assessment. Therefore, such deployments as smart warehouses cannot rely on commonly established approaches for radio communication, but rather need specialised solutions for this application.
In future work we will incorporate the PhyNetLab to develop and validate more energy-efficient communication protocols, which satisfy the the demands for scalability, extreme energy-efficiency, and a reasonable latency.