Determinism on the Shop Floor

Until recently, communications technologies like Ethernet and OPC Unified Architecture (UA) have experienced limited success in supporting complex industrial processes with real-time requirements. This, in turn, has restricted the applications that discrete and process manufacturers could adopt. New developments, however, promise to address these shortcomings.

In March, the OPC Foundation released the Publish-Subscribe (PubSub) extension of the OPC UA standard, aimed at pushing communications enhancements to controllers, sensors and embedded devices at the deepest levels of the shop floor. Concurrently, the IEEE has extended the Ethernet, providing for deterministic communications through a set of IEEE 802 standards that enable time-sensitive networking (TSN).

Industry technology developers contend these steps will provide the building blocks required to create a holistic communications infrastructure tailored to better meet the needs of the industrial internet of things (IIoT). By combining PubSub and TSN with OPC UA and standard Ethernet communications, they aim to deliver deterministic performance, improved quality of service and solid security while maintaining broad interoperability among third-party devices and systems.

Adding PubSub

A key element of this technology mix, PubSub, homes in on issues arising from the IIoT’s emergence and the consequent explosive growth of devices and data serviced by automation networks. This technology seeks to provide a more efficient, uniform model for distributing data and events within production environments.

Until recently, the standard model for devices communicating on an industrial network was the request-response, or client-server, paradigm. Using this approach, a client device (e.g., human-machine interfaces) requests data or services, and a server (e.g., programmable logic controllers or programmable automation controllers) responds to those needs.

The catch here is that each client must open a direct connection with the server. As the IIoT takes shape and the amount of data generated at the machine level grows, network performance can degrade, hindering or precluding certain control functions.

“The client-server communication model puts a strain on the server, or the producer of data, because each client initiates a separate transaction with the server, and this strain grows with the number of clients,” says Abdul Jabbar, senior researcher in the edge computing group at GE Global Research. “In industrial controls, the servers of data are also critical to machine operation. This poses a problem. If hundreds of clients request their own copy of the same data, the controller can become overloaded serving these requests.”

The PubSub model alleviates this problem by using a central source called a broker, or server, to act as a data clearinghouse. Clients can publish data to the broker or subscribe to data from the broker, or both.

A broad cross-section of the networking industry has come together to update standard Ethernet, creating a communications infrastructure tailored to deliver deterministic performance, improved quality of service and broad interoperability among third-party devices and systems. Image courtesy of Avnu Alliance.

With PubSub, the network defines a fixed time window in which clients and brokers exchange data. The model eliminates point-to-point connections, relying solely on multi-point links via user datagram protocol. This relieves the broker of individual client requests. Instead, the broker publishes each dataset once. This reduces the workload of controllers and cuts network traffic, and enables smaller embedded control devices that lack resources for typical client-server transactions to publish.

PubSub further reduces network traffic by supporting transactions that only contain data that has changed since the previous data transfer. Clients and brokers transmit data only when the content has changed. This avoids repetitive blind requests and data transfers that ensure the acquisition of the most current data. Compared with the request-response model, the number of connections decreases, and they are replaced by a lightweight link for each client-to-server connection. This link remains open and carries only two types of messages: changed data and a tiny heartbeat that tells the broker that the client is still present. The broker does not store data, it only moves it from publishers to subscribers.

The PubSub model sets the stage for deterministic network performance. “The publish-subscribe model, in and of itself, does not enable deterministic communications,” says Jabbar. “However, since the data transmission from publishers is quantized through datasets and relatively stable/static over time, it is well suited to deterministic networks, which require predefined data flows.”

In addition, PubSub provides a foundation for OPC UA TSN. “The Publish Subscribe extension for OPC UA is now available in release-candidate form, enabling the exchange of OPC UA over UDP connections,” says Ludwig Leurs, director of Ethernet convergence at Bosch Rexroth, which is a member of the Avnu Alliance. “This is the prerequisite for running OPC UA TSN.”

The bottom line: PubSub makes sense for the IIoT. It efficiently and reliably moves data among several sources and destinations on the plant floor and allows for direct and seamless integration with modern cloud platforms. Access to cloud resources enables manufacturers to leverage modern analytics for system optimization and predictive maintenance.

Picking up Where PubSub Leaves off

The PubSub model defines communications patterns among devices. It does not dictate how data transmission occurs. When a data frame traverses a network switch on the way to its destination, it can encounter traffic on the egress port, leading to buffering and variable delay, breaking the key tenant of real-time communications. TSN addresses this by providing automation and industrial control applications with a system to ensure consistent data delivery from sensors to controllers and actuators.

“TSN ensures that specific traffic is delivered in a timely manner by securing bandwidth and time in the network infrastructure to support coexistence of critical and non-critical forms of traffic,” says Paul Didier, IoT solutions architect at Cisco Systems and Industrial Internet Consortium liaison. “This enables users and vendors to derive benefits from increased connectivity to more devices over a single, standard physical network, enabling services such as VoIP, video, telemetry, analytics and critical control to share the same network.”

To enable these advances, TSN relies on three key features. These include time-aware scheduling, synchronization and frame pre-emption. When combined, these features enable precisely timed transmission of data throughout its path, from source to destination, which is the essence of deterministic communication.

Getting in Sync

A core component of TSN is a common sense of time. Deterministic communication requires timed transmission of data. This, in turn, demands that all network elements have synchronized clocks. In the absence of an out-of-band synchronization mechanism (e.g., GPS) on all nodes, IEEE 802.1 AS provides a mechanism that enables time synchronization.

The specification does this by standardizing the precision time protocol, the working clock used to drive measurement and control functions on end stations, synchronizing the operation of controllers, sensors and actuators with the rest of the system.

System-wide time synchronization makes it possible to schedule transmissions and coordinate traffic with the network to ensure the highest quality of service. In the process, it pushes network performance toward faster motion control and testing, as well as greater quality assurance. TSN devices available on the market can achieve time synchronization within 10 nanoseconds with hardware timestamping.

“When all the parts of a network are running with the same sense of time, traffic can be coordinated based on a schedule, allowing for precise time synchronization and better control of critical traffic types,” Didier says. “With TSN, the scheduled critical traffic is guaranteed, and the non-mission-critical traffic can coexist without disrupting the delivery of the critical traffic.”

Time-Aware Scheduling

When working with IEEE 802.1 AS, the time-aware shaper plays a key role in enabling the creation of truly deterministic networks. Once it is implemented in a network end point or switch, 802.1Qbv operates a gate on every queue of an egress port, providing the network configuration manager with the ability to control the gate state (open or closed) with a schedule.

Time-aware gates immediately precede a bridge’s transmission selection function. Each transmission gate corresponds to a traffic class associated with a specific queue, with potentially multiple queues associated with a given port. A configurable gate control list determines whether a traffic class’ gate is open or closed at any point in time, and the gate control list is executed over a configurable cycle of time.

“Real-time communications imply that data is transferred from the source to the destination instantly, without delay,” says Jabbar. “Ideally, this means the latency is simply bounded by the physics of data transmission and propagation, with no additional delay. With TSN’s time-aware scheduler, it is possible to precisely time the gate state at each network hop in between the source and destination in such a way that a data frame experiences no additional delay or buffering. For example, some switches can control the transmission of a frame within a bound of 100 nanoseconds.”

This kind of traffic control provides all data transmissions with the level of service required. “An industrial control application can minimize latency and path delay variation for control traffic by configuring the gate control list so that high-priority control traffic will transmit at known time intervals,” says Didier. “Similarly, gate operations for lower priority, or best effort, traffic can be allocated in the gate control list to time slots not used by high-priority control traffic, ensuring that all traffic types will flow.”

Frame Preemption

With time synchronization and time-aware scheduling, it is possible to control gate states (open/close) at each network element. This should lead to precisely timed transmissions and therefore deterministic communications—except for one issue: Ethernet is a store-and-forward network. If a frame begins transmission just before the start of a time period reserved for a high-priority transmission, it can extend the lower priority transmission outside its allocated window, potentially interfering with more critical traffic. Therefore, a guard band (an unused portion of the frequency band) equal in size to the largest possible interfering frame must be added before the window starts.

Time-aware traffic scheduling precisely coordinates data transfers across the network, enabling scheduled transmission for given classes of traffic across a TSN-enabled network. Image courtesy of Avnu Alliance.

By using the frame preemption feature defined in IEEE 802.1Qbu and the related physical layer standard interspersing express traffic (802.3br), the network avoids this kind of delay by pausing the lower priority transmission, executing the scheduled transmission of the high-priority frame at the predetermined time, and later resuming the lower priority transmission.

Frame preemption does this by breaking the interfering frame into smaller fragments and inserting a guard band between the two competing transmissions. In this case, the guard band need only be as large as the largest possible interfering fragment instead of the largest possible interfering frame, which would be a less efficient use of network bandwidth.

Frame preemption’s contribution to greater network efficiency, however, doesn’t stop there. Industrial automation applications, such as machine control, typically take the form of long line configurations. To minimize wiring cost and complexity, a typical installation uses a daisy-chain configuration, where each node has external switched ports and an internal port that goes to the end node. Operating under these conditions, the accumulated latency per hop becomes problematic. Frame preemption mitigates this problem by minimizing the frame delay at each hop.

New Design Tools

Looking at the synchronization, time-aware scheduling, and frame-preemption features, it becomes apparent that the TSN standards were created with a system view in mind, rather than a component perspective. Scheduling becomes more calculable from a mathematical perspective, allowing system designers to determine in advance if the network will be successful.

“This dramatically changes the workflow for designing and planning networks,” says Didier. “TSN values network calculus and planning as part of the solution toward managing traffic and guaranteeing performance. In this new paradigm, payload, sampling frequency and maximum latency can all be managed from a system-wide view to calculate flows and configure bridges and infrastructure to meet these demands.”

This holistic approach to network design requires new tools, with new operating parameters. For example, National Instruments, Cisco and Intel have collaborated on an early-access technology platform for TSN that enables developers to more quickly build distributed systems using TSN. The platform can be used to perform synchronized input/output, code execution and deterministic communication for distributed control and measurement loops, all using standard Ethernet.

One of the remaining challenges in realizing deterministic communications entails the task of scheduling and configuration. Because TSN requires timed data transmission at every hop between source and destination, a schedule must be computed for gate control of every queue, on every port, for all elements in the network. This scheduling problem is equivalent to a NP-complete graph-coloring problem, which makes finding a feasible schedule difficult. Furthermore, TSN configuration involves predefining data flow requirements and setting up gate schedules in all network elements.

A number of companies offer tools to simplify these tasks. One of these is TTTech’s network scheduler Slate XNS, a browser-based software package that promises to streamline topology modeling, schedule development and TSN network configuration. The company contends the software’s graphical user interface enables offline network configuration, which provides a topology view or table-based editor for managing components and data streams.

Another development offering is GE’s TSN Toolset, which combines flow requirements acquisition, schedule generation and device programming in a single toolset that can natively integrate with industrial applications.

Through TSN’s methods of modeling, system designers can proactively answer questions about their network’s capability before the system is integrated, and plan for the traffic that needs management. If designs prove inadequate or a solution is not achievable given system constraints, then network design can be modified to accommodate the system requirements.