The Lucid protocol was developed by the WITS PSA (Protocol Standards Association) as a complementary protocol to WITS-DNP3. But why did the WITS-PSA go to the trouble of developing another protocol when they were already in control of an established one? This article tries to answer that question and in doing so will give you a good background to the drivers for the development of Lucid.
Things we learned from using WITS-DNP3
WITS-DNP3 has been in use for over 10 years, primarily within the water companies of the UK. During that time both the users and the WITS-PSA learned lessons from the implementation and rollout of WITS-DNP3. These lessons fed into the design of Lucid. We consider some of the major influences below:
- Large code base. DNP3 is a large protocol covering a wide range of functions, meaning it tends to have a large code base in both master station and field device implementations. A large code base takes more time to integrate into a product, increasing the cost of bringing such a product to market. Lucid is based on lighter IoT technologies like MQTT and JSON, allowing for quicker and therefore lower cost implementations.
- Problem for smaller devices. Although less of an issue these days, the large size of the DNP3 code base can be a concern for the very small devices used in IoT-like products. Lucid’s use of relatively small underlying IoT standards like MQTT and JSON makes it more suited to these smaller devices.
- Chatty protocol. DNP3 is normally used in a “chatty” fashion, with many messages across the network. Lucid’s standard permits a less verbose communications regime allowing devices to send what they need to the comms broker, thus keeping bandwidth usage to a minimum.
- Only two master product offerings. There are many DNP3 Master Stations, but there are only two WITS-DNP3 master stations available in the market. Also, a great deal of work is required to implement WITS-DNP3 on a Master Station. It is therefore unlikely that we will see another WITS-DNP3 Master Station enter the market any time soon. However, the more Master Stations we have, the better for the resilience, and the adoption, of the standard within industry. Lucid is based on common and openly available internet standards which should make adoption much easier. With MQTT as a base, the number of ways Lucid can be deployed is also larger and we hope this will promote many more “Supervisory Application” products, (as Master Stations are called within Lucid).
- Licensing and support costs. Traditionally many WITS-DNP3 Field Device vendors have utilised commercially-available DNP3 stacks to provide the DNP3 functionality for their products. These stacks have licensing and ongoing support costs that end up affecting the cost of the product in the market. Lucid utilises open standards which generally have commonly available, free-to-use implementations, reducing the final product cost.
- Testing costs. As well as writing the code it is obviously essential to test the code; DNP3 and WITS-DNP3 both have large testing systems to ensure their correct implementation. These systems are not without cost, both in procuring and supporting them, but also in setting up and using them. This again has a knock-on effect to the cost of the product in the marketplace. Lucid uses smaller underlying protocols to control this testing burden and support lower costs to market whilst maintaining the same high levels of testing capability. Those same protocols normally also have freely available testing tools, further easing the test burden.
Removing barriers for new vendors
New vendors coming to the WITS-DNP3 marketplace face a number of barriers to adoption of their products which they have to consider carefully before commencing on their implementations. Below are a few of the most important considerations and how we think Lucid addresses these.
- Market. If you are building a product to compete in an already existing market, then you are already at a disadvantage. The WITS-DNP3 Field Device marketplace is somewhat like this and closed publication of the protocol (you have to be a member of WITS-PSA) does not necessarily help with this. Lucid will be published totally openly on the internet to allow organisations to assess, pick up and utilise the protocol – hopefully resulting in the growth of the market. The larger the market, the better for users, who will have a greater choice of device. A larger market also benefits vendors as there are more potential customers.
- Staff availability. WITS-DNP3 used a protocol which does not have widespread development skills availability. This makes staff harder to recruit and retain. Lucid has been designed on commonly-available underlying protocols with widespread use in the IoT and IT arena and therefore should make the acquisition of capable staff easier.
- Complexity. WITS-DNP3 is quite a complex protocol which builds on an already large protocol in DNP3. Lucid eases that complexity by using simpler underlying internet and IoT based standards. The reduction in complexity should pay dividends through the design, development and testing lifecycle for vendors adopting the protocol. For example, DNP3’s binary protocol requires the use of specialised tools to decode and examine message, whilst Lucid uses a human readable message format.
- Cost. There are several costs which may come to bear when creating and maintaining a WITS-DNP3 Field Device. These include DNP3 stack purchase and support, DNP3 test tools and support, WITS protocol and testing documentation and less tangible costs like complexity and staff acquisition which we have discussed above. Lucid addresses these costs through selection of simpler and less expensive underlying protocols and an open attitude to publication.
Making testing quicker and easier
Testing a protocol which provides many functions will always have its fair share of issues. Initially WITS-PSA wanted devices which conformed to the WITS-DNP3 protocol to be plug and play; a Field Device that demonstrated conformance to the protocol through testing should be easily plugged in to a Master Station that supported that version of protocol. For many and varied reasons, this did not always happen, leaving a testing burden on users which was not intended.
The reasons that plug’n’play was not achieved include some of the following:
- Tests were normally performed on a single device, but problems sometimes only manifested themselves when many devices were added to the system.
- Tests are always non-exhaustive, but in the case of WITS-DNP3 maybe they did not provide enough coverage to smooth the introduction of products into service.
- The testing was complex and expensive to prepare and execute.
- Re-certification was as difficult as initial testing and therefore regression testing was uncommon, leading to new versions of firmware not re-running the full recertification tests.
Lucid plans to address these issues by a creating a community-developed test harness which is available to all those developers and users. This is expected to be available within the WITS-PSA community only and it is still a work in progress. The intention is that a test harness be available for a Field Device and another for a Supervisory Application. Initially, testing will be of main functions only, but it is hoped that the development community as a whole will slowly add to the lists of tests over time, growing them to be a large and appropriately representative test set. Current vendors would be able to share the costs of development of the test harness, a cost they all bear individually at the moment for WITS-DNP3. Users would also be able to run the tests and add to the tests to ensure that devices they intend to use should work as expected. New vendors would also have improved access to testing facilities for lower cost.
Controlling a device’s entire configuration
WITS-DNP3 controlled configuration of Field Devices through two methods:
- The Bulk Configuration File (BCF), which was a proprietary file understood only by the device and the device vendors Configuration Application (CA), the software used to configure the device.
- Incremental Configuration file (IC), which is a binary description of some of the device’s configuration which is specified and understood by WITS-DNP3.
Lucid simplifies this configuration landscape by reducing the two methods to a single JSON configuration file that is specified and used by Lucid.
Lucid has been designed so that all configuration can be included in the configuration advertised by the Field Device. There is no proprietary binary “blob” or BCF, just a specified format of JSON file which can be consumed and produced by any Supervisory Application. Therefore, no special tool is required to create and / or set up the configuration. The hope is that this will improve the plug’n’play capabilities of Lucid devices, reduce the number of CA tools a user may have to learn, and to permit top-down configuration in an easier fashion.
None of this should prevent devices from implementing their own special features, and where they do, Lucid is extensible which allows the Field Device to suggest to a Supervisory Application how it might show the settings for that feature in a dynamically created UI. This allows Supervisory Applications to adapt and to allow configuration of other features which were not necessarily part of the original standard.
Summary
In engineering, one does not often get the chance to recreate something that has already been made; instead, one normally has to live with the consequences of those decisions be they good or bad. Lucid gave us the chance to revisit some of the decisions we made during development of WITS-DNP3 and to apply learning from the implementation of that standardDrawing from above, some of the good things we have put into the Lucid standard are:
- Lucid is underpinned with public and available standards like MQTT and JSON.
- Implementations of those underpinning standards are freely available.
- Lower complexity should drive down development costs.
- Staff availability with skills in MQTT and JSON should be higher than for DNP3.
- A smaller codebase than WITS-DNP3
- Bandwidth usage can be kept to a minimum.
- Lucid is a public protocol standard freely accessible to all.
- There is a plan to create a common community-developed test harness.
- Entire device configurations arecaptured in a JSON description of the device ensuring better interporation between products.
Mark Davison (Terzo Digital) and Dave Howarth (NWL)
January 2024
Please see our Lucid reading page for a collated list of other articles and information on Lucid.