During 2007, the detail design of the WITS protocol was undertaken by a joint development team including engineers from all contributing founder vendors. This team has seen the WITS protocol move successfully from requirements through to a working protocol implemented, tested and certified on all the vendors equipment by the end of 2010. This article describes a key, and very successful, feature of the WITS development process – the WITS plugfests.

The WITS development process

The development of the ‘production’ protocol implementation for all founder vendors ( CSE-Servelec, ITT Water and Waste Water, Schneider Electric, Serck Controls and Technolog ), and alongside this the generation of the formal WITS ‘Application Notes’ ( specification ) and test specification, proceeded in a number of distinct phases :-

• Detail design, culminating in ‘specification freeze’
• Incremental implementation / plugfest, with specification errors / enhancements defined in ‘Technical Bulletins’
• vendor productisation – a chance to turn prototypes into products
• Formal product testing and finally the enhancement of the Application Notes to include Technical Bulletins
• WITS 1.1 release

Initial Implementation work began at the end of 2007 after the WITS specification – encapsulated within the Application Notes – was frozen. implementation proceeded through 2008, but rather than developing vendor WITS implementations which could be tested as complete ‘devices’, the team formulated an incremental implementation / test process where functionality was added to each vendors implementation in a series of well-defined steps. Each step culminated in a one day ‘plugfest’ where formal tests were applied to every combination of Master station – Field device ‘pair’.

Any problems or shortcomings identified during this phase were identified within Technical Bulletins, which recorded the problem / enhancement along with proposed changes to the WITS Application Notes required to implement the change.

After the plugfest phase, each vendor then moved into the final development phase and ‘productised’ their product – a chance to tie up loose ends, complete configuration GUI’s, integrate more fully into production code etc.

The final phase, done in the latter stages of 2010, involved formal testing of the complete WITS 1.1 protocol between each Master-Field Device pair.At the conclusion of the formal testing the Application Notes were then updated with all included Technical Bulletins, and released in what is now known as WITS 1.1 – and of course, we now have 5 WITS certified ‘devices’ developed by the WITS team; two Master Stations ( CSE-Servelec and Serck Controls ) and three field devices ( ITT Water and Waste Water, Schneider Electric and Technolog ).

The ‘plugfest’ testing approach

The plugfest idea was initially put forward as the name for as a series of proposed monthly ‘workshops’, each focused on a single functional area, intended to keep up the momentum once WITS implementation started.The intent of each workshop being to basically plug each prototype Field Device into each prototype Master station in turn, apply a few relevant test scenario’s, stand back  . . . . . . and watch the sparks fly !

As the idea grew, the WITS implementation plan was formalised and detailed, with a logical implementation sequence debated and agreed between both the vendors involved, and the industry / DNP3 representatives who were bringing governance to the process. Each step building on the previous one to add functionality to each prototype in a controlled and methodical way.

This sequence, and the estimated work required for each step, was reworked and refined such that each step ( and hence the time between plugfests ) was between 4 and 6 weeks.

Each plugfest required that vendors complete the implementation of the agreed elements of the Application Notes, that the team as a whole defined a formal series of tests designed to test the new functionality and ensure compliance with the fledgling Application Notes.

Plugfests in action

Of course not everything goes smoothly, and designs are not infallible ! Fairly early during the implementation process some inconsistencies in the specification were identified and having debated and designed a solution the question is what then ?

The decision taken, and with hindsight a very good one, was that the Application Notes could not be modified once implementation was underway, and that any problems identified would be detailed in Technical Bulletin (TB) – covering problem, solution and actual text changes to the Application Notes. These TB’s were given a formal review / authorisation / implementation life-cycle, and the WITS specification at any time was the current Application Notes + all agreed TB’s.

But back on plugfests, after some debate the WITS protocol was functionally broken down into a series of areas that could be implemented sequentially. Over time these were expanded and resulted in the following series of plugfests beginning in January 2008 and proceeding through early 2009 due to the late verification of the DNP3 security implementation :-

• DNP3 connectivity
• Healthcheck / Callback
• Datasets / Logging
• Consolidation plugfest – a bringing together of the 1st three plugfest functionality
• Alarms
• Configuration- as this is a major area, it was split into three separate plugfests
  • basic configuration
  • Incremental configuration
  • Applications configuration and startup scenario’s
• Security – left until last as we were waiting for the DNP3 security standard, which WITS security is based on, to be finalised and agreed

A room, five piles of equipment . . . and plenty of coffee !

Plugfests were rotated at different vendors premises – all we needed was a room, five piles of equipment . . .and plenty of coffee ! oh, and the odd mains socket or two. Each Plugfest was scheduled to take a single day, and sometimes required working into the evening depending on the complexity of the area being tested. Each vendor brought their prototype WITS ‘device’ and each Field Device was tested in turn with each Master, going through a plugfest test spec usually hastily put together for the occasion. The days were quite intense as the usual array of bugs, errors, mismatches and misunderstandings presented themselves. Working lunches were worked through with plenty of breakouts to discuss general issues as the occasional ambiguity revealed itself and workarounds were needed in order to proceed. All done in a very open, friendly and good natured if intense atmosphere. Plugfests were witnessed by reps from the WITS user community and WITS committee.

Of course the 1st plugfest, where we tested basic DNP3 connectivity, resulted in a few comic moments; blank looks, blank screens and much head shaking – to be expected as the early prototypes staggered into life – but this baby step was the 1st step towards interoperability, a fundamental WITS design goal – and subsequent plugfests found the prototypes in better shape as the code was slowly kicked into shape, and each of us developed a better understanding of the implications and impact of the Application Notes.

Ambiguities or shortcomings were dealt with quickly and efficiently using Technical Bulletins – highlighting the problem or clarifying actual user requirements, developing a solution, providing detailed changes to Application Notes – all done quickly by the team, with each TB passing through a refining process, often resulting in 5 or more versions as the detail of each TB was fleshed out. It became clear in this stage how important it was to challenge each and every sentence, and clarify or correct the detail – each member of the team checking each detail in a constructive way – this was the only way to ensure interoperability and a robust and resilient protocol. Even so, the TB’s continued to be generated through into the later formal testing phase.

In the period between plugfests there were also many instances of ‘ad-hoc’ testing carried out remotely between vendors, minimising problems when plugfests arrived, or proving that post-plugfest fixes had been successful.

Beyond plugfests – on into formal ‘product’ testing

In order to formally verify WITS 1.1 the team decided on a ‘super plugfest’ approach during the latter stages of 2010, where each vendor ‘pair’ would apply a complete test set to their connected products.At the end of the formal testing, which was completed in early November 2010, each of the original two WITS Master stations had been formally tested with all three original WITS field devices, and each field devices proven on both Master Stations. This formal product testing was originally estimated and scheduled for between one and two weeks between each vendor ‘pair’, with different pairs of vendors able to test in parallel. In several cases this testing had to be extended to 3-4 weeks, as even at this stage interoperability problems revealed either minor implementation errors or in some cases ambiguity in the Application Notes or of course errors in the test specification itself. While this was time consuming, the end result is a robust set of Application Notes and a comprehensive test specification that have been applied and tested in a very challenging way, and we can anticipate that future vendors will benefit greatly from this ‘testing’ of the Application Notes and development of a rigorously applied test specification.

A few final thoughts

The WITS ‘road’ has been a challenging one to say the least, but in terms of success; it has delivered the WITS 1.1 protocol in a tested and robust form, along with five fully tested and certified production Master Stations and Field Devices.

However, we can note a couple of other successes along the way – one being the ‘plugfest’ approach taken during the iterative implementation phase, which proved very effective and invaluable, focussing the team on single functional areas through implementation, testing and specification proving / correction. It also provided on-going and very visible evidence of progress in full view of the User community.

Finally, another ‘success’ from the WITS ‘project’ was the way the multi-vendor team worked together particularly during this implementation / test phase – ‘the joy of plugfests’ – and latterly on through finalising WITS and ultimately passing the baton to the PSA to carry it forward.On a personal note it’s been a great pleasure and privilege to work closely with colleagues from ITT Water and Waste Water, Schneider Electric, Serck Controls and Technolog on this industry wide endeavour – everyone involved has been open and constructive, and have brought their varied skills to bare within a sometimes difficult and challenging project. While our companies are in many cases business rivals, this was set aside as the team worked both as professionals and as friends – and this continues through into the interim PSA – a personal thanks to you all, Gavin Rawson, CSE-Servelec Ltd