— Certifying Devices, Stacks & Systems to the Last Bit
(Module 9 · Advanced & Specialised Topics)
Chapter focus
Modbus owes much of its longevity to interoperability. The only way to guarantee interoperability is to prove that devices, firmware, and software stacks follow the specification under every legal and many illegal (negative-test) conditions.
This chapter is a laboratory manual + governance handbook rolled into one. We cover:
| § | Topic | What you will gain |
|---|---|---|
| 29.1 | Why conformance matters, layers of “compliance” | Business, legal, and cyber implications |
| 29.2 | Official Modbus Org test program — scope, process, fees | Step-by-step certification timeline |
| 29.3 | De-facto open-source test suites & fuzzers | Toolchain for nightly CI |
| 29.4 | Building an automated “Red-Black-Grey” test rig | Hardware, wiring, clock sync, safety relays |
| 29.5 | Creating a test-case taxonomy: Positive, Negative, Performance, Security | Spreadsheet blue-print + YAML example |
| 29.6 | Regression CI/CD pipelines — GitHub Actions, Jenkins, GitLab | Sample workflow files |
| 29.7 | Field acceptance & FAT/SAT checklists | Printable score-cards |
| 29.8 | Handling non-conformance: triage, firmware patch, delta certification | Governance model |
| 29.9 | Best-practice cheat sheet & template repositories | Ready-to-fork resources |
(Visual placeholders Fig-29-x, code Listing 29-y.)
29.1 “Compliance” versus “Conformance”
| Term (ISO/IEC) | Meaning in Modbus context | Example |
|---|---|---|
| Compliance | Device implements required features; may not be formally tested | Vendor datasheet claims “Modbus/TCP compliant” |
| Conformance | Device has passed a defined test plan under accredited lab or witnessed self-test | Certificate #MTCP-23-104, posted on Modbus-org |
Guideline: only trust conformance for risk-critical applications; treat mere compliance claims as marketing until proven.
29.2 Official Modbus Org Conformance Test Program
29.2.1 Program outline
| Phase | Actor | Deliverable | Time |
|---|---|---|---|
| 1 Registration | Vendor → Modbus Org | Test ID, quote, NDA | 1 week |
| 2 Self-pretest | Vendor | Test log, checklist | Variable |
| 3 Witnessed test (remote or lab) | Cert Lab | Signed test report | 1 day |
| 4 Issue certificate | Modbus Org | PDF + public DB entry | 1 week |
Fees (2025): $3 000 per profile (RTU slave, TCP server, etc.); 25 % discount for members.
29.2.2 Test harness
- Host PC: Windows 10, official Modbus Conformance Test Tool v3.5
- Physical interfaces:
- RS-232 loopback dongle
- Isolated RS-485 adapter (FTDI FT232)
- NIC adapter, no firewall
- Custom “Break-box” to insert noise, line reversal, open-wire.
[Fig-29-1] schematic of lab.
29.2.3 Coverage matrix (extract)
| Section | Purpose | # Tests |
|---|---|---|
| PDU syntax (FC01–FC24) | Check legal frame decode | 128 |
| Exception generation | Verify correct code per error | 42 |
| Timing RTU | T1.5/3.5 compliance @ all baud rates | 18 |
| TCP session | Reuse, RST handling, out-of-order | 15 |
| Security profile (MBSec) | TLS 1.3 cipher, chain, revocation | 26 |
| Stress | 1 000 reg burst ×10 000 loops | 8 |
Total ≈ 237 mandatory points.
29.3 Open-source & community test suites
| Suite | URL | Coverage highlight | Licence |
|---|---|---|---|
modbus-test-tool | github.com/mbtools/modbus-test-tool | YAML-driven cases, auto-CRC | GPL-3 |
modbus-fuzz | gitlab.com/icsfuzz/modbus-fuzz | Mutation fuzzer, AFL++ integration | MIT |
pymodbus selftest | Pymodbus repo /test | Client/server loopback, asyncio | BSD |
| Zeek ICS scripts | zeek.org/ics-modbus | Conformance & security alerts | BSD |
(Table 29-A.)
Combine official + open-source: the official harness is authoritative; the community suites catch regressions nightly.
29.4 The Red-Black-Grey Test Rig
| Channel | Colour code | Description |
|---|---|---|
| Black | Nominal frames (positive tests) | |
| Red | Malformed, oversize, illegal FC (negative & security) | |
| Grey | Stress/performance, e.g., partial-line noise, 1 000 req/s |
29.4.1 Hardware BOM
| Qty | Part | Cost |
|---|---|---|
| 1 | Fanless i5 NUC (Ubuntu LTS) | $550 |
| 1 | National Instruments USB-8451 for precise timing | $350 |
| 1 | RS-485 fault-injector board (open-source design) | $120 |
| 1 | 8-relay interlock crate | $80 |
| 1 | Saleae Logic 8 12-bit analyzer | $475 |
Total ≈ $1 600. Pays itself after first QA cycle vs vendor lab fees.
29.4.2 Clock synchronisation
ptp4l → < ±1 µs between Saleae and NUC; enables timing assertions (gap, latency) inside Python test.
(Listing 29-1: docker-compose.yml spinning test harness containers.)
29.5 Test-case taxonomy & YAML spec
29.5.1 YAML schema (excerpt)
- id: FC03_valid_10regs
type: positive
request: {addr:17, fc:3, start:0, qty:10}
expect:
pdu: [0x11,0x03,0x14] # bytecount
data_length: 20
timeout_ms: 100
- id: FC09_illegal_function
type: negative
request: {addr:17, fc:9, start:0, qty:1}
expect:
exception: 1 # Illegal Function
Tool modtest-runner consumes YAML, exports JUnit XML for CI.
29.5.2 Coverage goals
- Every public FC must have ≥1 positive + ≥1 boundary + ≥1 illegal test.
- Address offset 0, max, max+1.
- Timing: shortest, nominal, >T3.5 gap.
- Security: TLS cipher iterate, expired cert, wrong CA, CRL revoked.
29.6 CI/CD integration examples
29.6.1 GitHub Actions
jobs:
modbus-ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: docker compose up -d # bring DUT + test harness
- run: pytest tests/ --junitxml=report.xml
- uses: actions/upload-artifact@v4
with: {name: modbus-report, path: report.xml}
DUT firmware image flashed into containerised QEMU (for ARM cross-target).
29.6.2 Fail criteria
- Any unexpected exception.
- Frame response time > spec ×1.25.
- Memory leak >2 KB over 10 000 cycles (valgrind leak check).
- CPU >85 % for >2 s under stress.
29.7 Factory Acceptance (FAT) & Site Acceptance (SAT) templates
Checklist sections
- Physical wiring & termination measured (<1 Ω).
- Official conformance cert verified.
- Custom map diff matches design rev.
- Stress test: 72-h soak, 100 % bus utilisation, exception rate <0.05 %.
- Fail-over: disconnect master 1, master 2 continues within <1 s.
- Security: TLS mutual-auth handshake success, DPI rejects FC 16 from rogue IP.
[Fig-29-2] sample FAT stamp card (QR code links to digital pipeline logs).
29.8 Handling non-conformance
| Discovery | Severity | Response |
|---|---|---|
| Wrong exception code | Medium | Patch firmware; retest subset |
| Off-by-one register mapping | High (data integrity) | Block release, hot-fix |
| CRL not checked | Critical (cyber) | P1 incident; disable TLS until patch |
| Memory leak in stress | Medium | Patch, add valgrind to CI |
Governance model: Non-Conformance Board (NCB) meeting weekly; Jira auto-tickets from CI. Firmware re-cert after major fix.
29.9 Best-practice cheat sheet
| ✔︎ | Rule |
|---|---|
| ☐ | Align device firmware version with conformance cert ID. |
| ☐ | Integrate open-source tests into nightly pipeline; fail build on new regressions only (use baseline). |
| ☐ | Automate PDF cert upload to asset DB; SCADA shows link in “device details”. |
| ☐ | Keep raw PCAP & Saleae capture per cert; attach to release tag. |
| ☐ | For RTU, include gap-timing tests at every supported baud. |
| ☐ | Security profile: validate TLS cipher suite & CRL on every firmware push. |
| ☐ | On site, repeat mini-test after wiring; do not assume lab certificate survives electrician errors. |
Chapter recap
Conformance testing is not optional QA polish; it is the backbone that lets thousands of Modbus devices from hundreds of vendors cooperate for decades. By embedding automated, layered test-suites into CI, FAT, and SAT, you ensure that performance optimisations (Chapter 28) and security hardening (Chapters 22–23) never drift out of spec—even at 3 AM on a holiday shutdown.
Assets to publish
| ID | Visual / File |
|---|---|
| Fig-29-1 | Official Modbus conformance lab layout |
| Fig-29-2 | FAT/SAT stamp card |
| Table 29-A | Open-source suite comparison |
| Listing 29-1 | Docker-compose test harness |
| Repo template | modbus-ci-skeleton GitHub repo |
Next (final Chapter 30): Comparing Modbus with Other Industrial Protocols—a systematic, metrics-driven decision guide to decide when to stick with Modbus, when to gateway, and when to leap to a next-generation protocol.