Developer Guide

Everything needed to build, run, test and change Rusty SunSpec Collector.

For release builds, Docker images and cross-compiling see Building & Packaging. For configuration options see Configuration, for the system design see Architecture, and for deployment see Deployment.

Prerequisites

Required

SQLite (through sqlx) and zstd are bundled and need no system packages.

Optional

Debian / Ubuntu

sudo apt-get update
sudo apt-get install -y build-essential cmake pkg-config libssl-dev libcurl4-openssl-dev

macOS

xcode-select --install          # C/C++ toolchain and SDK (includes the curl headers)
brew install cmake openssl@3

The OpenSSL build scripts find Homebrew's OpenSSL on their own; the resulting debug and release binaries link it dynamically from Homebrew.

Repository layout

rusty-sunspec-collector/
  Cargo.toml              Workspace root (members, MSRV 1.88, shared dependencies)
  Cargo.lock              Committed; CI and Docker builds use --locked
  CHANGELOG.md            Keep a Changelog format
  CODE_OF_CONDUCT.md      Contributor Covenant 2.1
  SECURITY.md             How to report vulnerabilities privately
  Dockerfile              Multi-target build: builder, runtime, simulator, collector
  docker-compose.yml      Kafka + simulator + collector + Prometheus
  Cross.toml              Maps the release targets to the custom cross images
  rustfmt.toml            Formatting rules (max width 100)
  .github/
    workflows/ci.yml          fmt, clippy, build, test, cargo audit, Docker E2E
    workflows/release.yml     Release archives and images (dry run unless a v* tag)
    workflows/deploy-docs.yml mdBook -> GitHub Pages
    workflows/scorecard.yml   OpenSSF Scorecard (README badge, Security tab)
    workflows/codeql.yml      CodeQL static analysis of the Rust code and the workflows
    workflows/fuzz.yml        ClusterFuzzLite: runs the fuzz targets
    dependabot.yml            Weekly Cargo, GitHub Actions and Docker image updates
    ISSUE_TEMPLATE/           Issue forms: bug report, feature request, device report
    pull_request_template.md  PR checklist
  crates/
    types/                Shared DTOs (serde only)
    discovery/            CIDR subnet scan
    modbus-client/        Modbus TCP client
      tests/sim_tests.rs
    sunspec-parser/       SunSpec model definitions and decoder
      models/             Official SunSpec JSON definitions (Apache-2.0)
      build.rs            Embeds every models/model_<id>.json
      src/registry.rs     Model registry and the definition types
      src/decode.rs       decode_model()
      tests/decode_tests.rs
    poller-actor/         Per-device supervisor, model-chain walk, polling
      tests/device_tests.rs
    buffer/               SQLite queue
      tests/buffer_tests.rs
    avro-kafka/           Record schema, single-object codec, Kafka publisher
      schemas/sunspec_sample.avsc
      tests/              codec, producer config and (ignored) Kafka integration tests
    collector-app/        The collector binary and its library
      src/main.rs         CLI, logging, signals
      src/app.rs          run(), run_with_publisher(), shutdown order
      src/config.rs       Config loading and validation
      src/devices.rs      Device set and per-device tasks
      src/ingest.rs       Channel -> SQLite batches
      src/uplink.rs       SQLite -> Kafka
      src/retention.rs    Row and age limits, buffer gauges
      src/health.rs       Heartbeats and /healthz report
      src/metrics_server.rs  Prometheus recorder, /metrics, /healthz
      src/systemd.rs      READY and watchdog notifications (Linux)
      tests/              End-to-end, CLI, config, metrics and signal tests
    sunspec-sim/          SunSpec Modbus TCP simulator (library + binary)
  fuzz/                   cargo-fuzz targets (own workspace, nightly); seeds.py writes seed corpora
  .clusterfuzzlite/       Build image and script for running the fuzz targets in CI
  docker/
    Dockerfile.cross      Cross-compilation image for the release targets
    prometheus.yml        Scrape config for the compose stack
  scripts/                Release packaging, install.sh, CHANGELOG section extraction
  fixtures/devices/       collector-app --probe dumps replayed by the fixture test
  docs/                   This documentation (mdBook), example config, systemd unit

Building

cargo build --workspace                  # debug build of every crate
cargo build --release -p collector-app   # optimised collector binary

Binaries land in target/debug/ or target/release/ (collector-app, sunspec-sim). The first build compiles librdkafka from source and takes noticeably longer than later ones. See Building & Packaging for release builds, Docker images and cross-compiling.

Running locally with the simulator

No hardware or Kafka is needed. Start two simulated inverters behind one address:

# Terminal 1
cargo run -p sunspec-sim -- --bind 127.0.0.1:1502 --units 1,2

Then point the collector at them:

# Terminal 2
SUNSPEC_STATIC_DEVICES=127.0.0.1:1,127.0.0.1:2 SUNSPEC_PORT=1502 cargo run -p collector-app

Without SUNSPEC_KAFKA_BROKERS the collector encodes each record and only logs it, at debug level, then removes it from the buffer. To see one log line per record:

SUNSPEC_STATIC_DEVICES=127.0.0.1:1,127.0.0.1:2 SUNSPEC_PORT=1502 \
  RUST_LOG=info,avro_kafka=debug cargo run -p collector-app

RUST_LOG=info,collector_app=debug,poller_actor=debug shows the collector's own detail, such as every poll cycle and uplink batch. A bare RUST_LOG=debug also turns on the libraries' debug output (Avro schema parsing, every SQL statement), which buries those lines. The buffer is sunspec-buffer.sqlite in the current directory. Check the collector from a third terminal:

curl localhost:9090/healthz
curl -s localhost:9090/metrics | grep '^sunspec_'

With Kafka

Start only the broker from the compose stack and publish to its host listener:

docker compose up -d kafka
SUNSPEC_STATIC_DEVICES=127.0.0.1:1,127.0.0.1:2 SUNSPEC_PORT=1502 \
  SUNSPEC_KAFKA_BROKERS=localhost:29092 cargo run -p collector-app

# Record keys (values are binary Avro)
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic sunspec.telemetry --from-beginning \
  --property print.key=true --property print.value=false

docker compose up --build runs the whole stack instead, with the collector and the simulator in containers; see Building & Packaging.

Other ways to run

cargo run -p collector-app -- --config docs/config.example.toml   # edit devices and brokers first
cargo run -p collector-app -- --print-schema                      # print the Avro schema and exit

The simulator

crates/sunspec-sim is a SunSpec Modbus TCP device simulator, used by the test suite, the compose stack and local development.

What it serves

• Read holding registers (function 0x03) for one or more unit IDs on one listener. Each unit has its own register map: the SunS marker at the base address, each model's header (ID, length) and a zero-filled body, then the end marker 0xFFFF 0x0000.
• Reads of more than 125 registers (or 0) are rejected with exception 0x03, like a real device.
• Other function codes get exception 0x01; addresses outside the map get exception 0x02 (or zeros, see Unmapped); unknown unit IDs get exception 0x0B (gateway target device failed to respond).

Binary

cargo run -p sunspec-sim -- [--bind ADDR:PORT] [--units 1,2] [--models 1,103,160:48] [--base 40000]
cargo run -p sunspec-sim -- [--bind ADDR:PORT] --fixture fixtures/devices/<device>.json

Invalid arguments print the usage and exit with code 2. The binary also runs a live demo for every unit: it writes nameplate strings into model 1 (Mn, Md, SN, DA), sets the scale factors of model 103, and once per second updates W, A, PhVphA, Hz, St and WH, so dashboards show changing values. Points of models that are not in the chain are skipped.

Library

Tests start a simulator on an ephemeral port and drive it through SimHandle. Add it as a dev-dependency (sunspec-sim = { path = "../sunspec-sim" }).

SimHandle methods (cloning a handle shares the same simulator):

Example:

#![allow(unused)]
fn main() {
use sunspec_sim::{SimConfig, UnitSpec};

#[tokio::test]
async fn reads_scaled_power() {
    let sim = sunspec_sim::start(
        SimConfig::single(UnitSpec::inverter(1)),
        "127.0.0.1:0".parse().unwrap(),
    )
    .await
    .expect("start simulator");
    sim.set_point(1, 103, "W", &[4321]);
    sim.set_point(1, 103, "W_SF", &[(-1i16) as u16]); // W = 432.1
    sim.set_text(1, 1, "Mn", "Example");

    // Point a ModbusClient or a CollectorConfig at 127.0.0.1:sim.port() ...

    sim.set_stalled(true); // requests now time out
    sim.go_offline(); // connections are refused
    sim.go_online(); // and accepted again
}
}

Tests

cargo test --workspace                                # everything except the Kafka tests
cargo test -p sunspec-parser                          # one crate
cargo test -p collector-app --test app_tests          # one test file
cargo test -p poller-actor stalled_session_is_restarted  # one test

The suite has about 60 tests and needs no hardware, network access or Kafka: device behaviour is tested against the simulator on ephemeral ports, and the collector end to end with an in-memory publisher. Tests use short real intervals (100 ms polls), so they take seconds once built.

Integration test files

Unit tests inside the crates

Kafka integration tests

crates/avro-kafka/tests/kafka_integration_tests.rs has two #[ignore]d tests that need a broker:

• publishes_keyed_records_to_kafka publishes two records to SUNSPEC_KAFKA_BROKERS.
• published_records_decode_to_samples consumes the topic and checks that the records published by a running collector decode, carry the content-type header, are keyed by their device, and include a decoded model 103 power value. It expects records from devices on SUNSPEC_E2E_DEVICE_HOST, by default simulator, the compose stack's simulator (keys simulator/1, simulator/2). For a collector polling a simulator on the host, set SUNSPEC_E2E_DEVICE_HOST=127.0.0.1.

Run them against the compose stack, once the collector is healthy and publishing:

docker compose up -d --build
curl -fsS localhost:9090/healthz                       # wait until this returns 200
SUNSPEC_KAFKA_BROKERS=localhost:29092 \
  cargo test -p avro-kafka --test kafka_integration_tests -- --ignored --nocapture
docker compose down -v

The CI Docker E2E job does the same: it builds the simulator and collector images, starts Kafka, the simulator and the collector with docker compose, waits for /healthz to return 200 and for sunspec_uplink_records_total to be non-zero, checks with the console consumer that records are keyed simulator/1 and simulator/2, runs the ignored tests above with SUNSPEC_KAFKA_BROKERS=localhost:29092, and finally stops the collector with SIGTERM and checks that it logged collector stopped and exited with code 0.

Writing tests

• Use the simulator rather than mocks, and bind it to 127.0.0.1:0.
• A test that runs the collector sets metrics_port = 0 so it gets a free port: a taken port fails startup, and tests run in parallel.
• Put SQLite buffers in the system temp directory with a unique name and remove the file and its -wal and -shm files afterwards.
• A test that installs the global metrics recorder must live in its own test file (the recorder can be installed once per process).
• Tests that set environment variables hold the file's ENV_LOCK mutex.
• Tests that need external services are #[ignore = "reason"], with the reason saying what they need.

CI

.github/workflows/ci.yml runs on every push to main and on every pull request, with RUSTFLAGS=-Dwarnings:

"Code" is anything compiled, tested or built into an image: crates/, the Cargo files, rustfmt.toml, Cross.toml, the Dockerfiles and docker/, docker-compose.yml, fixtures/, ci.yml, and the three docs files the tests read (docs/configuration.md, docs/config.example.toml, docs/sunspec-collector.env). A PR that changes only other files (README, other docs, images, issue templates, the release or docs workflows) skips the jobs it does not need. The four jobs stay required for merging: a job skipped by its if: reports as skipped, which GitHub counts as passing. If you add a file that tests or the image build read, add it to the Changes job's list. Pushes to main always run every job.

A new push to a pull request cancels that PR's run in progress. audit.yml runs cargo audit on main every day as well, because advisories are published for dependencies that have not changed.

Other workflows: audit.yml is the daily audit above, release.yml builds and tests the release archives and images, and publishes them on a v* tag (see Building & Packaging), deploy-docs.yml builds this book with mdbook-linkcheck on pull requests that change the docs and publishes it to GitHub Pages on pushes to main that touch docs/, README.md or CHANGELOG.md (mdBook 0.4.40), and scorecard.yml runs OpenSSF Scorecard weekly and on pushes to main, publishing the score behind the README badge and the findings to the repository's Security tab. codeql.yml runs CodeQL on the Rust code and on the workflows (pull requests that change code or workflows, every push to main, weekly), and fuzz.yml runs the fuzz targets (see Fuzzing). Neither is required for merging; CodeQL's check on a pull request lists the alerts it adds, and the Security tab has all of them.

Workflow hygiene, which Scorecard checks:

• Every action is pinned to a full commit SHA, with the release it is in as a comment (uses: actions/checkout@<sha> # v6.1.0); dtolnay/rust-toolchain is pinned to a commit on its master branch and given toolchain: stable. Base images in the Dockerfiles are pinned by digest, with the tag kept for readers, and the release workflow passes each target's pinned cross-rs image to docker/Dockerfile.cross. Dependabot proposes new SHAs and digests weekly.
• Each workflow's top-level permissions is contents: read; a job that needs more (publishing images, releases or Pages) asks for it in its own permissions.

The CI jobs use the latest stable Rust; the Docker images are built with Rust 1.88, so the Docker E2E job is the one that exercises the MSRV. Every crate inherits rust-version from the workspace, and the workspace uses resolver 3, so cargo update and Dependabot prefer dependency versions that support 1.88 (a crate whose newer releases need a newer Rust stays at its last compatible version). To check a lockfile change locally: rustup toolchain install 1.88 and cargo +1.88 check --locked --workspace --all-targets.

Before pushing

cargo fmt --all -- --check
cargo clippy --all-targets --workspace -- -D warnings
RUSTFLAGS=-Dwarnings cargo build --workspace
RUSTFLAGS=-Dwarnings cargo test --workspace
cargo audit

Changing RUSTFLAGS rebuilds every dependency, librdkafka included. Setting CARGO_TARGET_DIR=target/ci for the last two commands keeps that build separate from your normal one.

Fuzzing

See Fuzzing.

Adding a SunSpec model definition

Every model in the pinned commit of github.com/sunspec/models is already bundled. To add a model published later, or to move to a newer commit:

1. Copy json/model_<id>.json into crates/sunspec-parser/models/, unmodified, and update the commit recorded in crates/sunspec-parser/models/README.md.
2. Nothing else needs registering: build.rs embeds every model_<id>.json in that directory. every_model_file_is_bundled and every_bundled_model_decodes_its_fixed_points (crates/sunspec-parser/tests/decode_tests.rs) check that the file parses, decodes, and has unique point names.
3. If the model has repeating groups, count points or float points, add a value test for it. Point types the decoder does not know are published as null.
4. Update the model tables in crates/sunspec-parser/models/README.md and Architecture, regenerate the docs' model pages (UPDATE_MODEL_DOCS=1 cargo test -p sunspec-parser --test models_docs; a new range of model IDs also needs a line in CATEGORIES there and in docs/SUMMARY.md), and add a CHANGELOG entry.

The simulator can then serve the model: --models 1,103,<id>, or <id>:<len> when the model has repeating groups (the bundled length only counts fixed groups).

Adding a crate

cargo new --lib crates/my-new-crate

Make sure it is listed in [workspace] members in the root Cargo.toml, inherit edition, rust-version, license and repository from [workspace.package] (edition.workspace = true, ...), use workspace dependencies where they exist, define a thiserror error type, and make sure cargo clippy and cargo fmt pass.

Coding conventions

Crates and errors

• Library crates return Result<T, CrateError> with an error enum built with thiserror; collector-app uses anyhow::Result with .context().
• types stays lightweight: serde is its only dependency.
• No unwrap() on runtime paths. expect() is used only for invariants such as the bundled schema and model JSON or a poisoned lock.
• After startup, failures are logged, counted and recovered rather than ending the process.
• No #![allow(dead_code)]; fix warnings instead (CI builds with -Dwarnings).

Logging

• Use tracing macros with structured fields, not formatted strings: warn!(device = %device, model_id = model.id, error = %err, "modbus read failed").
• Messages are short, lower case, without a trailing period.
• debug! for anything per read, per cycle or per batch; info! for lifecycle events (startup, models discovered, shutdown); warn! for recoverable failures (a session ended, a sample dropped, records not delivered). Steady-state operation should be quiet at info.

Metrics

• Names start with sunspec_; counters end in _total; durations are histograms in seconds ending in _seconds.
• Register HELP text for every new metric in describe_metrics() (crates/collector-app/src/metrics_server.rs) and add it to the metrics table in Metrics.
• Per-device metrics use the device label with DeviceIdentity::key() (host/unit). Do not add labels with unbounded values (batch sizes, row IDs, error messages).

Formatting

rustfmt.toml: edition 2021, max_width = 100, reorder_imports and reorder_modules. Run cargo fmt --all before committing.

Record schema

Any change to crates/avro-kafka/schemas/sunspec_sample.avsc changes the schema fingerprint, so consumers holding the old schema can no longer decode new records. Treat it as a breaking change: bump the namespace (com.rusty.sunspec.v2 to v3), update the codec and its tests, and describe the change in the CHANGELOG.

Changelog and releases

• CHANGELOG.md follows Keep a Changelog. Every user-visible change gets an entry under ## [Unreleased] in the same pull request, in Added, Changed, Fixed, Removed or Security.
• Changes that need action from users (record format, metric names, config keys and validation) start with Breaking:.
• All crates share one version, version under [workspace.package] in Cargo.toml. Releases are published by .github/workflows/release.yml when a v* tag is pushed; the steps are in Cutting a release.
• Dependabot opens weekly pull requests for Cargo dependencies (deps prefix; patch updates grouped into one PR, minor and major updates one PR each, since a 0.x minor bump is breaking), GitHub Actions (ci prefix; minor and patch releases grouped) and the base image digests of the Dockerfiles and .clusterfuzzlite/Dockerfile (docker prefix, grouped). The Rust builder image stays on the MSRV, so Dependabot updates only its digest.

Documentation

The documentation is an mdBook in docs/ (docs/SUMMARY.md lists the pages). It is organized by what readers come to do: getting-started/, guides/, reference/, contributing/ and project/, plus the larger pages at the top of docs/ (configuration, architecture, this guide, building and packaging, buffer maintenance). Links between pages are relative ([Architecture](ARCHITECTURE.md) from the top, [Metrics](../reference/metrics.md) from a folder); the changelog page includes CHANGELOG.md from the repository root.

docs/reference/models/ is generated from the bundled SunSpec definitions, one page per group of model IDs plus an index. Do not edit it by hand: after changing the bundled models, run UPDATE_MODEL_DOCS=1 cargo test -p sunspec-parser --test models_docs. Without the variable the same test fails when the pages are stale or missing from docs/SUMMARY.md.

cargo install mdbook --version 0.4.40 --locked
cargo install mdbook-linkcheck --version 0.7.7 --locked   # optional: checks links and files
mdbook serve docs --open    # local preview with reload
mdbook build docs           # static site in docs/book (docs/book/html with mdbook-linkcheck)

The Deploy Docs workflow runs the same build, with mdbook-linkcheck, on pull requests that change the docs, and deploys the site from main. A page that moves gets a redirect in docs/book.toml ([output.html.redirect]).

Debugging

Verbose logging

RUST_LOG=info,collector_app=debug,poller_actor=debug cargo run -p collector-app
RUST_LOG=info,poller_actor=debug,modbus_client=debug cargo run -p collector-app   # every read
RUST_LOG=debug cargo run -p collector-app             # everything, libraries included
SUNSPEC_LOG_FORMAT=json cargo run -p collector-app    # one JSON object per line

Common problems