Meshtastic data ingestion

[oscgonfer]

Drop in your thoughts or mesthastic integration!

[oscgonfer]

fablabbcn/smartcitizen-meshtastic#1 and fablabbcn/smartcitizen-meshtastic#3

[oscgonfer]

One important aspect to consider is how we can start filtering devices in the ingestion process for mesh devices.

This requires some thinking to consider:

1. How devices begin being ingested for the first time (onboarding) and identified (mapped)
2. Whether or not we only ingest SC hardware payloads

Onboarding and mapping

A very simple onboarding would be for an user to claim a device, if they know their private key. This could be simply done by logging in and adding the device (or devices) private keys.

A key question to understand is then how the individual mapping happens, and what are the keys that help us understand who is who in a network, since thousands of devices can be used at the same time. In this sense, we would like to have:

1. a single key that all the deployed devices share (for a user), which can be sent to the devices, (private key in meshtastic?)
2. an individual device ID that helps us know who is who, sent in the payload in protobuf or mqtt (payload > id in MQTT json)

Selection of devices (what data gets ingested)

Considering we can open the door to flooding the platform with uncontrolled, non SC hardware, I suggest implementing a mechanism to avoid ingesting data generated by non-sc hardware. Meshtastic allows for hardware identification in their payloads, which could be used as a naive way to filter out devices. However, this can be a "obfuscating" mechanism, because I assume that device ID can be changed with a different firmware. This is an open question.

On the other hand, for non-SC hardware, I'd suggest a mechanism to throttle payloads, only with a certain frequency (maybe every 6h?). This would allow for someone to test our "production" platform, but also open the door for people to deploy their own. As a crazy idea, this could also open the door to monetize hosted deployments if we think that's something interesting (which is something we have never done, but could be an income).

@pral2a @timcowlishaw

[Pixeldieb]

Tried to formulate it and induce some knowledge from LoRaWAN World here:

Ingestion Filtering and Onbaording Strategy for Mesh Devices

Problem Statement

To maintain the integrity and performance of the ingestion pipeline for mesh network devices (e.g., Meshtastic-based nodes), we would need to:

• Prevent ingestion of untrusted or "spoofed" devices (spoofed devices could generate their own ID's)
• Ensure reliable onboarding and mapping of authorized devices (aka making it not too hard to authorize)
• Support limited experimentation from third parties without compromising the platform

Currently, there is no robust filtering or onboarding mechanism, which opens the door for:

• Data flooding by non-approved or malicious hardware
• Device ID spoofing via custom firmware
• Difficulty in associating devices to users or deployments

Refresher on LoRaWAN

The most common and secure way to connect a device is called OTAA (Over-The-Air Activation).

Steps:

1. The device sends a "join request" containing:
   • DevEUI – the unique ID of the device
   • AppEUI – the application or network ID
   • AppKey – a secret key stored in the device

Where do these come from you ask? Simple: Most commonly, DevEUIs are assigned by the hardware manufacturer.
Manufacturers that are members of the IEEE get a block of EUI-64 addresses (similar to how MAC addresses are assigned).
They use these blocks to create unique DevEUIs for each device. These are considered globally unique and traceable.

2. The network checks the request. If the keys match, it sends back session keys the device will use to communicate securely.

3. Only after this handshake, the device is allowed to send data.
   This process ensures that only approved devices can join and send data.

Every device has what i call the "license plates for sensors":

• A unique hardware ID (DevEUI)
• A secret key (AppKey)
• A temporary network address (DevAddr) assigned after joining

These values help the network know which device is sending data.

• Devices that are not registered or have the wrong keys cannot join the network.
• Their messages are ignored -> This prevents random or fake devices from flooding the system.

All messages are encrypted. Each message includes a "Message Integrity Code" (MIC), which proves it wasn’t changed during transmission. LoRaWAN also blocks replay attacks using message counters.

---

Goals

• Design a secure and simple onboarding flow for new devices
• Filter out or throttle unknown/non-authorized devices during ingestion
• Enable device identification at both: group (deployment) and individual levels
• Explore mechanisms to monetize or manage hosted deployments long term

(let me know if i rephrased that right)

---

✅ Definitions of Done

• A documented onboarding process that allows users to register their devices using private keys
• Devices must send a deployment key and unique ID in their payload (MQTT/Protobuf)
• Ingestion system filters out non-registered devices or throttles them (e.g. every 6h for demo)
• Identification method implemented to distinguish SC hardware from unknown devices, maybe supply DeviceEUI similar to LoRaWAN
• Fingerprinting method designed (ID + firmware info + behavior)
• Optional: signed payload or firmware verification implemented
• Monetization concept documented (e.g., paid tiers for full data ingestion)

---

Proposed Solution

🔐 Device Onboarding

Web UI for users to:

• Register device private keys (similar to Chirpstack in LoRaWAN basically)
• Associate devices with a "deployment key" (shared across their fleet)
  Device payload must include:
• deployment_key (shared)
• device_id (unique)
• Optionally: signature for verification

Ingestion Filtering

• Whitelist: Only ingest data from devices with known keys
• Throttle: Unknown devices limited to one payload per 6 hours
• Filter based on:
  • Known hardware identifiers
  • Behavioral fingerprinting (transmission pattern, firmware metadata)

Monetization (Optional, i dont see it as an issue with whitelisting enabled)

• Allow unknown devices limited access for testing
• Offer paid access or hosted plans for:
  • Higher ingestion rates
  • Full integration and dashboard access

[oscgonfer]

More info here: fablabbcn/smartcitizen-meshtastic#3

[oscgonfer]

@timcowlishaw check this script for ingestion: https://github.com/fablabbcn/smartcitizen-meshtastic/blob/feat/mqtt-scripts/scripts/mqtt-read.py

[oscgonfer]

Important note, both ends of the "line" should have up to date protobufs. This means that we should run hooks when we update our protobuf fork (or directly upstream? ⚠ ) so that the platform can ingest them.

[oscgonfer]

An additional ver important aspect to consider: meshtastic is measurement first, whereas smartcitizen is sensor first. This means: when we receive a payload from meshtastic, we get the following decoded protobuf:

decoded {
  portnum: TELEMETRY_APP
  payload: "time: 1751476649 environment_metrics {   temperature: 31.63   relative_humidity: 67.63 }"
}

---

decoded {
  portnum: TELEMETRY_APP
  payload: "time: 1751476649 air_quality_metrics {   co2: 747   co2_temperature: 28.4981308   co2_humidity: 75.2849655 }"
}

This means that we do not know what sensor is behind that co2 or temperature. I am not sure this will be something that could be changed in the payload itself, as it would break protobufs.

However, we could consider having a user interface in the onboarding for users to define what the device is sending. For instance, the user could select something similar to the old kit blueprints, which would effectively make a mapping for that device in the components, or if not available, they could select a list of sensors and associate it to the measurements received.

To be discussed further, but important for next week, as we already have the potential issue with different types of co2 sensors, which would be interesting to consider. @timcowlishaw @pral2a