Labelling multimodal industrial datasets without losing your mind

Labelling multimodal industrial datasets is where many sensing projects quietly lose months. The recording works, the sensors are fine, and then someone has to decide what every second of vibration, sound, video and thermal data actually shows. Done ad hoc, that job grows without end and produces labels nobody fully trusts.

This article sets out a workflow we find practical: design the scheme first, label where it is easiest, propagate carefully, borrow labels from the process, and measure quality instead of assuming it.

Why labelling multimodal industrial datasets is harder than labelling images

Image labelling has mature tools and a simple unit of work: one picture, one answer. Industrial multimodal data breaks that in several ways.

• Most data is unremarkable. A machine runs normally for weeks. Faults are rare, short and often unclear.
• Not every modality is human-readable. An operator can recognise a misplaced part on video. Very few people can label a raw 200 kSPS vibration trace by eye.
• Streams run at different rates and on different clocks. A label set on a video frame must be mapped onto vibration and thermal data recorded at other rates.
• Ground truth is often delayed. Whether a weld was good may only be known after a destructive test or an end-of-line check, hours later.
• The experts are busy. The people who know what a fault looks like are maintenance engineers and process specialists, not full-time annotators.

A workable process has to respect all five constraints.

Design the label scheme before anyone labels

The single most effective step is to agree the label scheme in writing before the first label is set. A good scheme has levels that match how models will be used later, which typically means a ladder from "is it normal?" to "what is it?" to "how bad is it?". Our article on anomaly detection, classification and severity grading describes that ladder from the model side.

Three rules make the scheme robust:

1. Always include "uncertain". Forcing a labeller to choose between "good" and "bad" when they do not know creates confident wrong labels.
2. Separate sensor problems from machine problems. A loose accelerometer is not a bearing fault, and mixing the two teaches models the wrong lesson.
3. Decide event versus segment labels per class. A spatter event has a moment; a worn bearing has a period. Write down which classes are points in time and which are intervals.

Name fault classes in the vocabulary your maintenance and quality teams already use. For condition monitoring, ISO 13379-1:2025 sets out common concepts for diagnostics so that users and manufacturers describe machine states the same way, which makes it a sensible reference for class definitions.

Label in the modality that shows the fault best, then propagate

For each fault class, ask in which stream a human recognises it most easily. A part out of position is obvious on video. A cooling anomaly is obvious on thermal images. An impact may be clearest on the vibration envelope or the spectrogram of the sound. Label there.

Then carry the label to the other streams using time stamps. This is where time-stamping discipline pays off. If each stream's offset to the reference clock and its uncertainty are recorded, the propagated label becomes a window: the labelled moment plus and minus the combined uncertainty, plus any physical delay between modalities, such as heat taking time to reach the surface. Our article on time-stamping multimodal sensor data explains how those offsets are recorded.

Two practical tips:

• Label windows generously, then trim. A label window that is too tight cuts off the start of the event in the slower or later stream. Models can be trained to find the event inside a slightly wider window.
• Show the streams together. A labelling view that displays video, thermal frames, spectrogram and vibration envelope on a common time axis lets labellers confirm events across modalities, which is faster and more reliable than labelling streams one at a time.

Use the process as a labeller

Many industrial labels already exist somewhere; they just are not attached to the sensor data. These are weak labels: labels that are cheap and plentiful but less precise in time or content than a human annotation.

• End-of-line and destructive test results label a part or weld as good or bad, even if they do not say when during the process the problem occurred.
• Maintenance work orders label a period before a repair as "fault developing" and a period after as "repaired".
• Control and MES data label machine states, product types and recipe changes automatically.
• Operator notes and shift logs mark unusual events, often with rough times.

Weak labels are ideal for the state and normality levels and for selecting which segments deserve expert attention. They are less suited to precise fault localisation. A common pattern is to use weak labels to pick a few hundred candidate segments, then have a specialist label those precisely.

Measure label quality instead of assuming it

Labels are measurements, and like any measurement they have errors. Even carefully curated public benchmarks contain them: Northcutt, Athalye and Mueller estimated an average of at least 3.3 % label errors across the test sets of ten commonly used datasets, and at least 6 % of the ImageNet validation set. Industrial data, labelled by busy experts under time pressure, is unlikely to do better.

Practical quality measures:

• Double-label a sample. Have two people label the same 5 to 10 % of segments independently and compare. Agreement statistics such as Cohen's kappa correct for agreement by chance; more useful still is reading the disagreements, which usually reveal unclear class definitions.
• Keep a review queue. Route "uncertain" labels and model-label disagreements to a specialist.
• Track label provenance. Record who set each label, when, in which modality, with which tool version and under which scheme version.
• Re-check after model training. Segments where a trained model strongly disagrees with the label are good candidates for review; some will turn out to be label errors.

Split and document the dataset

The way you split data into training and test sets matters as much as the labels. If segments from the same machine, batch or day appear in both, a model can learn to recognise the machine or the day rather than the fault. Split by machine, batch, specimen or time period, and keep a test set that nobody touches until the end. Our article on validating industrial AI models covers this shortcut-learning risk in detail.

Document the dataset as you build it. The Datasheets for Datasets proposal by Gebru and colleagues suggests recording a dataset's motivation, composition, collection process and recommended uses, modelled on the datasheets that accompany electronic components. For an industrial dataset, add sensor types and positions, mounting, sample rates, time-stamping method, label scheme version and known gaps.

Store labels in a plain, appendable format. JSON Lines, with one JSON value per line, works well for label records: each label is one line, files can be appended and diffed, and any tool can read them.

A labelling workflow that stays manageable

Putting it together, this procedure keeps labelling effort bounded:

1. Agree the label scheme, including "uncertain" and sensor-status labels, and version it.
2. Import process data (tests, work orders, states) as weak labels.
3. Use weak labels and simple anomaly scores to select candidate segments for expert review.
4. Label candidates in the clearest modality, in a view that shows all streams on a common time axis.
5. Propagate labels to other streams with windows based on recorded offset uncertainty.
6. Double-label a sample and resolve disagreements by refining class definitions.
7. Split by machine, batch or time; freeze the test set.
8. Document the dataset and keep label provenance with every record.

How much labelled data you need depends heavily on the task; for one modality, our piece on training data for acoustic quality inspection discusses the trade-offs.

How NeuroControls approaches labelling

Today, our engineers build labelled datasets together with customers and train custom models on them, from feature extraction and anomaly detection to fault and multi-fault classification and severity grading; the models and validation page describes the approach. Every stream is time-stamped on the hub, and each node's clock offset is written into every record with its uncertainty, which is what label propagation needs.

On our cognitive sensors, signal health and dropouts are flagged at the edge and gaps are noted in the record, so dropouts can be used as sensor-status labels. Recordings export to NWB, MKV/HEVC, JSONL and CSV through our recording software. A self-service framework with labelling, training and deployment screens is coming soon.