Achieving a 155× speedup in a LiDAR data processing pipeline

Real-world data processing pipelines often grow incrementally over years and accumulate inefficiencies that no single processing step explains. We present a measurement-driven case study of optimizing one such system end-to-end: a multi-stage pipeline for high-resolution LiDAR road condition assessment. The original system required weeks of computation for a typical measurement campaign covering a 10 km road network and approximately 200GB of raw data. Our primary contribution is a continuous, measurement-driven optimization methodology. Automated per-commit benchmarks on fixed reference datasets track per-stage runtime throughout the project. This revealed that the dominant bottleneck shifted repeatedly as optimization proceeded. It allowed us to target each successive bottleneck and detect regressions early. Guided by this methodology, we consolidated a distributed eight-server architecture onto a single high-performance node, eliminated intermediate file I/O through in-memory stream processing, migrated heterogeneous runtimes (MATLAB, Python, C++) to a single managed C# codebase, and applied pervasive SIMD vectorization and parallelization. We achieved a 155× end-to-end speedup, enabling the pipeline to process data faster than it is recorded. We decompose this speedup per stage and estimate that software and architectural changes, rather than hardware alone, account for at least an order of magnitude of it. We also observe that carefully vectorized managed code matched or exceeded the original native code implementations for our workload.We do not claim novel algorithms, but contribute a reproducible methodology and quantitative evidence that disciplined engineering can yield order-of-magnitude improvements in legacy data processing systems.

Schlagwörter

Fachgebiet (DDC)

005 - Computer Programmierung, Programme und Daten

Projekt

Veranstaltung

Startdatum der Ausstellung

Enddatum der Ausstellung

Startdatum der Konferenz

Enddatum der Konferenz

Datum der letzten Prüfung

ISBN

ISSN

2169-3536

Sprache

Englisch

Während FHNW Zugehörigkeit erstellt

Ja

Zukunftsfelder FHNW

Publikationsstatus

Veröffentlicht

Begutachtung

peer-reviewed

Open Access-Status

Gold

Lizenz

Zitation

Felix, S., Stutz, M., Hartmann, J., Bächtiger, Y., Timmel, V., Wassmer, A., Schramka, F., Strobach, E., Leckebusch, J., & Angst, C. (2026). Achieving a 155× speedup in a LiDAR data processing pipeline. IEEE Access, 11. https://doi.org/10.1109/access.2026.3732940

Komplettanzeige