Achieving a 155× speedup in a LiDAR data processing pipeline

Real-world data processing pipelines often grow incrementally over years and accumulate inefficiencies that no single processing step explains. We present a measurement-driven case study of optimizing one such system end-to-end: a multi-stage pipeline for high-resolution LiDAR road condition assessment. The original system required weeks of computation for a typical measurement campaign covering a 10 km road network and approximately 200GB of raw data. Our primary contribution is a continuous, measurement-driven optimization methodology. Automated per-commit benchmarks on fixed reference datasets track per-stage runtime throughout the project. This revealed that the dominant bottleneck shifted repeatedly as optimization proceeded. It allowed us to target each successive bottleneck and detect regressions early. Guided by this methodology, we consolidated a distributed eight-server architecture onto a single high-performance node, eliminated intermediate file I/O through in-memory stream processing, migrated heterogeneous runtimes (MATLAB, Python, C++) to a single managed C# codebase, and applied pervasive SIMD vectorization and parallelization. We achieved a 155× end-to-end speedup, enabling the pipeline to process data faster than it is recorded. We decompose this speedup per stage and estimate that software and architectural changes, rather than hardware alone, account for at least an order of magnitude of it. We also observe that carefully vectorized managed code matched or exceeded the original native code implementations for our workload.We do not claim novel algorithms, but contribute a reproducible methodology and quantitative evidence that disciplined engineering can yield order-of-magnitude improvements in legacy data processing systems.

Keywords

Subject (DDC)

005 - Computer Programmierung, Programme und Daten

Project

Event

Exhibition start date

Exhibition end date

Conference start date

Conference end date

Date of the last check

ISBN

ISSN

2169-3536

Language

English

Created during FHNW affiliation

Yes

Strategic action fields FHNW

Publication status

Published

Review

peer-reviewed

Open access category

Gold

License

Citation

Felix, S., Stutz, M., Hartmann, J., Bächtiger, Y., Timmel, V., Wassmer, A., Schramka, F., Strobach, E., Leckebusch, J., & Angst, C. (2026). Achieving a 155× speedup in a LiDAR data processing pipeline. IEEE Access, 11. https://doi.org/10.1109/access.2026.3732940

Full item page