Achieving a 155× speedup in a LiDAR data processing pipeline

Abstract

Real-world data processing pipelines often grow incrementally over years and accumulate inefficiencies that no single processing step explains. We present a measurement-driven case study of optimizing one such system end-to-end: a multi-stage pipeline for high-resolution LiDAR road condition assessment. The original system required weeks of computation for a typical measurement campaign covering a 10 km road network and approximately 200GB of raw data. Our primary contribution is a continuous, measurement-driven optimization methodology. Automated per-commit benchmarks on fixed reference datasets track per-stage runtime throughout the project. This revealed that the dominant bottleneck shifted repeatedly as optimization proceeded. It allowed us to target each successive bottleneck and detect regressions early. Guided by this methodology, we consolidated a distributed eight-server architecture onto a single high-performance node, eliminated intermediate file I/O through in-memory stream processing, migrated heterogeneous runtimes (MATLAB, Python, C++) to a single managed C# codebase, and applied pervasive SIMD vectorization and parallelization. We achieved a 155× end-to-end speedup, enabling the pipeline to process data faster than it is recorded. We decompose this speedup per stage and estimate that software and architectural changes, rather than hardware alone, account for at least an order of magnitude of it. We also observe that carefully vectorized managed code matched or exceeded the original native code implementations for our workload.We do not claim novel algorithms, but contribute a reproducible methodology and quantitative evidence that disciplined engineering can yield order-of-magnitude improvements in legacy data processing systems.