Science › Instruments2 min read

Twente and Eindhoven teams modify Whittaker–Henderson smoothing, cutting noise over 40-fold in simulated data

Dutch researchers added cross-validated local weights to a century-old smoother. On synthetic Bragg reflector data it reduced noise by more than a factor 40 while keeping fine spectral features.

Twente and Eindhoven teams modify Whittaker–Henderson smoothing, cutting noise over 40-fold in simulated data
Photo: Jean-Pierre Dalbéra, Wikimedia Commons, CC BY 2.0 · License: CC BY 2.0

Researchers from the Technical Universities of Twente and Eindhoven have modified the Whittaker–Henderson smoothing procedure so that it balances noise and spectral features locally, according to a paper in Measurement Science and Technology iop.orgMeasurement Science and Technologyiop.orgOpen source ↗. The method was tested on synthetic noisy data, not on a physical measurement.

The authors set out to resolve features such as spectral peaks (center, width, height) and derivatives from noisy measurements. Most smoothing procedures lose effectiveness when feature widths vary strongly within one dataset.

How the weighted smoother works

Classic Whittaker–Henderson smoothing balances the smoothness of the output against its closeness to the input data, using one smoothing parameter. A value that is too low leaves noise; one that is too high removes the features.

The modification introduces adjustable weights for the data points. They are optimized by cross-validation, so the filter adapts locally to the data iop.orgMeasurement Science and Technologyiop.orgOpen source ↗. Because the measurement errors are used, a straightforward error analysis of the smoothed result is possible, the authors report.

The weights are found with the limited-memory BFGS algorithm. For N = 1000 complex-valued reflectivity data points this takes on average less than two seconds on a typical computer, according to the paper.

Test on a Bragg reflector

To illustrate the method, the authors derived the chirp of an optical pulse (group delay dispersion) for an optical Bragg reflector nanostructure from synthetic phase data with noise. The smoother reconstructs the group delay dispersion and reduces noise by more than a factor 40, which reveals details otherwise buried in noise.

In the comparison described in the Nanowerk summary, unfiltered noisy data obscured all but the strongest features. A Savitzky–Golay filter reduced the noise and made some features visible. A standard Whittaker–Henderson smoother preserved fast-moving features better, and the weighted version with cross-validation gave the strongest noise reduction and the most faithful reproduction of the dispersion.

The authors also present a general framework for further common problems: smoothing unequally sampled data, identifying and quantifying discontinuities including kinks in derivatives, smoothing near the boundaries of the data domain, and multidimensional smoothing iop.orgMeasurement Science and Technologyiop.orgOpen source ↗. The team is now working on making the method more computationally efficient for large data volumes and applications.

Sources · 2

Found a mistake?
  1. ↩

Read also

Up next

The next story opens automatically

More
Follow us

Type at least two letters — results appear instantly

↑↓to choose↵to openescto close