Enhancing precision in large-scale data analysis: an innovative robust imputation algorithm for managing outliers and missing values

dc.contributor.authorTempl, Matthias
dc.date.accessioned2024-05-28T08:52:46Z
dc.date.available2024-05-28T08:52:46Z
dc.date.issued2023
dc.description.abstractNavigating the intricate world of data analytics, one method has emerged as a key tool in confronting missing data: multiple imputation. Its strength is further fortified by its powerful variant, robust imputation, which enhances the precision and reliability of its results. In the challenging landscape of data analysis, non-robust methods can be swayed by a few extreme outliers, leading to skewed imputations and biased estimates. This can apply to both representative outliers – those true yet unusual values of your population – and non-representative outliers, which are mere measurement errors. Detecting these outliers in large or high-dimensional data sets often becomes as complex as unraveling a Gordian knot. The solution? Turn to robust imputation methods. Robust (imputation) methods effectively manage outliers and exhibit remarkable resistance to their influence, providing a more reliable approach to dealing with missing data. Moreover, these robust methods offer flexibility, accommodating even if the imputation model used is not a perfect fit. They are akin to a well-designed buffer system, absorbing slight deviations without compromising overall stability. In the latest advancement of statistical methodology, a new robust imputation algorithm has been introduced. This innovative solution addresses three significant challenges with robustness. It utilizes robust bootstrapping to manage model uncertainty during the imputation of a random sample; it incorporates robust fitting to reinforce accuracy; and it takes into account imputation uncertainty in a resilient manner. Furthermore, any complex regression or classification model for any variable with missing data can be run through the algorithm. With this new algorithm, we move one step closer to optimizing the accuracy and reliability of handling missing data. Using a realistic data set and a simulation study including a sensitivity analysis, the new alogorithm imputeRobust shows excellent performance compared with other common methods. Effectiveness was demonstrated by measures of precision for the prediction error, the coverage rates, and the mean square errors of the estimators, as well as by visual comparisons.
dc.identifier.doi10.3390/math11122729
dc.identifier.urihttps://irf.fhnw.ch/handle/11654/43478
dc.identifier.urihttps://doi.org/10.26041/fhnw-7443
dc.issue12
dc.language.isoen
dc.publisherMDPI
dc.relation.ispartofMathematics
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.spatialBasel
dc.subject.ddc330 - Wirtschaft
dc.titleEnhancing precision in large-scale data analysis: an innovative robust imputation algorithm for managing outliers and missing values
dc.type01A - Beitrag in wissenschaftlicher Zeitschrift
dc.volume11
dspace.entity.typePublication
fhnw.InventedHereYes
fhnw.ReviewTypeAnonymous ex ante peer review of a complete publication
fhnw.affiliation.hochschuleHochschule für Wirtschaftde_CH
fhnw.affiliation.institutInstitut für Unternehmensführungde_CH
fhnw.openAccessCategoryGold
fhnw.publicationStatePublished
relation.isAuthorOfPublication8b0a85e1-60d7-48f9-8551-419197a127e7
relation.isAuthorOfPublication.latestForDiscovery8b0a85e1-60d7-48f9-8551-419197a127e7
Dateien
Originalbündel
Gerade angezeigt 1 - 1 von 1
Lade...
Vorschaubild
Name:
Templ_2023_Enhancing_precision_in_large-scale_data_analysis.pdf
Größe:
852.19 KB
Format:
Adobe Portable Document Format
Beschreibung:
Lizenzbündel
Gerade angezeigt 1 - 1 von 1
Lade...
Vorschaubild
Name:
license.txt
Größe:
1.36 KB
Format:
Item-specific license agreed upon to submission
Beschreibung: