Please use this identifier to cite or link to this item: https://hdl.handle.net/20.500.11779/1512
Title: On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers
Authors: Arslan, Şuayb Şefik
Zeydan, Engin
Keywords: Estimation
Kernel density estimation (KDE)
Kernel
Reliability
Probability density function
Measurement
Modeling
Predictive models
hard-disk systems
Data analytics
Data models
Data storage
Publisher: IEEE
Source: Arslan, S. S., & Zeydan, E. (2021). On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers. IEEE Transactions on Reliability, 70(2), 507–524. https://doi.org/10.1109/tr.2020.3007127
Abstract: It has become commonplace to observe frequent multiple disk failures in big data centers in which thousands of drives operate simultaneously. Disks are typically protected by replication or erasure coding to guarantee a predetermined reliability. However, in order to optimize data protection, real life disk failure trends need to be modeled appropriately. The classical approach to modeling is to estimate the probability density function of failures using nonparametric estimation techniques such as kernel density estimation (KDE). However, these techniques are suboptimal in the absence of the true underlying density function. Moreover, insufficient data may lead to overfitting. In this article, we propose to use a set of transformations to the collected failure data for almost perfect regression in the transform domain. Then, by inverse transformation, we analytically estimated the failure density through the efficient computation of moment generating functions, and hence, the density functions. Moreover, we developed a visualization platform to extract useful statistical information such as model-based mean time to failure. Our results indicate that for other heavy-tailed data, the complex Gaussian hypergeometric distribution and classical KDE approach can perform best if the overfitting problem can be avoided and the complexity burden is overtaken. On the other hand, we show that the failure distribution exhibits less complex Argus-like distribution after performing the Box–Cox transformation up to appropriate scaling and shifting operations.
URI: https://hdl.handle.net/20.500.11779/1512
https://doi.org/10.1109/TR.2020.3007127
ISSN: 1558-1721
0018-9529
Appears in Collections:Bilgisayar Mühendisliği Bölümü Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection
WoS İndeksli Yayınlar Koleksiyonu / WoS Indexed Publications Collection

Files in This Item:
File Description SizeFormat 
On the Distribution Modeling.pdfFull Text - Article1.43 MBAdobe PDFThumbnail
View/Open
Show full item record



CORE Recommender

SCOPUSTM   
Citations

4
checked on Nov 9, 2024

WEB OF SCIENCETM
Citations

3
checked on Nov 9, 2024

Page view(s)

34
checked on Nov 4, 2024

Download(s)

2
checked on Nov 4, 2024

Google ScholarTM

Check




Altmetric


Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.