Please use this identifier to cite or link to this item:
https://hdl.handle.net/20.500.11779/1512
Title: | On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers | Authors: | Arslan, Şuayb Şefik Zeydan, Engin |
Keywords: | Estimation Kernel density estimation (KDE) Kernel Reliability Probability density function Measurement Modeling Predictive models hard-disk systems Data analytics Data models Data storage |
Publisher: | IEEE | Source: | Arslan, S. S., & Zeydan, E. (2021). On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers. IEEE Transactions on Reliability, 70(2), 507–524. https://doi.org/10.1109/tr.2020.3007127 | Abstract: | It has become commonplace to observe frequent multiple disk failures in big data centers in which thousands of drives operate simultaneously. Disks are typically protected by replication or erasure coding to guarantee a predetermined reliability. However, in order to optimize data protection, real life disk failure trends need to be modeled appropriately. The classical approach to modeling is to estimate the probability density function of failures using nonparametric estimation techniques such as kernel density estimation (KDE). However, these techniques are suboptimal in the absence of the true underlying density function. Moreover, insufficient data may lead to overfitting. In this article, we propose to use a set of transformations to the collected failure data for almost perfect regression in the transform domain. Then, by inverse transformation, we analytically estimated the failure density through the efficient computation of moment generating functions, and hence, the density functions. Moreover, we developed a visualization platform to extract useful statistical information such as model-based mean time to failure. Our results indicate that for other heavy-tailed data, the complex Gaussian hypergeometric distribution and classical KDE approach can perform best if the overfitting problem can be avoided and the complexity burden is overtaken. On the other hand, we show that the failure distribution exhibits less complex Argus-like distribution after performing the Box–Cox transformation up to appropriate scaling and shifting operations. | URI: | https://hdl.handle.net/20.500.11779/1512 https://doi.org/10.1109/TR.2020.3007127 |
ISSN: | 1558-1721 0018-9529 |
Appears in Collections: | Bilgisayar Mühendisliği Bölümü Koleksiyonu Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection WoS İndeksli Yayınlar Koleksiyonu / WoS Indexed Publications Collection |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
On the Distribution Modeling.pdf | Full Text - Article | 1.43 MB | Adobe PDF | View/Open |
CORE Recommender
SCOPUSTM
Citations
4
checked on Nov 9, 2024
WEB OF SCIENCETM
Citations
3
checked on Nov 9, 2024
Page view(s)
34
checked on Nov 4, 2024
Download(s)
2
checked on Nov 4, 2024
Google ScholarTM
Check
Altmetric
Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.