Parallel dynamic data-driven model for concept drift detection and prediction

S-Y. Lin, Y-C. Chiu, Jacek Lewandowski, Kuo-Ming Chao

Research output: Research - peer-reviewArticle

Abstract

The traditional data analysis and prediction method assumes that data distribution is normal and will not change. Therefore, it can predict unlabeled data by analyzing the static and historical data. However, in today’s big-data environment, which is changing frequently, the traditional approaches can no longer be effective, as they cannot handle concept drift problems in a Dynamic Data Driven Application System (DDDAS). This study proposes a parallel detection and prediction method for concept drift problems in DDDAS. The proposed method can detect dynamic and changing data, and then feedback to the prediction model to revise for better subsequent predictions. Furthermore, this method computes a global prediction result by aggregating local predictions in the resource bounded environment. Therefore, the prediction accuracy increases, and the computation time decreases. In the simulation, the Map-Reduce technology is used for parallel processing. The simulation results show that the prediction accuracy is raised by 14%, and the execution time is improved by almost 45%.
LanguageEnglish
Pages1413-1426
JournalJournal of Intelligent & Fuzzy Systems
Volume32
Issue number2
Early online date23 Dec 2016
DOIs
StatePublished - 30 Jan 2017

Fingerprint

Data-driven
Prediction
Model
MapReduce
Historical Data
Parallel Processing
Normal distribution
Execution Time
Simulation
Predict
Decrease
Resources
Feedback

Bibliographical note

The final publication is available at IOS Press through http://dx.doi.org/10.3233/JIFS-169138

Keywords

  • Dynamic data-driven application system
  • concept drift
  • Map-Reduce

Cite this

Parallel dynamic data-driven model for concept drift detection and prediction. / Lin, S-Y.; Chiu, Y-C.; Lewandowski, Jacek; Chao, Kuo-Ming.

In: Journal of Intelligent & Fuzzy Systems, Vol. 32, No. 2, 30.01.2017, p. 1413-1426.

Research output: Research - peer-reviewArticle

@article{05f8c2a17b8b469c8b88e848d2e56e56,
title = "Parallel dynamic data-driven model for concept drift detection and prediction",
abstract = "The traditional data analysis and prediction method assumes that data distribution is normal and will not change. Therefore, it can predict unlabeled data by analyzing the static and historical data. However, in today’s big-data environment, which is changing frequently, the traditional approaches can no longer be effective, as they cannot handle concept drift problems in a Dynamic Data Driven Application System (DDDAS). This study proposes a parallel detection and prediction method for concept drift problems in DDDAS. The proposed method can detect dynamic and changing data, and then feedback to the prediction model to revise for better subsequent predictions. Furthermore, this method computes a global prediction result by aggregating local predictions in the resource bounded environment. Therefore, the prediction accuracy increases, and the computation time decreases. In the simulation, the Map-Reduce technology is used for parallel processing. The simulation results show that the prediction accuracy is raised by 14%, and the execution time is improved by almost 45%.",
keywords = "Dynamic data-driven application system, concept drift, Map-Reduce",
author = "S-Y. Lin and Y-C. Chiu and Jacek Lewandowski and Kuo-Ming Chao",
note = "The final publication is available at IOS Press through http://dx.doi.org/10.3233/JIFS-169138",
year = "2017",
month = "1",
doi = "10.3233/JIFS-169138",
volume = "32",
pages = "1413--1426",
journal = "Journal of Intelligent & Fuzzy Systems",
issn = "1064-1246",
publisher = "IOS Press",
number = "2",

}

TY - JOUR

T1 - Parallel dynamic data-driven model for concept drift detection and prediction

AU - Lin,S-Y.

AU - Chiu,Y-C.

AU - Lewandowski,Jacek

AU - Chao,Kuo-Ming

N1 - The final publication is available at IOS Press through http://dx.doi.org/10.3233/JIFS-169138

PY - 2017/1/30

Y1 - 2017/1/30

N2 - The traditional data analysis and prediction method assumes that data distribution is normal and will not change. Therefore, it can predict unlabeled data by analyzing the static and historical data. However, in today’s big-data environment, which is changing frequently, the traditional approaches can no longer be effective, as they cannot handle concept drift problems in a Dynamic Data Driven Application System (DDDAS). This study proposes a parallel detection and prediction method for concept drift problems in DDDAS. The proposed method can detect dynamic and changing data, and then feedback to the prediction model to revise for better subsequent predictions. Furthermore, this method computes a global prediction result by aggregating local predictions in the resource bounded environment. Therefore, the prediction accuracy increases, and the computation time decreases. In the simulation, the Map-Reduce technology is used for parallel processing. The simulation results show that the prediction accuracy is raised by 14%, and the execution time is improved by almost 45%.

AB - The traditional data analysis and prediction method assumes that data distribution is normal and will not change. Therefore, it can predict unlabeled data by analyzing the static and historical data. However, in today’s big-data environment, which is changing frequently, the traditional approaches can no longer be effective, as they cannot handle concept drift problems in a Dynamic Data Driven Application System (DDDAS). This study proposes a parallel detection and prediction method for concept drift problems in DDDAS. The proposed method can detect dynamic and changing data, and then feedback to the prediction model to revise for better subsequent predictions. Furthermore, this method computes a global prediction result by aggregating local predictions in the resource bounded environment. Therefore, the prediction accuracy increases, and the computation time decreases. In the simulation, the Map-Reduce technology is used for parallel processing. The simulation results show that the prediction accuracy is raised by 14%, and the execution time is improved by almost 45%.

KW - Dynamic data-driven application system

KW - concept drift

KW - Map-Reduce

U2 - 10.3233/JIFS-169138

DO - 10.3233/JIFS-169138

M3 - Article

VL - 32

SP - 1413

EP - 1426

JO - Journal of Intelligent & Fuzzy Systems

T2 - Journal of Intelligent & Fuzzy Systems

JF - Journal of Intelligent & Fuzzy Systems

SN - 1064-1246

IS - 2

ER -