Risk Intelligence: Making Profit from Uncertainty in Data Processing System
In extreme scale data processing systems, fault tolerance is an essential and indispensable part. Proactive fault tolerance scheme (such as the speculative execution in MapReduce framework) is introduced to dramatically improve the response time of job executions when the failure becomes a norm rath...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2014-01-01
|
| Series: | The Scientific World Journal |
| Online Access: | http://dx.doi.org/10.1155/2014/398235 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832559214778646528 |
|---|---|
| author | Si Zheng Xiangke Liao Xiaodong Liu |
| author_facet | Si Zheng Xiangke Liao Xiaodong Liu |
| author_sort | Si Zheng |
| collection | DOAJ |
| description | In extreme scale data processing systems, fault tolerance is an essential and indispensable part. Proactive fault tolerance scheme (such as the speculative execution in MapReduce framework) is introduced to dramatically improve the response time of job executions when the failure becomes a norm rather than an exception. Efficient proactive fault tolerance schemes require precise knowledge on the task executions, which has been an open challenge for decades. To well address the issue, in this paper we design and implement RiskI, a profile-based prediction algorithm in conjunction with a riskaware task assignment algorithm, to accelerate task executions, taking the uncertainty nature of tasks into account. Our design demonstrates that the nature uncertainty brings not only great challenges, but also new opportunities. With a careful design, we can benefit from such uncertainties. We implement the idea in Hadoop 0.21.0 systems and the experimental results show that, compared with the traditional LATE algorithm, the response time can be improved by 46% with the same system throughput. |
| format | Article |
| id | doaj-art-6c76950926064c3b9b3bdd809377adbe |
| institution | Kabale University |
| issn | 2356-6140 1537-744X |
| language | English |
| publishDate | 2014-01-01 |
| publisher | Wiley |
| record_format | Article |
| series | The Scientific World Journal |
| spelling | doaj-art-6c76950926064c3b9b3bdd809377adbe2025-02-03T01:30:34ZengWileyThe Scientific World Journal2356-61401537-744X2014-01-01201410.1155/2014/398235398235Risk Intelligence: Making Profit from Uncertainty in Data Processing SystemSi Zheng0Xiangke Liao1Xiaodong Liu2School of Computer, National University of Defense Technology, ChinaSchool of Computer, National University of Defense Technology, ChinaSchool of Computer, National University of Defense Technology, ChinaIn extreme scale data processing systems, fault tolerance is an essential and indispensable part. Proactive fault tolerance scheme (such as the speculative execution in MapReduce framework) is introduced to dramatically improve the response time of job executions when the failure becomes a norm rather than an exception. Efficient proactive fault tolerance schemes require precise knowledge on the task executions, which has been an open challenge for decades. To well address the issue, in this paper we design and implement RiskI, a profile-based prediction algorithm in conjunction with a riskaware task assignment algorithm, to accelerate task executions, taking the uncertainty nature of tasks into account. Our design demonstrates that the nature uncertainty brings not only great challenges, but also new opportunities. With a careful design, we can benefit from such uncertainties. We implement the idea in Hadoop 0.21.0 systems and the experimental results show that, compared with the traditional LATE algorithm, the response time can be improved by 46% with the same system throughput.http://dx.doi.org/10.1155/2014/398235 |
| spellingShingle | Si Zheng Xiangke Liao Xiaodong Liu Risk Intelligence: Making Profit from Uncertainty in Data Processing System The Scientific World Journal |
| title | Risk Intelligence: Making Profit from Uncertainty in Data Processing System |
| title_full | Risk Intelligence: Making Profit from Uncertainty in Data Processing System |
| title_fullStr | Risk Intelligence: Making Profit from Uncertainty in Data Processing System |
| title_full_unstemmed | Risk Intelligence: Making Profit from Uncertainty in Data Processing System |
| title_short | Risk Intelligence: Making Profit from Uncertainty in Data Processing System |
| title_sort | risk intelligence making profit from uncertainty in data processing system |
| url | http://dx.doi.org/10.1155/2014/398235 |
| work_keys_str_mv | AT sizheng riskintelligencemakingprofitfromuncertaintyindataprocessingsystem AT xiangkeliao riskintelligencemakingprofitfromuncertaintyindataprocessingsystem AT xiaodongliu riskintelligencemakingprofitfromuncertaintyindataprocessingsystem |