Keywords
big-data, MapReduce, Hadoop, caching
Abstract
The buzz-word big-data refers to the large-scale distributed data processing applications that operate on exceptionally large amounts of data. Google’s MapReduce and Apache’s Hadoop, its open-source implementation, are the defacto software systems for big-data applications. An observation of the MapReduce framework is that the framework generates a large amount of intermediate data. Such abundant information is thrown away after the tasks finish, because MapReduce is unable to utilize them. In this paper, we propose Dache, a data-aware cache framework for big-data applications. In Dache, tasks submit their intermediate results to the cache manager. A task queries the cache manager before executing the actual computing work. A novel cache description scheme and a cache request and reply protocol are designed. We implement Dache by extending Hadoop. Testbed experiment results demonstrate that Dache significantly improves the completion time of MapReduce jobs.
Publisher
Tsinghua University Press
Recommended Citation
Yaxiong Zhao, Jie Wu, Cong Liu. Dache: A Data Aware Caching for Big-Data Applications Using the MapReduce Framework. Tsinghua Science and Technology 2014, 19(1): 39-50.