eduzhai > Applied Sciences > Engineering >

Big and Stream Data Analytics Using Incremental MapReduce Framework for Smart City

  • Save

... pages left unread,continue reading

Document pages: 5 pages

Abstract: Smart city explores the Internet of Things (IoT) to administrate the voluminous, continuous and variety of data produced by the various sensors and digital devices. Smart services build the analytical models over the batch or stream data which can help in periodic or near real-time decision making. Machine Learning and Deep Learning models can be exploited to extract the hidden knowledge. However, voluminous and stream data demands the huge computation. MapReduce, a distributed processing framework, can effectively and efficiently handle the voluminous data with fault-tolerance. However, it is not effectual for real time analytics as MapReduce phases such as Map, Sort Shuffle and Reduce are dependent on disk I O which incurs the high latency. In order to address these challenges, voluminous and stream data can be processed incrementally by exploiting the MapReduce framework over the cluster of commodity hardware. This paper presents primary reference architecture for smart city having the capability of batch and stream processing. MapReduce processes the batches of data which is accumulated over a time for periodic analysis, and incremental MapReduce processes the recent batches of stream data for real-time analysis. Proposed Incremental MapReduce architecture exploits the in-memory buffers to reduce the latency incurred due to disk I O. It also replaces the sort-merge architecture of MapReduce with hash based architecture which reduces the CPU cycles to support stream data processing.

Please select stars to rate!

         

0 comments Sign in to leave a comment.

    Data loading, please wait...
×