A synchronized process based scheduling to improve map/reduce execution strategy

Abstract

MapReduce is a widely used parallel computing framework for large scale data processing. The two major performance metrics in MapReduce are job execution time and cluster throughput. They can be seriously impacted by straggler machines. Speculative execution is a common approach for dealing with the straggler problem by simply backing up those slow running tasks on alternative machines. To improve speculative execution strategies MCP (Maximum Cost Performance) can identify slow task and EWMA (Exponentially Weighted Moving Average) to predict process speed and calculate task completion time. Multiple speculative execution strategies have been proposed, but there is a pitfall: incoming jobs are allocated to nodes present in server and fail to schedule process type allocate to node for processing. To overcome this process we proposed a new scheduling based speculative execution strategy. For scheduling we first calculate number of name node residing in server, minimum threshold of resources allocated to name node. We use minimum to avoid huge interaction among the name node when the competition for resources arises. To choose a proper work node for computing the task, we take both time scheduling and ability of work node to compute the task.