Analysis of Earliest Deadline First Scheduler in Hadoop Mapreduce Environments

Abstract

The MapReduce applications used for processing petabytes of data across the enterprise. Controlling the allocation of resources in shared MapReduce environments is a key challenge. Many users require job completion time guarantees. There is an increasing number of MapReduce applications associated with live business intelligence that require completion time guarantees (SLOs). There is a lack of performance models and workload analysis tools for automated performance management of such MapReduce jobs. None of the existing Hadoop schedulers support completion time guarantees (SLOs). A key challenge in shared MapReduce clusters is the ability to automatically tailor and control resource allocations to different applications for achieving their performance SLOs. We implemented a novel SLO-based scheduler in Hadoop by making use of ARIA framework that determines job ordering and the amount of resources to allocate for meeting the job deadlines. The new scheduler effectively meets the jobs' SLOs until the job demands exceed the cluster resources. This paper is all about EDF Scheduler Design, EDF Scheduling Algorithm, and Algorithmic Complexity of EDF Scheduling Scheme as well as SWOT analysis of EDF Scheduler.