The vast proliferation of monitoring and sensing devices equipped with Internet connectivity, commonly known as the “Internet of Things” (IoT) generates unprecedented volumes of data, which requires Big Data Analytics Systems (BDAS) for processing and extracting actionable insights. The diversity of IoT data processing applications require the deployment of multiple processing frameworks under the coordination of a resource allocator. To enable prompt actuation, these applications must meet deadlines and their processing takes place where data is generated, in private clouds or edge computing clusters, which have limited resources. It is an open research question as to whether current BDAS systems are suitable for such applications and settings.
With this dissertation, we investigate this research question in multiple ways. First, we evaluate the performance and behavior of BDAS in resource-constrained multi-analytics clusters to understand how they interact. Second, using this understanding, we investigate new admission control and resource allocation mechanisms that are better suited to the resource constraints of the next generation in IoT analytics deployments. Finally, we develop new techniques that enable these resource allocators
to adapt to changing cluster conditions to satisfy deadlines and preserve fairness in multi-analytics settings. In this talk, we overview our research contributions and evaluate their efficacy. We compare our resource allocators against existing allocators using trace based simulation and production, Big Data workloads. We show that our work improves upon the state of the art in terms of deadlines satisfied, productive work completed, fairness, and system utilization.