Dynamically-scalable distributed cluster and cloud computing framework for IoT media data

Abstract

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT REQUEST OF AUTHOR.] The Internet of Things (IoT) continues to expand; as daily new smart-devices are connected to Internet and adding to a deluge of data created by our society. Compounding the challenges is that this data is very heterogeneous, including device or human activity trace data, structured data, sensor information, and media data. The computing needs for the IoT continue to rise in terms of scalable compute power, storage, and complex data processing pipelines to accommodate these diverse sources of data. For this reason, it becomes essential to develop a flexible framework that is able to efficiently manage the IoT data in a real-time and scalable approach. In this thesis, we propose a novel framework to handle IoT data. Our framework is dynamically extensible, lightweight, resources efficient, and has the ability to handle stream data as well as batch data. We leverage intelligent agents along with the publish-subscribe pattern to achieve a run-time extensible, event-driven, high-performance computational architecture. Additionally, we have incorporated localized and centralized databases into the framework to support structured and unstructured data for compute processing and analytical tasks. We have implemented the proposed framework and evaluated its performance using a visual object-detection case study on both a local cluster and within cloud-computing infrastructure. Our analysis shows that this framework utilizes the CPU, memory, and network resources efficiently. Additionally, the framework can scale horizontally as adding more processing nodes reduces the time and increases the goodput. Moreover, we can measure the data velocity and active state of compute agents to employ proactive system scaling for streaming-data fluctuations instead of reactive measures based on CPU utilization at the host-level.