Distributed Machine Learning Optimization for Large-Scale Cloud Resource Scheduling: A Hybrid Deep Learning and Evolutionary Algorithm Approach

Abstract

Cloud computing resource management represents one of the most critical challenges in modern distributed systems, where efficient allocation of computational resources directly impacts system performance, energy consumption, and operational costs. This paper presents a novel hybrid approach combining deep reinforcement learning (DRL) with genetic algorithms (GA) for optimizing cloud resource scheduling in large-scale distributed environments. The proposed framework, termed Distributed Adaptive Learning Resource Scheduler (DALRS), integrates convolutional neural networks (CNN) for workload prediction with deep Q-networks (DQN) for dynamic resource allocation decisions. We evaluate our approach on a comprehensive simulation platform modeling realistic cloud infrastructure with heterogeneous resource configurations. Experimental results demonstrate that DALRS achieves 34.7% improvement in resource utilization, 28.3% reduction in task completion time, and 31.5% decrease in energy consumption compared to state-of-the-art baselines. Furthermore, the hybrid genetic algorithm component provides Pareto-optimal solutions balancing multiple objectives including cost, latency, and throughput. The paper also addresses scalability challenges through a distributed implementation using Apache Spark, enabling efficient processing of workloads exceeding 10,000 concurrent tasks. Our results validate the effectiveness of combining machine learning with evolutionary optimization for managing complex resource allocation problems in cloud infrastructure.