Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Introduction to the Big Data ecosystem and Spark's role in contemporary data platforms
• Exploring Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
• Establishing and configuring SparkSession, with a focus on fundamental application configuration

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data management
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images, Kubernetes pods, and Spark driver/executor processes—providing a conceptual view for cluster users rather than administrators
• Observing the execution of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Differentiating between controls via Spark session parameters and those managed by the platform team through cluster provisioning, and why this distinction is crucial for tuning
• Practical Exercise: Examine an active Spark-on-Kubernetes job within the sandbox cluster to identify driver and executor pods

Module 4: Advanced Spark Session Configuration on Kubernetes

• Analyzing the trade-offs between executor count and executor size, including memory versus core allocations
• Managing driver and executor memory, including overhead, and translating these into pod resource specifications
• Dynamic allocation: examining its behavior on Kubernetes, identifying scenarios where it conserves resources versus where it does not
• Practical Exercise: Execute identical jobs with varying executor and core configurations to compare runtime and resource usage

Module 5: Scaling Dynamics & Cost Optimization

• Assessing how adding or removing nodes impacts job behavior, completion time, and resource consumption
• Evaluating the performance and cost trade-offs between using many small executors versus fewer large ones
• Optimizing shuffle behavior and partition tuning, including estimating the cost implications of configuration choices
• Practical Exercise: Expand the sandbox cluster from five to ten nodes and analyze the impact on job completion time and resource consumption

Module 6: Optimized Data Ingestion & Partitioning

• Addressing the small-files problem: understanding why sources split into numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during both read and write operations
• Writing Parquet files efficiently to prevent recurring small-file issues downstream
• Practical Exercise: Load a dataset consisting of numerous small Parquet partitions, apply varied repartitioning strategies, and benchmark performance before and after optimization

Module 7: Pandas Memory Control & Failure Analysis

• Identifying root causes and symptoms of out-of-memory errors in Pandas
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Utilizing chunked processing and dtype optimization for resource-constrained environments
• Practical Exercise: Recreate a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Solution

• Comparing Polars with Pandas: analyzing performance characteristics, lazy evaluation, and memory behavior
• Defining the role of Polars alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Practical Exercise: Refactor a Pandas-intensive transformation into Polars to compare memory usage and processing speed

Module 9: Implementing Optimization in ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles to a realistic ETL pipeline
• Addressing optimization considerations specific to machine-learning workloads running on the same cluster
• Employing a systematic tuning workflow to diagnose cost and performance issues
• Practical Exercise: Complete an end-to-end mini-project involving data loading, transformation, and a simple model-training step, with participants independently tuning the Spark configuration