Module 2: Spark and distributed processing · Lesson 2.3 · about 25 min
Making Spark fast and cheap
File formats, file sizes, shuffle partitions, broadcast joins, skew, caching and the Spark UI: the few ideas behind almost every Spark performance fix.
Module 2 · about 25 minutes · read first: 2.1, 2.2. You’ll be able to choose file and table formats for a lake, fix the four common slowdowns (small files, wrong partition counts, bad joins and skew), and read the Spark UI to find the slow stage.
1. Why this matters
In the cloud you pay for machine time. A job that runs 40 minutes on 20 machines costs about the same as 20 minutes on 40 machines, so throwing machines at a slow job rarely saves money. Making a job cheaper means making it do less: read fewer columns and partitions, shuffle less, avoid skew, and shut the cluster down afterwards. A good answer to “how did you speed it up?” names the bottleneck you measured and the cost before and after.
2. Start with storage: formats decide speed
| Format | Shape | Use it for |
|---|---|---|
| CSV / JSON | Rows of text, no types | Landing raw data only |
| Parquet | Columns, compressed, typed | The default for analytics files |
| ORC | Columns, like Parquet | Older Hadoop and Hive estates |
| Avro | Rows, compact, schema attached | Kafka messages and row-by-row writes |
| Iceberg / Delta / Hudi | Table formats over Parquet | Lakehouse tables with updates, deletes, time travel |
Think of a filing cabinet. A CSV is a pile of whole forms: to total one box on every form you read every form. Parquet cuts each form into strips by field and files all the “amount” strips together, so totalling amounts means opening one drawer. Iceberg and Delta add an index card at the front listing exactly which strips belong to the current version, so you can add, fix or roll back forms safely.
3. The four common slowdowns
Slowdown 1: too many small files, or a few giant ones
Spark makes about one task per file chunk. 200,000 files of 50 KB means 200,000 tiny tasks and slow listing; every reader pays that price forever. Aim for files of roughly 128 MB to 1 GB. Fix it with repartition(n) or coalesce(n) before writing, a regular compaction job, or the table format’s own command (Delta’s OPTIMIZE, Iceberg’s rewrite_data_files).
Slowdown 2: the wrong number of shuffle partitions
After a shuffle Spark splits data into spark.sql.shuffle.partitions pieces (200 by default). For 2 TB that is 10 GB per task, which runs out of memory; for 2 GB it is 200 tiny tasks. Adaptive Query Execution (AQE), on by default since Spark 3.2, looks at the real sizes while the job runs, merges small partitions and splits skewed ones. Keep it on and know it exists.
Slowdown 3: joins
- Broadcast hash join: the small table is copied to every executor, and the big table never moves. Spark chooses it automatically when a table is under a size threshold (10 MB by default, often raised; see the tuning docs). Hint it with
big.join(F.broadcast(small), "sku"). - Sort-merge join: both sides are shuffled by the key and sorted. The default for two big tables.
- Filter and select before joining, so less data moves.
Slowdown 4: skew
One key holds most of the rows: the null customer ID, the “online store”, one giant advertiser. One task gets a huge partition and runs forever while the others sit idle. Picture a supermarket with eight tills where every shopper with a full trolley is sent to till 1.
online#0 to online#3. Its rows now spread over four tasks of about 60 seconds each.Fixes, in the order to try them: let AQE’s skew-join handling split it; handle nulls separately (filter them out and process them on their own); salt the key as above. In code, salting looks like this:
salted = sales.withColumn("salt", (F.rand() * 4).cast("int"))
partial = salted.groupBy("store_id", "salt").agg(F.sum("revenue").alias("rev"))
total = partial.groupBy("store_id").agg(F.sum("rev").alias("rev"))
4. Caching: only when reused
df.cache() keeps a DataFrame in memory after the first action. It is worth it only if you use that DataFrame several times in the same job. Caching everything wastes memory and can make jobs slower. Call unpersist() when you are done.
5. Read the Spark UI
Every Spark application has a web page (port 4040 on your laptop; on Dataproc it is linked from the cluster page as the history server). Look at:
- Jobs → Stages: which stage takes longest?
- Stage detail → task durations: a maximum far above the median means skew.
- Shuffle read and write sizes: big numbers mean a wide operation worth questioning.
- Spill (memory and disk): data did not fit in memory; partitions are too big.
- SQL tab: the physical plan, showing which join was chosen.
- Small-files problem
- Too many tiny files: too many tasks and slow listing for every reader.
- Broadcast join
- Copy the small table everywhere so the big one never moves.
- Skew
- One key with far more rows than the others, so one task does most of the work.
- Salting
- Splitting a hot key into a few random pieces, then adding the pieces back up.
- AQE
- Spark adjusting partition counts and join choices while the job runs, using real sizes.
- Physical meaning
- Picture an ad network whose attribution job has been "slow" for months. The Spark UI shows one task running 40 minutes on the rows with no user ID. Handling those rows on their own could cut the run from about 50 minutes to about 12, and the monthly bill by a similar share. Most real fixes look like that: find the one bad stage, remove the extra work.
“Caching makes Spark faster, so cache everything.” Caching costs memory and time to fill. It only pays off when the same DataFrame is used more than once in a job. Caching a DataFrame you read once just pushes other data out of memory and can cause spills.
6. Try it
In your Lesson 2.2 notebook:
- Run the province join with and without
F.broadcast(stores)and compare theexplain()output. Look forBroadcastHashJoinversusSortMergeJoin. - Create skew by setting 80 % of rows to
store_id = 1, run the daily group-by, and openhttp://localhost:4040to find the long task. - Write the data with
repartition(500)and withcoalesce(4)and count the files in each folder.
What to check: in step 2, the slowest task’s duration is many times the median; in step 3, the file counts are 500 and 4 (per partition folder if you used partitionBy).
New words: columnar, table format, compaction, shuffle partitions, AQE, broadcast hash join, sort-merge join, skew, salting, cache, spill.
- Cheaper means less work: fewer bytes read, fewer rows shuffled, even task sizes. More machines rarely cut cost.
- Store analytics data as Parquet, or as Iceberg or Delta tables when you need updates and time travel, in files of roughly 128 MB to 1 GB.
- Broadcast small tables so the big table never shuffles; filter and select before joins.
- Skew shows as one task far slower than the median; fix it with AQE, separate handling of nulls, or salting.
- Use the Spark UI to find the slow stage before changing anything.
Check yourself
Sources
- Apache Spark documentation: performance tuning (adaptive query execution, join strategy hints) https://spark.apache.org/docs/latest/sql-performance-tuning.html
- Apache Spark documentation: monitoring and the web UI https://spark.apache.org/docs/latest/web-ui.html
- Delta Lake documentation https://docs.delta.io/latest/index.html
- Apache Iceberg documentation https://iceberg.apache.org/docs/latest/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.