🧱 Apache Spark Fundamentals

DataFrame API, Spark SQL, transformations, actions, optimizatsiya

Databricks Data Engineer · DE-Associate · ⏱️ 25 min · ❓ 6 savol

⚡ Spark Architecture

Driver — SparkContext yaratadi, DAG rejalashtiradi. Executors — haqiqiy ishni bajaradi (task). Cluster Manager: YARN, Kubernetes, Databricks (k8s asosida). DAG (Directed Acyclic Graph) — transformations rejasi. Stage → Task ierarxiyasi.

🔄 Transformations vs Actions

Lazy Evaluation — transformation bajarilmaydi, action chaqirilguncha. Narrow transformations: filter, select, map — shuffle yo'q. Wide transformations: groupBy, join, distinct — shuffle kerak (expensive!). Actions: show(), count(), collect(), write().

📊 DataFrame API

Structured API. `spark.read`, `df.select()`, `df.filter()`, `df.groupBy().agg()`, `df.join()`, `df.write`. Catalyst Optimizer — query optimallashtiradi. Tungsten — memory management. Adaptive Query Execution (AQE) — runtime optimallashtirish.

💡 Spark SQL & Caching

SQL queries to'g'ridan-to'g'ri Spark'da. `spark.sql("SELECT ...")`. Temp Views: `df.createOrReplaceTempView("name")`. Caching: `df.cache()` yoki `df.persist(StorageLevel.MEMORY_AND_DISK)`. Cache — faqat bir xil datasetni ko'p marta ishlatganda foydali.

💡 Asosiy nuqtalar

Lazy evaluation: action bo'lmaguncha hech narsa bajarilmaydi
Wide transformation (groupBy, join) = shuffle = expensive
AQE: runtime query plan optimallashtirish
cache() = reuse uchun, lekin memory band qiladi
Catalyst Optimizer barcha SQL/DataFrame optimizatsiyasini bajaradi

📋 Kod misoli

from pyspark.sql import functions as F

# DataFrame operations
df = spark.read.format("delta").load("/data/sales")

result = (df
  .filter(F.col("date") >= "2024-01-01")
  .groupBy("category")
  .agg(
    F.sum("amount").alias("total"),
    F.count("*").alias("count")
  )
  .orderBy(F.desc("total"))
)

result.show(10)

# Spark SQL alternative
df.createOrReplaceTempView("sales")
spark.sql("""
  SELECT category, SUM(amount) as total
  FROM sales
  WHERE date >= '2024-01-01'
  GROUP BY category
  ORDER BY total DESC
""")

🎯 Imtihon maslahatlari

Narrow vs Wide: filter/select/map = narrow (no shuffle), groupBy/join/distinct = wide (shuffle)
repartition() vs coalesce(): repartition — full shuffle (sayla ko'paytirsa), coalesce — minimal shuffle (faqat kamaytirsa)
Broadcast Join — kichik table (< 10MB default) large table bilan join da shuffle oldini oladi: `F.broadcast(small_df)`
collect() — barcha data driver ga keladi: OOM xavfi! Faqat kichik natijalar uchun
Spark UI: Stage, Task, Shuffle read/write — slow job debugging uchun birinchi qarang

⚠️ Ko'p adashadigan

count() transformation deb o'ylash — count() = ACTION, darhol bajariladi
cache() har doim foydali deb hisoblash — bir marta ishlatiladigan DF da cache memory isrof
groupBy ga distinct ham shuffle qilishini bilmaslik — `df.distinct()` ham wide transformation

🧠 Eslab qolish: "FAST Cache" = Filter/select (narrow, fast), Aggregation/Sort/Transformation wide, Cache for reuse. "Actions Start Computation: show, count, write, collect"