🧱 Delta Lake
ACID transactions, Time Travel, Schema Evolution, Optimization
🏔️ Delta Lake nima?
Open-source storage layer, Parquet formatida yozadi + _delta_log transaction log. ACID transactions — Atomicity, Consistency, Isolation, Durability. Scalable metadata — billions of files. Databricks, Apache Spark, Flink bilan integratsiya.
⏰ Time Travel
Har bir write operatsiyasi versiya yaratadi. Eski versiyaga qaytish: `VERSION AS OF 5` yoki `TIMESTAMP AS OF '2024-01-01'`. RESTORE TABLE — jadvalning eski versiyasini tiklash. VACUUM — eski fayllarni tozalash (default 7 kun).
📐 Schema Enforcement & Evolution
Schema Enforcement — schema ga mos kelmaydigan yozuvlarni rad etadi. Schema Evolution: `mergeSchema` option bilan yangi ustun qo'shish. Delta constraints: NOT NULL, CHECK.
⚡ Optimizatsiya
OPTIMIZE — kichik fayllarni birlashtirish (small file problem). Z-ORDER — co-locate related data (filter uchun data skipping). Auto Optimize: Auto Compaction + Optimized Writes. Data Skipping — min/max statistika asosida file skip.
💡 Asosiy nuqtalar
- Delta = Parquet + _delta_log (transaction log)
- ACID: concurrent writes xavfsiz
- Time Travel: version/timestamp bo'yicha eski holat
- VACUUM: eski fayllarni o'chirish (min 7 kun saqla)
- OPTIMIZE + ZORDER: read performance oshirish
- Schema enforcement: yomon data kirishini bloklaydi
📋 Kod misoli
-- Delta table yaratish
CREATE TABLE sales (
id BIGINT,
amount DOUBLE,
date DATE
) USING DELTA
LOCATION '/mnt/data/sales';
-- Time Travel
SELECT * FROM sales VERSION AS OF 10;
SELECT * FROM sales TIMESTAMP AS OF '2024-01-15';
-- Eski versiyani tiklash
RESTORE TABLE sales TO VERSION AS OF 5;
-- Optimizatsiya
OPTIMIZE sales ZORDER BY (date, category);
-- Eski fayllarni tozalash (7 kundan eski)
VACUUM sales RETAIN 168 HOURS;
-- Table history
DESCRIBE HISTORY sales LIMIT 10;🎯 Imtihon maslahatlari
- VACUUM RETAIN 0 HOURS — time travel imkoniyatini yo'qotadi! Default 7 kun (168 soat) saqlang
- Schema Evolution: `.option("mergeSchema", "true")` — yangi ustun qo'shish. Mavjud ustun typini o'zgartirish — schema enforcement xatosi
- Delta table = managed yoki external bo'lishi mumkin: managed — `DROP TABLE` data o'chiradi, external — faqat metadata
- DESCRIBE HISTORY — o'zgarishlarni kim qildi, qachon, qanday (audit trail)
- Delta Sharing — boshqa tashkilotlarga Delta table ni share qilish (Databricks-specific feature)
⚠️ Ko'p adashadigan
- VACUUM dan keyin time travel ishlaydi deb o'ylash — VACUUM eski fayllarni o'chiradi, time travel imkonsiz bo'ladi
- OPTIMIZE va VACUUM maqsadini aralashtirib yuborish: OPTIMIZE=fayl birlashtirish, VACUUM=eski fayl o'chirish
- Schema Enforcement va Schema Evolution farqini bilmaslik: Enforcement=bloklaydi, Evolution=qabul qiladi
🧠 Eslab qolish: "Delta = Parquet + Log": P (Parquet fayllar) + L (_delta_log) = Delta Lake. "TOVA" = Time travel, Optimize, Vacuum, Acid — Delta Lake 4 asosiy xususiyati
Delta Lake bo'yicha o'zingizni sinab ko'ring
Bepul interaktiv quiz, mock imtihon va to'liq darslar — CertMaster platformasida.
Bepul boshlash →