🧱 Data Governance & Lakehouse

Lakehouse arxitekturasi, Data Catalog, Lineage, Delta Sharing

Databricks Data Engineer · DE-Associate · ⏱️ 18 min · ❓ 4 savol

🏛️ Lakehouse Pattern

Lakehouse = Data Lake (arzon storage, flexible) + Data Warehouse (ACID, schema enforcement, BI performance) birlashmasi. Delta Lake bu konsepsiyani amalga oshiradi. Bir manba — batch, streaming, ML, BI uchun. Databricks = Lakehouse platform. Apache Iceberg va Apache Hudi — alternativ open table formats.

🔍 Unity Catalog Data Discovery

Data Explorer — catalog, schema, table, view, function browse. Table Tags — metadata labeling (PII, Confidential). Comments — table va column description. Search — keyword bo'yicha table topish. Data Profile — avtomatik statistika (null count, distinct, min/max).

🔗 Data Lineage

Unity Catalog — avtomatik data lineage: qaysi table qayerdan keldi. Column-level lineage: bitta ustun qaysi transformationlardan o'tdi. GDPR va compliance uchun muhim. Lineage Graph — vizual ko'rinish. Faqat Unity Catalog da mavjud (Hive Metastore da yo'q).

📤 Delta Sharing

Delta Sharing — open protocol, boshqa tashkilotlar yoki cloud lar bilan Delta table ni share qilish. Recipient — Databricks yoki non-Databricks (pandas, Spark). Share qilingan data real-time yangilanadi. Read-only. Data copy yo'q — virtual access. Apache license, cloud-neutral.

💡 Asosiy nuqtalar

Lakehouse = Lake (arzon) + Warehouse (ACID, BI) kombinatsiyasi
Unity Catalog: data discovery, tags, comments, search
Data Lineage: table va column darajasida (faqat UC)
Delta Sharing: open protocol, tashqi share, data copy yo'q
Data Profile: avtomatik statistika
Iceberg/Hudi — Delta Lake open table format alternativlari

🎯 Imtihon maslahatlari

Lakehouse = Lake + Warehouse — bitta platforma barcha analytics uchun
Delta Sharing recipient: Databricks yoki non-Databricks (Python, pandas)
Column-level lineage — faqat Unity Catalog da, Hive Metastore da yo'q
Data Lineage GDPR compliance uchun muhim — kim qaysi PII data ga kirdi
Delta Sharing: data copy qilinmaydi, virtual read-only — storage xarajat yo'q

⚠️ Ko'p adashadigan

Lakehouse = Data Lake deb o'ylash — Lakehouse ACID transaction va schema enforcement qo'shadi
Delta Sharing = data replication deb o'ylash — data copy yo'q, virtual read-only access
Unity Catalog lineage barcha operations ni track qiladi deb o'ylash — SQL va DataFrame API

🧠 Eslab qolish: "Lakehouse = Lake + House": Lake=arzon storage, House=uyushgan (ACID, schema). "Delta Sharing = Kutubxona": original kitob (data) saqlanadi, boshqalar o'qiydi — nusxa ko'chirmaydi