🧱 Databricks Platform

Workspace, Clusters, DBFS, Repos va asosiy konseptlar

Databricks Data Engineer · DE-Associate · ⏱️ 20 min · ❓ 5 savol

🏠 Databricks Workspace

Workspace — Databricks muhitining asosiy interfeysi. Notebooks — Python, SQL, Scala, R. Repos — Git integration (GitHub, GitLab, Bitbucket). DBFS (Databricks File System) — distributed file system, S3/ADLS/GCS ustida abstraksiya.

⚙️ Clusters

All-Purpose Cluster — interaktiv development, notebook ishlatish. Job Cluster — avtomatik yaratiladi, job tugagach o'chiriladi (cost-efficient). Cluster Policy — konfiguratsiya cheklash. Auto-scaling va Auto-termination.

🔒 Unity Catalog

Unified governance va data discovery. Metastore → Catalog → Schema (Database) → Table/View/Function. 3-level namespace: `catalog.schema.table`. Column-level security, row-level filtering, data lineage.

💡 Asosiy nuqtalar

All-Purpose: interactive, Job Cluster: automated (arzon)
DBFS: S3/ADLS ustidagi virtual filesystem
Unity Catalog: 3-level namespace, governance
Repos: Git integration notebook version control uchun
Auto-termination: idle cluster'ni avtomatik o'chirish

📋 Kod misoli

# Cluster konfiguratsiya misoli
Runtime: 14.3 LTS (Spark 3.5, Scala 2.12)
Node type: Standard_DS3_v2
Min workers: 2, Max workers: 8 (autoscale)
Auto-termination: 30 daqiqa

# DBFS path misollari
dbfs:/FileStore/data/my_file.csv
/dbfs/FileStore/data/my_file.csv  # local path
%fs ls dbfs:/FileStore/

🎯 Imtihon maslahatlari

Job Cluster — automated job da har doim afzal: start → run → terminate (cost-efficient)
Unity Catalog 3-level: catalog.schema.table — AWS Glue da faqat 2-level (database.table)
DBFS root (`dbfs:/`) — Databricks o'zi boshqaradi, foydalanuvchi S3 bucket EMAS
Cluster Policy — admin tomonidan cheklangan konfiguratsiya. Non-admin userlar bu policyga rioya qilishi shart
Repos — Git commit/push notebook dan to'g'ridan-to'g'ri qilish mumkin

⚠️ Ko'p adashadigan

All-Purpose Cluster production job uchun ishlatish — qimmat! Job Cluster ishlatish kerak
DBFS = S3 deb o'ylash — DBFS S3 ustidagi abstraktsiya, alohida storage emas
Unity Catalog metastore = Hive metastore deb aralashtirib yuborish — ikkalasi mavjud bo'lishi mumkin

🧠 Eslab qolish: "AJR" = All-purpose (interactive), Job cluster (production), Repos (git) — 3 muhim Databricks tushuncha. "UC 3L" = Unity Catalog 3-Level: Catalog → Schema → Table