🧱 MLflow & Model Registry

MLflow Tracking, Model Registry, Experiment management, autolog

Databricks Data Engineer · DE-Associate · ⏱️ 24 min · ❓ 4 savol

🧪 MLflow nima?

MLflow — open-source ML lifecycle management platform. 4 komponenti: Tracking (experiment, run, params, metrics, artifacts), Projects (reproducible ML code packaging), Models (model packaging va serving), Registry (centralized model store, versioning, stage management). Databricks da fully managed MLflow.

📊 MLflow Tracking

Experiment — related runlar to'plami. Run — bitta ML training session. Log qilinadigan narsalar: params (hyperparameters: learning_rate, n_estimators), metrics (accuracy, loss), artifacts (model file, plot, CSV). `mlflow.autolog()` — sklearn, xgboost, pytorch kabi ko'p frameworklar uchun avtomatik log.

🏪 MLflow Model Registry

Registered Model — versiyalangan model. Stages: None → Staging → Production → Archived. Model alias — "champion", "challenger" kabi. Unity Catalog bilan: 3-level namespace `catalog.schema.model`. Model versioning, lineage, approval workflow.

⚡ Spark Performance Tuning

Partitioning: `repartition(n)` — shuffle bilan, `coalesce(n)` — shuffle siz (faqat kamaytirish). Optimal partition: core * 2-4. Broadcast Join: kichik DataFrame ni barcha executorga yuborish (`F.broadcast(small_df)`). Caching: `df.cache()` — ko'p marta ishlatilsa. AQE: adaptive query execution.

💡 Asosiy nuqtalar

MLflow: Tracking, Projects, Models, Registry — 4 komponent
mlflow.autolog() — sklearn, xgboost, pytorch uchun avtomatik log
Model Registry stages: None → Staging → Production
repartition = shuffle (kattalashtirish va kamaytirish)
coalesce = shuffle siz (faqat kamaytirish)
Broadcast join: <10MB table uchun shuffle oldini olish

📋 Kod misoli

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier

# Experiment
mlflow.set_experiment("fraud_detection_v2")

with mlflow.start_run(run_name="rf_baseline"):
    mlflow.log_param("n_estimators", 100)
    mlflow.log_param("max_depth", 5)

    model = RandomForestClassifier(n_estimators=100, max_depth=5)
    model.fit(X_train, y_train)

    accuracy = model.score(X_test, y_test)
    mlflow.log_metric("accuracy", accuracy)

    # Model saqlash va register qilish
    mlflow.sklearn.log_model(model, "model",
        registered_model_name="fraud_detector")

# Broadcast join
from pyspark.sql import functions as F
result = large_df.join(F.broadcast(small_lookup), on="id")

🎯 Imtihon maslahatlari

MLflow Tracking URI: Databricks da managed MLflow — URI avtomatik sozlanadi
repartition vs coalesce: repartition = shuffle (qimmat), coalesce = shuffle siz (faqat kamaytirish)
Model Registry "Production" stage = deployment uchun tayyor ekanligini bildiradi (avtomatik deploy emas)
mlflow.autolog() — `mlflow.sklearn.autolog()` yoki `mlflow.autolog()` ikkalasi ishlaydi
Unity Catalog Model Registry: 3-level namespace, fine-grained access control

⚠️ Ko'p adashadigan

repartition va coalesce: repartition kattalashtiradi ham, coalesce faqat kamaytiradi
MLflow params va metrics farqi: params = static (hyperparameter), metrics = o'zgaruvchan (epoch accuracy)
Broadcast join har doim ishlatish — katta DataFrame broadcast OOM xatosi beradi

🧠 Eslab qolish: "MLflow 4C" = Collect (tracking), Catalog (registry), Compare (experiments), Champion (production model). "rePARTITION = reshuffle, COALESCE = combine"

MLflow & Model Registry bo'yicha o'zingizni sinab ko'ring

Bepul interaktiv quiz, mock imtihon va to'liq darslar — CertMaster platformasida.

Bepul boshlash →