🔵 Data & Analytics Services

Pub/Sub, Dataflow, Dataproc, BigQuery optimization, Cloud Composer

Google Cloud ACE · ACE · ⏱️ 24 min · ❓ 5 savol

📨 Cloud Pub/Sub

Fully managed message broker. Topic — message chiqarish kanali. Subscription: Pull (subscriber so'raydi) yoki Push (Pub/Sub endpoint ga yuboradi). At-least-once delivery. Message ordering — ordering key bilan. Snapshot — subscription state saqlash. Dataflow, Cloud Functions, Cloud Run bilan integratsiya.

🌊 Cloud Dataflow

Fully managed Apache Beam runner. Batch va Streaming bitta API. Auto-scaling — worker soni avtomatik. Templates: Classic (GCS da JAR) va Flex (Docker). Pub/Sub → Dataflow → BigQuery — keng tarqalgan pipeline pattern. SQL-like Dataflow SQL. Streaming windowing: Fixed, Sliding, Session.

🐘 Cloud Dataproc

Managed Hadoop va Spark cluster. Cluster types: Standard, Single Node, High Availability. Autoscaling. Ephemeral cluster — job uchun yaratiladi, job tugagach o'chiriladi (tejamkor). GCS + Dataproc — HDFS o'rniga GCS ishlatish. Hive, Pig, Spark, Presto qo'llab-quvvatlaydi.

🎼 Cloud Composer

Managed Apache Airflow. DAG (Directed Acyclic Graph) — workflow definition. Python da yoziladi. Operators: BashOperator, BigQueryOperator, DataflowOperator, PubSubOperator. Composer 2 — Autopilot GKE asosida, versioning, environment update. Batch ETL orchestration uchun ideal.

📊 GCP Data Processing Xizmatlar

Xizmat	Framework	Use Case	Asosiy xususiyat
Pub/Sub	Native	Messaging, event bus	At-least-once, push/pull
Dataflow	Apache Beam	Batch + Streaming ETL	Unified, auto-scale, serverless
Dataproc	Spark/Hadoop	Existing Spark jobs	Managed cluster, ephemeral
BigQuery	SQL	Analytics (OLAP)	Serverless, PB scale
Composer	Apache Airflow	Workflow orchestration	DAG, scheduling, dependencies

💡 Asosiy nuqtalar

Pub/Sub: topic + subscription (pull/push), at-least-once
Dataflow: Apache Beam, batch+stream, auto-scaling
Dataproc: managed Spark/Hadoop, ephemeral cluster
Composer: managed Airflow, DAG, Python, ETL orchestration
Dataflow vs Dataproc: Beam API vs Spark/Hadoop API

🎯 Imtihon maslahatlari

"Apache Beam" → Dataflow. "Apache Spark existing code" → Dataproc
Dataproc ephemeral cluster: job uchun cluster yaratib, job tugagach o'chirish = tejamkor pattern
Pub/Sub pull: subscriber so'raydi. Push: Pub/Sub endpoint ga HTTP POST yuboradi
Composer DAG = Python: BashOperator, BigQueryOperator, GCSOperator — operator tanlash muhim
Dataflow template — pre-built pipeline: Pub/Sub to BigQuery, GCS to BigQuery

⚠️ Ko'p adashadigan

Dataflow = Dataproc deb o'ylash: Dataflow=Apache Beam (serverless), Dataproc=Spark/Hadoop (cluster)
Pub/Sub at-least-once = exactly-once deb o'ylash — Pub/Sub de-duplication uchun subscriber tomonida idempotent processing kerak
Composer = Dataflow deb chalg'ish: Composer=orkestrasiya (scheduler), Dataflow=execution (data processing)

🧠 Eslab qolish: "PDBC" = Pub/Sub (message), Dataflow (process), BigQuery (store/analyze), Composer (orchestrate) — GCP data pipeline to'rt qadami