Home / Learn / Data & AI / Data at Scale / Lesson 3.1
All lessons

Module 3: The Google Cloud data stack · Lesson 3.1 · about 20 min

A tour of the Google Cloud data stack

Projects, regions, permissions and billing, then every Google Cloud data service on one page, and how to choose BigQuery SQL, Dataproc or Dataflow.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 20 minutes · read first: 1.2. You’ll be able to explain projects, regions, service accounts and roles, name the Google Cloud data services and the job each does, and choose between BigQuery SQL, Dataproc and Dataflow for a pipeline.

On Google Cloud, everything lives in a project that holds its own permissions and bill; once you know who may do what and who pays, the data services are just tools in the seven boxes you already know.

1. Why this matters

Most mistakes beginners make on the cloud are not about Spark or SQL. They are about permissions (“why does my job get 403 Forbidden?”), location (“why is this query so slow and expensive?”) and billing (“who left this running?”). Get those four ideas straight and the services become easy. This course uses Google Cloud; the same ideas exist on AWS and Azure under other names (Lesson 1.2 has the table).

2. Four ideas to understand first

  • Project: the box that holds resources, permissions and billing. Companies usually have many: retail-data-dev, retail-data-prod, one per team. BigQuery tables are named project.dataset.table.
  • Region and location: where data physically lives (northamerica-northeast1 is Montréal, northamerica-northeast2 is Toronto; US and EU are multi-regions). Keep data and the compute that reads it in the same location: it is cheaper, faster, and privacy law often requires it.
  • IAM (identity and access management): who can do what. People and service accounts (identities for programs) get roles such as roles/bigquery.dataViewer or roles/storage.objectAdmin. The rule is least privilege: give only what is needed, to groups rather than to individuals.
  • Billing: every service is metered. Set budgets and alerts on day one, and label resources (team=retail-media) so costs can be split by team.

Think of an office building. The organisation is the company; projects are floors, each with its own budget; services are rooms (the library is BigQuery, the storeroom is Cloud Storage, the workshop is Dataproc); IAM is the key-card system; and a service account is a key card issued to a robot that works night shifts.

Watch one night-shift robot at work:

The job starts. At 02:00 the nightly pipeline runs as the service account sales-pipeline@retail-data-prod.iam.gserviceaccount.com, not as any person.
It asks BigQuery to run a query. IAM checks: does this account hold bigquery.jobUser on the project (allowed to run jobs) and bigquery.dataViewer on the dataset retail? Yes, so the query runs.
A mistake in the code reads the loyalty table. The account has no role on the dataset pii. BigQuery answers 403 Access Denied, and the attempt is written to the audit log. Least privilege just stopped a leak of personal data.
Someone pays. The bytes the query scanned are billed to retail-data-prod, tagged with the label team=retail-media, so the monthly cost report shows which team spent what.

3. Budgets and alerts

A budget is a monthly amount you expect to spend; alerts email you when actual (or forecast) spend crosses a share of it, such as 50 %, 90 % and 100 %. Important: a budget alert only warns. It does not stop anything from running. Someone has to read the email and act.

4. The services on one page

Job Service In one line
Store files Cloud Storage (GCS) Object storage for any file; the data lake
Warehouse BigQuery Serverless SQL warehouse over petabytes
Spark and Hadoop Dataproc Managed Spark clusters, or serverless Spark batches
Beam pipelines Dataflow Serverless batch and streaming pipelines (Apache Beam)
Messaging Pub/Sub Global, serverless publish-subscribe messaging
Kafka Managed Service for Apache Kafka Google-run Kafka clusters (or run your own, or Confluent Cloud)
Orchestration Cloud Composer Managed Airflow
SQL workflows Dataform SQL transformation workflows in BigQuery, similar to dbt
Change data capture Datastream Streams changes from MySQL, PostgreSQL, Oracle or SQL Server into BigQuery or Cloud Storage
Wide-column NoSQL Bigtable Millisecond reads and writes at huge scale
Relational Cloud SQL, AlloyDB, Spanner Postgres or MySQL; high-performance Postgres; globally distributed SQL
Governance Dataplex Discover, catalogue, profile and govern data across lake and warehouse
BI Looker, Looker Studio A semantic model and dashboards; free lightweight dashboards
ML and AI Vertex AI Training, pipelines, feature store, model registry, serving, Gemini models
Clean rooms BigQuery data clean rooms Share data with partners without exposing rows (Module 7)

Product names and features change; when in doubt, check the Google Cloud docs.

5. Which processing service?

  • BigQuery SQL when the data is in BigQuery and the logic is SQL. This covers most analytics transformations. Schedule it with Dataform, dbt or Composer.
  • Dataproc when you have existing Spark code, need the Spark ecosystem (Delta, Iceberg, MLlib), or want portable open-source jobs.
  • Dataflow for streaming pipelines (Pub/Sub or Kafka in, BigQuery or Bigtable out) and serverless batch in Beam.
  • Cloud Run functions or jobs for small glue: call an API, move a file, send an alert.

Google Cloud is especially strong where BigQuery and advertising matter: retailers, media and marketing companies, because Google’s ads and analytics data can be exported straight into BigQuery. Many retail media networks build on it for that reason and for its clean rooms.

Project
A container for resources, permissions and one bill.
Service account
An identity for a program, with its own roles. Never a person.
Role
A named bundle of permissions, such as "may read tables in this dataset".
Least privilege
Give each identity only the roles its job needs, and nothing more.
Budget alert
An email when spend passes a share of a monthly budget. It warns; it does not stop.
Physical meaning
If a restaurant chain's pipeline key leaks, least privilege decides the damage: a key that can only read the sales dataset exposes sales; an owner key exposes everything, including customer data, and can run up any bill.

“Setting a budget caps my spending.” A Google Cloud budget sends alerts; it does not switch anything off. To truly cap spend you need extra steps: quotas, a maximum bytes billed on queries, or an automated action triggered by the alert. Check the billing docs for the current options.

6. Try it

Open the free BigQuery sandbox (no credit card; see Free sandboxes). Create a project, open BigQuery Studio and paste:

SELECT name, SUM(number) AS babies
FROM `bigquery-public-data.usa_names.usa_1910_current`
WHERE year >= 2000
GROUP BY name
ORDER BY babies DESC
LIMIT 10;

Before you press Run, look at the top right of the editor: it tells you how many bytes the query will process. Write that number down. It is the main driver of BigQuery cost, and Lesson 3.3 is about making it smaller.

What to check: you can find the project ID in the table name you would write for your own data (your-project.dataset.table), and you have set a budget with at least one alert in the billing page (even a tiny one).

New words: project, region, multi-region, IAM, service account, role, least privilege, budget, label, Composer, Dataform, Datastream, Dataplex.

  • Projects hold resources, permissions and billing; tables are named project.dataset.table.
  • Keep data and compute in the same location: it is cheaper, faster and often legally required.
  • Programs run as service accounts with least-privilege roles; a missing role gives 403 and protects data.
  • Budgets send alerts but do not stop spending; label resources so cost can be split by team.
  • Default to SQL in BigQuery; use Dataproc for Spark code and Dataflow for streaming or Beam.

Check yourself

1A pipeline that runs on a schedule should sign in to BigQuery as...
2Which service is Google's managed Apache Airflow?
3Your transformations are plain SQL over data already in BigQuery. What is usually simplest?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.