Vertex AI Experiments Deep Intuition

1. One-Sentence Essence

Vertex AI Experiments is a thin, serverless tracking API layered on top of a graph metadata store — it doesn’t run your training, it remembers it, turning each run into a queryable node in a lineage graph so you can compare configurations and prove where a model came from.

Hold onto the phrase “it remembers, it doesn’t run.” Almost every gotcha, judgment call, and design quirk in this document flows from the fact that Experiments is a bookkeeping layer, not a compute layer. It watches your training from the sidelines and writes down what happened.

2. The Problem It Solved

Picture a data scientist three weeks into a churn model. They’ve trained maybe forty variants: different learning rates, three feature sets, two architectures, a handful of random seeds. The results live in a spreadsheet that’s half-abandoned, some notebook cells that got re-run and overwrote their own outputs, a few print() statements in a terminal that’s since been closed, and a folder of model_final_v3_actually_final.joblib files. Somebody asks: “Which config gave us the 0.91 AUC you mentioned in standup?” And the honest answer is I don’t know anymore.

This is the universal pain that experiment tracking exists to kill. The specific frustrations are: irreproducibility (you can’t recreate a result because you didn’t record the inputs), incomparability (you can’t line up run A against run B because they logged different things in different places), and lost lineage (you can’t prove which dataset version and which preprocessing produced a given model — which becomes a compliance problem the moment the model touches money or people).

Before managed trackers, teams either built their own logging into a database (real engineering effort, always half-finished) or adopted an open-source tool like MLflow (excellent, but you host and babysit the tracking server yourself) or a SaaS like Weights & Biases (polished, but per-seat pricing and your data lives on someone else’s cloud). Most teams pick their tracking tool based on what the first ML engineer installed, and end up with a large annual bill when a cheaper option would have served most of their needs.

Google’s insight with Vertex AI Experiments was to make tracking serverless and native to the platform. It does not require any always-on infrastructure like MLflow to function, which makes it a service where you pay only for storage, at very minimal cost. No server to run. No seats to license. It plugs into Google Cloud IAM for access control, into Vertex Pipelines and Training for automatic capture, and — crucially — it’s built on the same metadata graph (Vertex ML Metadata) that records artifact lineage across the whole platform. So tracking isn’t a bolt-on; it’s the same substrate that answers “where did this model come from” for the entire ML lifecycle. We’ll see in the Mental Model why that shared substrate is the whole point.

3. The Concepts You Need

These terms recur constantly. Read this section slowly; the rest of the document assumes you own this vocabulary.

The tracking hierarchy

Experiment — the top-level container. It’s a context that groups your runs and the artifacts you create into a logical session. Think “the churn-prediction project” or “the fraud-model Q3 push.” An experiment holds n runs. You name it once (fraud-detection-v2) and everything related lands under it.
Experiment Run — one attempt. A specific, trackable execution within an experiment, which logs inputs (algorithm, parameters, datasets) and outputs (models, checkpoints, metrics) to monitor and compare development iterations. One run = one configuration you tried. Forty variants = forty runs.
Parameters — the inputs you chose. Keyed input values that configure a run, regulate its behavior, and affect its results — learning rate, n_estimators, optimizer, feature-set version. Logged as key-value pairs via log_params(). Parameters are the knobs.
Summary metrics — the outputs you measure, as single final numbers. Single-value scalar metrics that represent a final summary of an experiment run — final accuracy, test F1, AUC. Logged via log_metrics(). Metrics are the scoreboard.
Time-series metrics — the same idea but longitudinal. Metric values where each value represents a step in the training routine — for example loss per epoch. These are your training curves. They do not live with the summary metrics; they live in TensorBoard (see below). Logged via log_time_series_metrics().
Classification metrics — confusion matrices and ROC curves, logged via log_classification_metrics(). A convenience for the two structured outputs everyone wants.

The metadata substrate

Vertex ML Metadata — the graph database underneath everything. A managed ML Metadata store based on the open-source ML Metadata (MLMD) library from Google’s TensorFlow Extended team, which lets you record, analyze, debug, and audit metadata and artifacts produced during your ML journey. This is the thing that actually stores your experiments. An Experiment is literally a metadata Context; a Run is a Context (or, in legacy form, an Execution) inside it.
Artifact — a discrete entity or piece of data produced and consumed by an ML workflow: datasets, models, input files, training logs. Artifacts are typed by schema — supported schema types include system.Dataset, system.Model, and system.Artifact.
Execution — a record of a step that ran (e.g. “preprocessing” or “training”). Executions consume input artifacts and produce output artifacts. This produce/consume relationship is what forms the lineage edges.
Lineage — the graph you get for free once artifacts and executions are linked: the chain from raw dataset → preprocessing execution → transformed dataset → training execution → model. This is the “prove where the model came from” capability.
Context — the generic MLMD grouping primitive. Both Experiments and Runs are Contexts. You rarely say “Context” out loud, but knowing that Experiment is a Context explains why the API behaves the way it does.

The visualization companion

Vertex AI TensorBoard — an enterprise-ready managed version of open-source TensorBoard, Google’s visualization tool for ML experiments. This is where time-series metrics are stored and rendered. Vertex AI Experiments stores only a reference to the TensorBoard resource; the actual step-by-step curves live in the TensorBoard backend. Remember this split — it’s the source of half the confusion in this tool.
TensorBoard instance — a regional resource that acts like a database that stores information on TensorBoard experiments, and for each one, detailed time-series data. You create one (often implicitly), and experiments back their time-series onto it.

The workflow neighbors (things Experiments hooks into but that aren’t tracking per se)

PipelineJob — an execution instance of an ML pipeline definition — a set of ML tasks interconnected by input-output dependencies. An experiment can contain pipeline runs alongside plain runs.
Autologging — a Vertex AI SDK feature that automatically logs parameters and metrics from model-training runs, eliminating the need to log manually. Built on MLflow’s autologging under the hood.

The single most important relationship to internalize: Experiment → contains → Runs → each of which points to → Parameters + Summary Metrics (in metadata) + a reference to Time-Series Metrics (in TensorBoard) + Artifacts (in metadata, linked by lineage). Everything else is detail.

4. The Distilled Introduction

Here’s everything a long tutorial would walk you through, compressed. The entire surface area is essentially one Python package: google-cloud-aiplatform, imported as aiplatform (older samples alias it vertex_ai).

Setup. Install and initialize:

pip install google-cloud-aiplatform

from google.cloud import aiplatform

aiplatform.init(
    project="your-project-id",
    location="us-central1",
    experiment="fraud-detection-v2",              # names/creates the experiment
    experiment_description="Comparing approaches for fraud detection",
)

The experiment name groups all related runs together — pick something descriptive, because you’ll be reading it months from now. Calling init() with an experiment= argument creates the experiment if it doesn’t exist. Behind the scenes, since aiplatform 1.25+, the init call checks for a backing TensorBoard instance and, if none exists, creates a default one and assigns it to the experiment. That auto-creation matters for cost (see Downsides).

The core loop — one run, manual logging. The fundamental pattern is: start a run, log the knobs, train, log the scoreboard, end the run.

with aiplatform.start_run("run-random-forest-v1") as run:
    aiplatform.log_params({"model_type": "random_forest",
                           "n_estimators": 200, "max_depth": 12})
    # ... train your model however you like ...
    aiplatform.log_metrics({"accuracy": 0.89, "f1_score": 0.86})

The with block is the idiomatic form; it calls end_run() for you on exit. Without the context manager you’d call aiplatform.start_run("name") … aiplatform.end_run() explicitly. You create the run with start_run() and get access to the resulting parameters and metrics after ending it with end_run().

log_params() and log_metrics() both take a plain dict of key-value pairs. Params are your inputs; metrics are your final numbers. That’s genuinely most of what you need day to day.

Time-series metrics (training curves). If you want per-epoch loss curves, you use a different call, and it requires a TensorBoard backing:

for step, epoch_loss in enumerate(training_losses):
    aiplatform.log_time_series_metrics({"loss": epoch_loss}, step=step)

All metrics logged through log_time_series_metrics are stored as time-series metrics in Vertex AI TensorBoard, which is the backing time-series metric store. The step argument is optional — if not provided, an increment over the latest step already logged is used; if the step already exists for a key, it’s overwritten. This is the one API that will error without a TensorBoard instance attached.

The distinction that trips everyone. Summary metrics (log_metrics) are a final scalar — “the model ended at 0.89 accuracy.” Time-series metrics (log_time_series_metrics) are a sequence — “accuracy at each epoch.” Summary metrics are single-value scalars stored next to time-series metrics and represent a final summary; a classic use case is early stopping, where the restored best model’s score is logged as a summary metric because the latest time-series value isn’t representative of the restored model. Log both when it helps: the curve to debug the training dynamics, the summary to rank the run.

Autologging — one line instead of many. If you’re using a supported framework, skip most of the manual calls:

aiplatform.autolog()
# ...train a Keras / sklearn / XGBoost / PyTorch Lightning model...
# params + metrics captured automatically

With autologging you log parameters, performance metrics, and lineage artifacts by adding one line of code to your training script, without explicitly calling other logging methods. Autologging is recommended if you use Fastai, Gluon, Keras, LightGBM, PyTorch Lightning, scikit-learn, Spark, Statsmodels, or XGBoost. Under the hood, Vertex AI SDK autologging uses MLflow’s autologging in its implementation. Two modes: let the SDK auto-create runs for you (auto-created runs get names like {framework}-{timestamp}-{uid}, e.g. tensorflow-2023-01-04-16-09-20-86a88), or call start_run("my-name") first to control the run name and pin the autologged data to it.

Comparing runs. The everyday payoff. Pull all runs of an experiment into a DataFrame:

df = aiplatform.get_experiment_df("fraud-detection-v2")
# columns look like: run_name, param.n_estimators, param.max_depth,
#                    metric.accuracy, metric.f1_score
best = df.sort_values("metric.accuracy", ascending=False)
rf_only = df[df["param.model_type"] == "random_forest"]

The comparison features — DataFrames, filtering, sorting — make it easy to find your best configuration and understand why it beats the alternatives. You can also compare visually in the Google Cloud console under the Experiments section, and open TensorBoard from there for the curves.

Lineage (the part people skip and shouldn’t). To record that a run consumed a dataset and produced a model, you use executions and artifacts:

with aiplatform.start_run("run-with-lineage"):
    with aiplatform.start_execution(
            schema_title="system.ContainerExecution",
            display_name="training") as execution:
        execution.assign_input_artifacts([dataset_artifact])
        # ...train...
        execution.assign_output_artifacts([model_artifact])

You instantiate an execution with start_execution(), attach inputs with assign_input_artifacts(), and attach outputs with assign_output_artifacts(). This is what builds the lineage graph. Autologging captures some of this automatically; manual lineage gives you full control.

Running a training job with tracking. For real jobs (not notebook toys), you attach the experiment to a Vertex custom training job:

job = aiplatform.CustomJob.from_local_script(...)
job.run(experiment="fraud-detection-v2", experiment_run="run-attempt-7")

Vertex AI’s managed training service lets you enable experiment tracking to capture parameters and performance metrics when submitting a custom job; both prebuilt and custom containers are supported. Inside the training script you just call aiplatform.log_params() / log_metrics() and it flows to the right run. The experiment must have a TensorBoard instance; if experiment_run isn’t specified, a run is auto-created.

Everything up to here was the API surface. The rest of this section is the part that was missing: four complete, runnable walkthroughs that show the tool doing real work, plus how to compare runs and pull the data into your own apps. Read these as the “10-hour tutorial, distilled” — every line is here because you’d hit it in practice.

Worked Example A — scikit-learn (Pipeline), manual logging

The scenario: a classic tabular classification task. We’ll train a decision tree inside a scikit-learn Pipeline (encoder + model, the way you’d actually structure it) on the Iris dataset, and log everything by hand. Manual logging is worth doing at least once even if you’ll later switch to autolog, because it makes the “params in, metrics out” model concrete.

from google.cloud import aiplatform
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, f1_score

aiplatform.init(
    project="your-project-id",
    location="us-central1",
    experiment="iris-sklearn",
)

# The configuration we're going to try. Logging these is what makes the run reproducible.
config = {"model_type": "decision_tree", "max_depth": 4,
          "criterion": "gini", "test_size": 0.2, "random_state": 42}

X, y = load_iris(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=config["test_size"], random_state=config["random_state"], stratify=y)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", DecisionTreeClassifier(
        max_depth=config["max_depth"], criterion=config["criterion"],
        random_state=config["random_state"])),
])

with aiplatform.start_run("dt-depth4-gini") as run:
    aiplatform.log_params(config)          # inputs: the knobs
    pipe.fit(X_train, y_train)
    preds = pipe.predict(X_test)
    aiplatform.log_metrics({                # outputs: the scoreboard
        "accuracy": float(accuracy_score(y_test, preds)),
        "f1_macro": float(f1_score(y_test, preds, average="macro")),
    })
    # confusion matrix + ROC get their own structured call:
    aiplatform.log_classification_metrics(
        labels=[str(c) for c in load_iris().target_names],
        matrix=__import__("sklearn.metrics", fromlist=["confusion_matrix"])
               .confusion_matrix(y_test, preds).tolist(),
        display_name="iris-confusion-matrix",
    )

Three things to notice. First, config is a plain dict and it’s logged before training — log intentions up front so a crashed run still records what it was attempting. Second, metrics are cast to float — the metadata store wants JSON-serializable scalars, and numpy types (np.float64) will sometimes trip serialization. Third, log_classification_metrics is the dedicated call for confusion matrices and ROC curves; don’t try to shove a matrix through log_metrics.

Now run a second, different configuration so we have something to compare — this is the entire point of tracking:

config2 = {**config, "model_type": "random_forest", "n_estimators": 300, "max_depth": 6}
from sklearn.ensemble import RandomForestClassifier

pipe2 = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(
        n_estimators=config2["n_estimators"], max_depth=config2["max_depth"],
        random_state=config2["random_state"])),
])

with aiplatform.start_run("rf-300-depth6"):
    aiplatform.log_params(config2)
    pipe2.fit(X_train, y_train)
    preds2 = pipe2.predict(X_test)
    aiplatform.log_metrics({
        "accuracy": float(accuracy_score(y_test, preds2)),
        "f1_macro": float(f1_score(y_test, preds2, average="macro")),
    })

Two runs, same experiment, different configs. We’ll compare them in a moment.

Worked Example B — the same task with autologging

Now watch how much of the above disappears when you let autolog do it. scikit-learn is a supported framework, so aiplatform.autolog() intercepts the fit() call and records the estimator’s parameters and evaluation metrics automatically.

from google.cloud import aiplatform

# Autolog can produce time-series metrics for some frameworks, so give the experiment
# a TensorBoard backing up front (see Core Idea 3 in the Mental Model).
tb = aiplatform.Tensorboard.create(display_name="iris-autolog-tb")
aiplatform.init(
    project="your-project-id",
    location="us-central1",
    experiment="iris-autolog",
    experiment_tensorboard=tb,
)

aiplatform.autolog()   # <-- the one line

# Automatic run creation: no start_run/end_run needed. Just train.
X, y = load_iris(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=42, stratify=y)
pipe = Pipeline([("scaler", StandardScaler()),
                 ("model", DecisionTreeClassifier(max_depth=4, random_state=42))])
pipe.fit(X_train, y_train)   # params + metrics captured automatically

That’s it — no start_run, no log_params, no log_metrics. The SDK creates a run for you named like sklearn-2026-07-06-.... If you want a readable run name instead of the auto-generated one, open a run yourself first and autolog will write into it:

aiplatform.autolog()
with aiplatform.start_run("dt-depth4-autologged"):
    pipe.fit(X_train, y_train)     # autologged into YOUR named run

And to turn autolog off (so a later block goes back to manual control):

aiplatform.autolog(disable=True)

The trade is visible here: autolog is one line but records what the framework considers its params and metrics — it won’t know about your business KPI or your feature-set version. The experienced pattern (see Judgment Call 1) is autolog as the floor, plus a few manual log_params/log_metrics calls for the things autolog can’t see:

aiplatform.autolog()
with aiplatform.start_run("dt-with-business-context"):
    aiplatform.log_params({"feature_set": "v3", "data_snapshot": "2026-07-01"})
    pipe.fit(X_train, y_train)                       # framework params/metrics: auto
    aiplatform.log_metrics({"cost_weighted_error": 0.031})  # your metric: manual

Worked Example C — PyTorch, manual time-series logging

Deep learning is where time-series logging earns its keep: you want the loss curve per epoch, not just the final number. PyTorch is not in autolog’s first-class list the way Lightning is, so plain PyTorch is the perfect case for showing manual log_time_series_metrics. We’ll train a tiny MLP on Iris.

import torch
import torch.nn as nn
from google.cloud import aiplatform
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Time-series logging REQUIRES a TensorBoard backing. Attach it at init.
tb = aiplatform.Tensorboard.create(display_name="iris-pytorch-tb")
aiplatform.init(
    project="your-project-id", location="us-central1",
    experiment="iris-pytorch", experiment_tensorboard=tb,
)

# Data
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                          random_state=42, stratify=y)
X_tr = torch.tensor(X_tr, dtype=torch.float32); y_tr = torch.tensor(y_tr)
X_te = torch.tensor(X_te, dtype=torch.float32); y_te = torch.tensor(y_te)

# Model
class MLP(nn.Module):
    def __init__(self, hidden=16):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(4, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 3))
    def forward(self, x): return self.net(x)

hparams = {"hidden": 16, "lr": 0.05, "epochs": 80, "optimizer": "adam"}
model = MLP(hparams["hidden"])
opt = torch.optim.Adam(model.parameters(), lr=hparams["lr"])
loss_fn = nn.CrossEntropyLoss()

with aiplatform.start_run("mlp-h16-adam"):
    aiplatform.log_params(hparams)                 # inputs, logged once

    for epoch in range(hparams["epochs"]):
        model.train()
        opt.zero_grad()
        logits = model(X_tr)
        loss = loss_fn(logits, y_tr)
        loss.backward(); opt.step()

        # validation
        model.eval()
        with torch.no_grad():
            val_logits = model(X_te)
            val_loss = loss_fn(val_logits, y_te).item()
            val_acc = (val_logits.argmax(1) == y_te).float().mean().item()

        # THE key call for deep learning: one point per epoch, keyed by step.
        aiplatform.log_time_series_metrics(
            {"train_loss": loss.item(), "val_loss": val_loss, "val_acc": val_acc},
            step=epoch,
        )

    # final scalars as SUMMARY metrics (these show up in the comparison DataFrame;
    # the per-epoch curves above do NOT — they live in TensorBoard)
    aiplatform.log_metrics({"final_val_acc": val_acc, "final_val_loss": val_loss})

This example makes Core Idea 3 tangible. The per-epoch curves went to log_time_series_metrics (stored in TensorBoard, viewable as smooth loss/accuracy curves in the TensorBoard UI). The final numbers went to log_metrics (stored in the metadata graph, and these are what appear as columns when you compare runs). If you had logged the epoch curves as summary metrics, each epoch would have overwritten the last and you’d see only the final value with no curve — and if you’d logged the final accuracy only as a time-series point, it would never appear in your comparison table. Right data, right store.

To sweep hidden-layer size, wrap it in a loop with distinct run names:

for hidden in [8, 16, 32]:
    model = MLP(hidden); opt = torch.optim.Adam(model.parameters(), lr=0.05)
    with aiplatform.start_run(f"mlp-h{hidden}-adam"):
        aiplatform.log_params({**hparams, "hidden": hidden})
        # ...training loop with log_time_series_metrics...
        aiplatform.log_metrics({"final_val_acc": val_acc})

Three runs, one experiment, ready to compare.

Worked Example D — tracking from inside a Vertex custom training job

Notebook tracking is where you explore; job tracking is where you commit. The moment a config looks like a production candidate, you promote it to a Vertex custom training job so the run is reproducible and runs on managed compute instead of your laptop. The tracking calls are identical — the difference is where the code executes and how the experiment gets attached.

You write an ordinary training script (task.py) that takes the experiment and run names as arguments and calls the same log_params / log_metrics you already know. This mirrors Google’s own get_started_with_vertex_experiments custom-job pattern:

# custom/trainer/task.py
import argparse, os
import google.cloud.aiplatform as aiplatform

parser = argparse.ArgumentParser()
parser.add_argument("--experiment", required=True)
parser.add_argument("--run", required=True)
parser.add_argument("--epochs", type=int, default=10)
parser.add_argument("--dataset-uri", required=True)
parser.add_argument("--model-dir", default=os.getenv("AIP_MODEL_DIR"))
args = parser.parse_args()

# Attach this job's execution to the experiment run
aiplatform.init(experiment=args.experiment)
aiplatform.start_run(args.run)

with aiplatform.start_execution(
        schema_title="system.ContainerExecution",
        display_name="training") as execution:

    # record the dataset we consumed (lineage input)
    dataset_artifact = aiplatform.Artifact.create(
        schema_title="system.Dataset", display_name="iris", uri=args.dataset_uri)
    execution.assign_input_artifacts([dataset_artifact])

    aiplatform.log_params({"epochs": args.epochs})
    # ... your real training here; write the model to args.model_dir ...
    model_artifact = aiplatform.Artifact.create(
        schema_title="system.Model", display_name="iris-model", uri=args.model_dir)
    execution.assign_output_artifacts([model_artifact])

    # stash a clickable lineage link as a metric so it shows in the comparison table
    aiplatform.log_metrics(
        {"lineage": execution.get_output_artifacts()[0].lineage_console_uri})

aiplatform.end_run()

Then, from your notebook or CI, you package that script into a CustomJob and run it against the experiment:

from google.cloud import aiplatform

aiplatform.init(project="your-project-id", location="us-central1",
                staging_bucket="gs://your-bucket")

job = aiplatform.CustomJob.from_local_script(
    display_name="iris-training",
    script_path="custom/trainer/task.py",
    container_uri="us-docker.pkg.dev/vertex-ai/training/scikit-learn-cpu.1-0:latest",
    requirements=["gcsfs"],
    machine_type="n1-standard-4",
)

job.run(
    args=["--experiment=iris-jobs", "--run=run-1",
          "--dataset-uri=gs://your-bucket/iris/iris.csv",
          f"--model-dir=gs://your-bucket/models/run-1"],
    service_account="your-sa@your-project.iam.gserviceaccount.com",
    sync=True,
)

Three things an experienced eye should catch. The experiment attaches two ways here — the script calls init(experiment=...) and you pass --experiment as an arg; both point at the same name, so the run the script opens is the run that shows up. The container_uri is a Vertex prebuilt training container (scikit-learn, TensorFlow, PyTorch, XGBoost are all available); use a custom container if your deps are exotic. And the experiment must have a TensorBoard instance if the script logs time-series metrics — a plain-metrics job like this one doesn’t need one. After it finishes, get_experiment_df("iris-jobs") shows the run exactly as if you’d trained locally, plus a clickable metric.lineage link into the metadata graph.

Worked Example E — multi-step lineage you can actually audit

Lineage is the feature that turns “trust me, this model is fine” into “here is the graph.” In a regulated domain — payments, lending, health — you will eventually be asked which exact dataset and which preprocessing produced this deployed model, and the answer needs to be a traversal, not an archaeology dig. This example builds a two-step lineage (preprocess → train) where each step is an execution that consumes and produces typed artifacts, following Google’s build_model_experimentation_lineage_with_prebuild_code pattern.

from google.cloud import aiplatform
from json import dumps

aiplatform.init(project="your-project-id", location="us-central1",
                experiment="lineage-demo", staging_bucket="gs://your-bucket")
aiplatform.start_run("run-1")

# The raw dataset is the root of the lineage graph
raw_dataset = aiplatform.Artifact.create(
    schema_title="system.Dataset", display_name="raw-iris",
    uri="gs://your-bucket/iris/raw.csv")

# --- Step 1: preprocessing execution ---
with aiplatform.start_execution(
        schema_title="system.ContainerExecution",
        display_name="preprocess") as exc:
    exc.assign_input_artifacts([raw_dataset])           # consumes raw
    aiplatform.log_params({"delimiter": ",", "index_col": 0})

    # ... read raw, transform, write processed.csv to GCS ...
    aiplatform.log_metrics({"n_records": 150, "n_columns": 5})

    processed_dataset = aiplatform.Artifact.create(     # produces processed
        schema_title="system.Dataset", display_name="processed-iris",
        uri="gs://your-bucket/iris/processed.csv")
    exc.assign_output_artifacts([processed_dataset])

# --- Step 2: training execution ---
with aiplatform.start_execution(
        schema_title="system.ContainerExecution",
        display_name="train") as exc:
    exc.assign_input_artifacts([processed_dataset])     # consumes processed
    aiplatform.log_params({"target": "species", "test_size": 0.2, "random_state": 7})
    # record the pipeline shape itself as a param — nice audit touch
    aiplatform.log_params({"pipeline_steps": dumps({"scaler": "StandardScaler",
                                                    "model": "GaussianNB"})})

    # ... train, evaluate ...
    aiplatform.log_metrics({"accuracy": 0.97, "f1_macro": 0.96})
    aiplatform.log_classification_metrics(
        labels=["setosa", "versicolor", "virginica"],
        matrix=[[10, 0, 0], [0, 9, 1], [0, 0, 10]],
        display_name="confusion-matrix")

    # upload the trained model to the Model Registry and record it as output
    model = aiplatform.Model.upload(
        display_name="iris-nb",
        artifact_uri="gs://your-bucket/models/iris-nb",
        serving_container_image_uri=(
            "us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest"))
    exc.assign_output_artifacts([model])

aiplatform.end_run()

# The payoff: a clickable graph, raw -> preprocess -> processed -> train -> model
print("Lineage:", exc.get_output_artifacts()[0].lineage_console_uri)

What you’ve built is a connected graph: raw-iris → preprocess → processed-iris → train → iris-nb model, with params and metrics hanging off each execution. Because the second step’s input is the first step’s output (the same processed_dataset artifact object), the edges connect and the graph is traversable. This is Core Idea 2 made real — and it’s the difference between an experiment log and an audit trail. Note the model was registered via Model.upload(), so the lineage stitches straight into the Model Registry: from a deployed model you can walk backward to the raw CSV.

Worked Example F — Pipelines + Experiments: comparing many pipeline runs

Once your workflow stabilizes (preprocess → train → evaluate as fixed steps) and you’re retraining often, you graduate from plain runs to pipeline runs. An experiment can hold n pipeline runs alongside n plain runs, and comparing pipeline runs is how you sweep configurations reproducibly — each run is a compiled, cached, re-runnable DAG, not a notebook you hope still works. This follows Google’s comparing_pipeline_runs notebook.

Define the training step as a KFP component. Note it logs metrics through the component’s Output[Metrics] artifact (metrics.log_metric(...)), which Vertex maps into the experiment automatically — you do not call aiplatform.log_metrics inside a component:

import kfp.dsl as dsl
from kfp import compiler
from kfp.dsl import Metrics, Model, Output, component
from google.cloud import aiplatform as vertex_ai

@component(
    base_image="python:3.10",
    packages_to_install=["pandas", "scikit-learn", "xgboost"],
)
def custom_trainer(
    train_uri: str, label_uri: str,
    max_depth: int, learning_rate: float, boost_rounds: int, model_uri: str,
    metrics: Output[Metrics],          # <- pipeline metrics artifact
    model_metadata: Output[Model],     # <- pipeline model artifact
):
    import pandas as pd, xgboost as xgb
    from sklearn.metrics import accuracy_score
    from sklearn.model_selection import train_test_split

    # /gcs/ fuse path lets you read GCS as a local file
    train_path = train_uri.replace("gs://", "/gcs/")
    label_path = label_uri.replace("gs://", "/gcs/")

    data = pd.read_csv(train_path).values
    labels = pd.read_csv(label_path).values.ravel()
    Xtr, Xte, ytr, yte = train_test_split(data, labels, test_size=0.2, random_state=7)

    booster = xgb.train({"max_depth": max_depth, "eta": learning_rate},
                        xgb.DMatrix(Xtr, label=ytr), num_boost_round=boost_rounds)
    preds = [round(v) for v in booster.predict(xgb.DMatrix(Xte))]
    metrics.log_metric("accuracy", accuracy_score(yte, preds))   # -> experiment
    model_metadata.uri = model_uri

@dsl.pipeline(name="iris-xgb-pipeline")
def pipeline(train_uri: str, label_uri: str, max_depth: int,
             learning_rate: float, boost_rounds: int, model_uri: str):
    custom_trainer(train_uri=train_uri, label_uri=label_uri, max_depth=max_depth,
                   learning_rate=learning_rate, boost_rounds=boost_rounds,
                   model_uri=model_uri)

compiler.Compiler().compile(pipeline_func=pipeline, package_path="pipeline.json")

Now sweep hyperparameters by submitting one PipelineJob per config, each attached to the same experiment via job.submit(experiment=...):

EXPERIMENT_NAME = "iris-xgb-sweep"
vertex_ai.init(project="your-project-id", location="us-central1",
               staging_bucket="gs://your-bucket")

runs = [
    {"max_depth": 4, "learning_rate": 0.2, "boost_rounds": 10},
    {"max_depth": 5, "learning_rate": 0.3, "boost_rounds": 20},
    {"max_depth": 3, "learning_rate": 0.1, "boost_rounds": 30},
    {"max_depth": 6, "learning_rate": 0.5, "boost_rounds": 40},
]

for i, run in enumerate(runs):
    job = vertex_ai.PipelineJob(
        display_name=f"{EXPERIMENT_NAME}-run-{i}",
        template_path="pipeline.json",
        pipeline_root="gs://your-bucket/pipelines",
        parameter_values={
            "train_uri": "gs://your-bucket/iris/iris_data.csv",
            "label_uri": "gs://your-bucket/iris/iris_target.csv",
            "model_uri": "gs://your-bucket/model",
            **run,
        },
    )
    job.submit(experiment=EXPERIMENT_NAME)   # attaches the pipeline run to the experiment

Pipeline runs are asynchronous, so the comparison DataFrame carries a state column — you poll it until everything is COMPLETE, then compare exactly as with plain runs:

import time

while True:
    df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)
    if all(state == "COMPLETE" for state in df.state):
        break
    if any(state == "FAILED" for state in df.state):
        print("At least one pipeline run failed"); break
    print("Still running..."); time.sleep(60)

df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)   # param.* + metric.* + state
best = df.sort_values("metric.accuracy", ascending=False).iloc[0]

# Jump from a run row straight to the Pipelines UI for that DAG:
job = vertex_ai.PipelineJob.get(df.run_name[0])
print("Pipeline run UI:", job._dashboard_uri())

The mental shift: with plain runs you call the logging APIs; with pipeline runs the component’s output artifacts carry the metrics and Vertex wires them into the experiment. Plain runs are lighter for iterating on a single model; pipeline runs bring caching (unchanged steps don’t re-run), step-level lineage, and repeatability — which is why they’re the right tool once you’re retraining on a schedule (Judgment Call 8).

Comparing experiment runs

There are two ways: the console (point-and-click, good for eyeballing) and the SDK (programmatic, good for anything repeatable). The SDK path is the one that matters for an experienced practitioner, and it’s a single call.

import aiplatform  # (from google.cloud import aiplatform)

df = aiplatform.get_experiment_df("iris-pytorch")

get_experiment_df() returns a pandas DataFrame with one row per run and columns named by convention: run_name, experiment_name, state, param.* for every logged parameter, and metric.* for every summary metric. (Note the flattening: a param called hidden becomes column param.hidden; a metric final_val_acc becomes metric.final_val_acc.) A common idiom from Google’s own notebooks is to call it without arguments and filter by name, which is handy when you’re iterating:

df = aiplatform.get_experiment_df()
df = df[df.experiment_name == "iris-pytorch"]
print(df.T)   # transpose: runs become columns, easy to read a few at a time

From there it’s just pandas — which is the quiet strength of the DataFrame interface. Rank, filter, and slice however you like:

# Best run by validation accuracy
best = df.sort_values("metric.final_val_acc", ascending=False).iloc[0]
print(best[["run_name", "param.hidden", "metric.final_val_acc"]])

# Only the runs above a threshold
strong = df[df["metric.final_val_acc"] > 0.95]

# Compare two specific configs side by side
df[df.run_name.isin(["mlp-h8-adam", "mlp-h32-adam"])].T

# Cross-experiment comparison: pull two experiments and concatenate
import pandas as pd
sk = aiplatform.get_experiment_df("iris-sklearn")
pt = aiplatform.get_experiment_df("iris-pytorch")
combined = pd.concat([sk, pt], ignore_index=True)
combined.sort_values("metric.accuracy", ascending=False)  # note: differing metric cols

That last snippet exposes a real gotcha (bite #2): scikit-learn runs logged metric.accuracy and PyTorch runs logged metric.final_val_acc, so the concatenated frame has both columns, half-populated. This is exactly why consistent metric names across runs matter — the comparison is a column join, and mismatched keys fracture it.

Because it’s a DataFrame, you also get the rich comparison visuals the console doesn’t give you natively — you just draw them yourself. The idiom Google uses in its own comparing_local_trained_models notebook is a parallel-coordinates plot, which is the single best view for seeing how hyperparameters relate to outcomes across many runs at once:

import matplotlib.pyplot as plt
import pandas as pd

df = aiplatform.get_experiment_df("iris-pytorch").reset_index(drop=True)

plt.rcParams["figure.figsize"] = [15, 5]
ax = pd.plotting.parallel_coordinates(
    df,
    "run_name",
    cols=["param.hidden", "param.lr", "param.epochs",
          "metric.final_val_acc", "metric.final_val_loss"],
)
ax.set_yscale("symlog")   # params and metrics live on very different scales
ax.legend(bbox_to_anchor=(1.0, 0.5))
plt.show()

Each line is a run; each vertical axis a param or metric. Lines that stay high on metric.final_val_acc while sitting at a particular param.hidden tell you which region of the space is working. This ~10-line plot is the honest counter to the “no native comparison UI” downside (§11): you don’t get it for free, but you’re one pandas call away from something as useful as what the paid tools ship.

For the curves (time-series), the DataFrame won’t help — those live in TensorBoard. Open the experiment in the Google Cloud console under Experiments and click Open TensorBoard to overlay loss/accuracy curves across runs, or query the TensorBoard time-series API directly if you need the raw points.

Pulling experiment data into Streamlit / marimo / any app

Because get_experiment_df() returns an ordinary DataFrame, wiring your experiment history into a dashboard is trivial — there’s no special export format, no bespoke client. The DataFrame is the integration point. This is genuinely one of the tool’s nicer properties for a data-science team that wants a shared “which model won” view.

Streamlit. A leaderboard app in ~20 lines:

# app.py  ->  run with:  streamlit run app.py
import streamlit as st
import pandas as pd
from google.cloud import aiplatform

st.title("Iris Experiment Leaderboard")

EXPERIMENT = st.text_input("Experiment name", "iris-pytorch")

@st.cache_data(ttl=300)   # cache 5 min so every interaction doesn't re-hit the API
def load_runs(name: str) -> pd.DataFrame:
    aiplatform.init(project="your-project-id", location="us-central1")
    df = aiplatform.get_experiment_df()
    return df[df.experiment_name == name]

df = load_runs(EXPERIMENT)

metric_cols = [c for c in df.columns if c.startswith("metric.")]
sort_by = st.selectbox("Rank by", metric_cols)
st.dataframe(df.sort_values(sort_by, ascending=False)
               [["run_name"] + metric_cols])
st.bar_chart(df.set_index("run_name")[sort_by])

Two practical notes. Wrap the fetch in @st.cache_data with a TTL — get_experiment_df is a network call, and without caching every widget interaction re-queries the API. And handle auth the way any GCP client does: Application Default Credentials on your machine (gcloud auth application-default login) or a service account when deployed (e.g. on Cloud Run).

marimo. Same idea, reactive style — marimo re-runs dependent cells when inputs change, so a dropdown drives the table automatically:

# a marimo notebook cell
import marimo as mo
import pandas as pd
from google.cloud import aiplatform

aiplatform.init(project="your-project-id", location="us-central1")
df = aiplatform.get_experiment_df()
df = df[df.experiment_name == "iris-pytorch"]

metric = mo.ui.dropdown(
    options=[c for c in df.columns if c.startswith("metric.")],
    label="Rank by",
)
metric

# a second cell — re-runs automatically when `metric` changes
mo.ui.table(df.sort_values(metric.value, ascending=False)) if metric.value else None

The general pattern for any app (Dash, Panel, a Jupyter widget, a scheduled report): init() once, get_experiment_df() to a DataFrame, filter to the experiment, then treat it as normal tabular data. The only things to get right are caching (it’s a network call, don’t hammer it), auth (ADC locally, service account in production), and column-name awareness (param.* / metric.* prefixes, and the fracturing-on-mismatched-keys gotcha). If you need the per-epoch curves in a custom app rather than TensorBoard, that’s a separate and clunkier path through the TensorBoard time-series API — worth knowing it exists, and worth knowing it’s the sharp edge of this otherwise smooth integration story.

The whole tool, in one breath

Create an experiment, start runs, log params and metrics (manually or via one-line autolog), send per-step curves to TensorBoard via log_time_series_metrics, wire multi-step lineage into the metadata graph with executions and artifacts, promote candidates to reproducible custom jobs and pipeline runs, then compare with get_experiment_df() in a notebook, the console, or your own app. Everything below is about doing this well and knowing what will bite you.

5. The Mental Model

Three ideas. Internalize these and you can predict how Vertex AI Experiments behaves in situations this document never mentions.

Core Idea 1: Experiments is a bookkeeping layer over a graph database — it records, it does not compute.

The service does not train your model, does not run your code, does not own any compute. It provides an API you call from inside your own training, and it writes what you tell it into Vertex ML Metadata. This is the master key.

What it predicts:

It works from anywhere — a laptop, a Workbench notebook, a Vertex custom job, a pipeline step. Experiments supports development using custom training, Workbench notebooks, and all Python ML frameworks. Because it’s just an API writing to a store, the compute location is irrelevant.
Nothing is captured that you don’t log (unless autolog is on). If you forget to log the random seed, the seed is gone. The tool has no magic view into your process; it’s a scribe, and it only writes what’s dictated.
It’s serverless and cheap precisely because it isn’t running anything — you pay only for storage. No idle server, no per-seat cost, because there’s no “it” to keep running.
Framework-agnostic by construction — because it’s based on ML Metadata, it can track experiments from any framework. The store doesn’t care whether the numbers came from TensorFlow or a hand-rolled loop.

Core Idea 2: Everything you log becomes a typed node or edge in a lineage graph.

An Experiment is a Context. A Run is a Context. A model is an Artifact. A training step is an Execution that consumes and produces Artifacts. These aren’t loose log lines — they’re nodes and edges in a graph that spans the whole platform.

What it predicts:

Lineage is queryable and automatic once you link artifacts — you can ask “what produced this model?” and walk the graph backward to the dataset. This is why the tool is credible for audit and compliance.
The same graph is shared with Pipelines, Model Registry, and Model Monitoring. A pipeline run and a notebook run both write to the same metadata store, so they show up in the same lineage. Tracking isn’t siloed from the rest of your MLOps.
Artifacts are typed (system.Model, system.Dataset), so the graph is semantically meaningful, not just a blob store. Tools can reason about “the models in this experiment” specifically.
Metadata accumulates and persists independent of any run’s lifetime — which is powerful (durable history) and a liability (it bloats and costs money if you never clean up).

Core Idea 3: Scalar metadata and time-series metrics live in two different stores, joined only by a reference.

This is the split that causes the most day-to-day confusion, so it earns its own core idea. Parameters and summary metrics live in Vertex ML Metadata. Training curves (time-series) live in Vertex AI TensorBoard. Experiments stores only a reference to the TensorBoard resource.

What it predicts:

log_time_series_metrics needs a TensorBoard instance; log_metrics does not. The former writes to a store that must exist first. This is why beginners hit “no TensorBoard” errors on curves but never on final metrics.
The two have different pricing and lifecycle. Metadata is cheap storage; TensorBoard was historically expensive and is billed separately (see Downsides). Deleting a run may leave orphaned TensorBoard data unless you delete the backing TensorBoard run too.
The console and TensorBoard show different views — Experiments is about comparing high-level metadata across runs (params, metrics, artifacts), while TensorBoard focuses on real-time training visualization like loss curves and histograms. As one writer put it, TensorBoard shows the journey of a single training run; Experiments shows the destination of many runs side by side.
The TensorBoard binding is sticky. Once experiment_tensorboard is set in a run, it can’t be changed; and the assign_backing_tensorboard method can only be called once, because there’s no mechanism to transfer TensorBoard experiment entries to another instance.

6. The Architecture in Plain English

Walk through what actually happens when you track a run.

You call aiplatform.init(experiment="x"). The SDK talks to the Vertex AI Metadata Service and ensures a Context of schema system.Experiment named x exists. It also resolves a default TensorBoard if one isn’t provided, via an internal _get_or_create_default_tensorboard() step. So after init, two things exist in the cloud: an experiment context in the metadata graph, and a TensorBoard instance (yours or a freshly-minted default).

You call start_run("attempt-1"). The SDK creates a Context of schema system.ExperimentRun (in current versions; legacy versions used a system.Run Execution for backward compatibility) and links it under the experiment context. It also creates a google.VertexTensorboardRun Artifact to link the metadata context to the TensorBoard resource — that’s the “reference” from Core Idea 3 made concrete.

You call log_params({...}). The SDK writes these as properties on the run context in the metadata store. Instant, cheap, durable.

You call log_metrics({...}). Summary metrics get written as properties too — single-value scalars stored next to (a reference to) the time-series metrics.

You call log_time_series_metrics({...}, step=n). Now the data leaves the metadata store entirely and goes to the TensorBoard backend, keyed by step. Vertex TensorBoard uses a database to store its logs rather than event files on disk, which is what makes the UI stay responsive as run counts grow.

You wrap training in start_execution(...) and assign artifacts. The SDK creates an Execution node and draws edges: input artifacts point into it, output artifacts point out of it. Those edges are the lineage graph. If you’d instead turned on autolog(), MLflow’s autologging plugin — a custom tracking plugin that configures MLflow’s tracking URI to a vertex-mlflow-plugin:// scheme — intercepts your framework’s training calls and issues the equivalent log operations for you.

You call get_experiment_df("x"). The SDK retrieves all runs by listing contexts with the system.ExperimentRun schema under the experiment and flattens their params and summary metrics into a pandas DataFrame — param.* columns for inputs, metric.* columns for outputs. Time-series data is not in this frame; for curves you go to TensorBoard.

Where does state live? Params, summary metrics, artifacts, executions, lineage → Vertex ML Metadata. Time-series curves → Vertex AI TensorBoard. Model files, datasets → wherever you put them (usually GCS), with the metadata graph holding typed references. The tracking service itself is stateless glue; it’s an SDK talking to two managed stores over the network. Access to all of it is governed by GCP IAM — permissions are controlled via IAM, and the whole thing integrates with Cloud Audit Logs for governance.

7. The Things That Bite You

Each of these connects back to the mental model. When one bites, name the core idea it violated.

Time-series logging fails without a TensorBoard instance. You expect log_time_series_metrics to “just work” like log_metrics. It doesn’t — it needs a backing TensorBoard (Core Idea 3). In notebooks since 1.25+ a default is auto-created, but in stripped-down job environments or older SDKs you’ll get an error. Fix: create one explicitly with aiplatform.Tensorboard.create(...) and pass it as experiment_tensorboard. A custom training job’s experiment must have a TensorBoard instance.
Inconsistent parameter names silently fracture your comparisons. Because comparison is a DataFrame join on key names (Core Idea 1: it records exactly what you dictate), if one run logs “lr” and another logs “learning_rate”, your comparison DataFrames get separate columns and comparison gets harder. Nothing errors — you just get a sparse, misaligned table. Fix: standardize key names across the whole team, ideally in a shared constants module.
The default run name is opaque. Autolog auto-names runs {framework}-{timestamp}-{uid}. Six weeks later you’re staring at tensorflow-2023-01-04-16-09-20-86a88 with no idea what it was. Fix: pass your own start_run("baseline-tfidf-lr") before autolog, or adopt a naming convention.
The TensorBoard binding is one-shot and non-transferable. assign_backing_tensorboard can only be called once, and there’s no mechanism to move TensorBoard entries to another instance. Bind an experiment to the wrong (e.g. expensive, or wrong-region) TensorBoard and you’re re-creating the experiment to fix it. Fix: decide your TensorBoard instance deliberately before first log.
Deleting a run can orphan TensorBoard data. Because the curve data lives in a separate store (Core Idea 3), deleting the run context doesn’t necessarily reclaim the TensorBoard storage. The delete API exposes a delete_backing_tensorboard_run flag governing whether to delete the backing TensorBoard run that stores the time-series metrics. Forget it and you pay for ghost data.
resume=True vs a fresh run. When the optional resume parameter is TRUE, the previously started run resumes; when not specified it defaults to FALSE and a new run is created. Re-run a cell expecting to append and you may instead spawn a duplicate run — or worse, silently overwrite a step, since if a step already exists for a metric key it’s overwritten.
Autolog version-compatibility hell. Because autolog rides on MLflow (Core Idea 1 — it’s glue over another library), version mismatches surface as cryptic failures. Compatible package versions are needed for autolog to work well; you have to read the docs carefully to get an environment where everything is compatible. For TensorFlow specifically, ensure protobuf < 4.0 to avoid conflicts.
No native rich comparison charts. People coming from W&B expect parallel-coordinates plots and scatter matrices in the UI. Vertex Experiments only records artifacts, parameters, and metrics; if you want comparison graphs like MLflow offers natively, you build them yourself in code. Fix: pull the DataFrame and plot it, or lean on TensorBoard for curve overlays.
Time-series and summary metrics look interchangeable but aren’t. Logging your final accuracy as a time-series point (or your per-epoch loss as a summary metric) puts data in the wrong store and the wrong view. It won’t error; it’ll just be in the place you don’t look. Keep the “curve vs. final number” distinction crisp.

8. The Judgment Calls

The decisions experienced practitioners actually weigh.

1. Autolog vs. manual logging. Autolog is one line and captures the standard params/metrics for supported frameworks (Fastai, Gluon, Keras, LightGBM, PyTorch Lightning, scikit-learn, Spark, Statsmodels, XGBoost). Manual gives you exactly what you choose — custom metrics, business KPIs, non-standard params. Experienced choice: autolog as the floor, then add manual log_params/log_metrics for the things autolog can’t know (feature-set version, data snapshot ID, the business metric you actually care about). The signal to go fully manual: an unsupported framework, or metrics autolog doesn’t understand.

2. Do you even need TensorBoard? TensorBoard is separate, separately billed, and sticky. If you only care about final metrics and comparison tables, you can live entirely in metadata and never attach a real TensorBoard. Experienced choice: attach TensorBoard only when you genuinely need per-step curves (deep nets trained over many epochs, debugging divergence). For a scikit-learn model that trains in one shot, curves are pointless overhead. The signal to attach: you train over epochs and need to see the dynamics.

3. Experiment granularity — how much is one experiment? Too coarse (one experiment for the whole team forever) and comparison tables become unreadable. Too fine (a new experiment per idea) and you lose cross-idea comparability. Experienced choice: one experiment per problem framing (e.g. fraud-detection-v2), with runs for every config; start a new experiment when the problem itself changes (new label definition, new dataset generation). The signal to split: you find yourself unable to meaningfully compare two runs because they’re solving different problems.

4. Vertex Experiments vs. MLflow vs. W&B. Vertex is serverless with minimal storage cost and no always-on infrastructure; MLflow is free and open-source but you host it; W&B is the most polished UX but charges per seat, roughly $50/user/month for teams in 2026. Experienced choice: if you’re already all-in on Google Cloud and want IAM-governed, lineage-integrated tracking without ops burden, Vertex Experiments is the path of least resistance. If you need the best comparison UI or you’re multi-cloud, W&B earns its price. If you need full data control or air-gapped operation, self-hosted MLflow wins. The signal for Vertex: your compute, data, and pipelines already live in GCP.

5. What to log — minimalist vs. maximalist. Experienced choice: maximalist, because storage is cheap but re-running experiments is expensive; log not just obvious hyperparameters but also preprocessing choices, feature-selection decisions, and random seeds. The one discipline on top: a consistent run_type tag — values like “baseline”, “production-candidate”, or “ablation-study” so you can quickly filter to the runs you care about.

6. Notebook-driven vs. job-driven tracking. Tracking from a notebook is fast for exploration; tracking from a Vertex custom job is reproducible and scalable. Experienced choice: explore in notebooks, but the moment a config looks like a “production candidate,” promote it to a tracked training job. The sooner you formalize experiments into pipelines, the easier and faster it is to move them to production. The signal to promote: you’re about to compare candidates that might actually ship.

7. Lineage now vs. lineage later. Manually wiring start_execution + assign_input/output_artifacts is extra code you can skip early. Experienced choice: skip it during pure exploration; add it the moment the model touches anything regulated or shared. In fintech/payments, lineage isn’t optional — you need to prove which dataset and preprocessing produced a deployed model. The signal to add lineage: someone outside the team will ask “where did this come from.”

8. Pipeline runs vs. plain runs in an experiment. An experiment can contain n experiment runs in addition to n pipeline runs. Plain runs are lighter; pipeline runs bring orchestration, caching, and step-level lineage. Experienced choice: plain runs while iterating on the model; pipeline runs once the workflow (preprocess → train → eval) stabilizes and you want retraining to be repeatable. The signal to pipeline-ize: you’re retraining frequently and preprocessing is non-trivial.

9. Storage lifecycle — keep everything vs. prune. Metadata and TensorBoard data accumulate and cost money (Core Idea 2). Experienced choice: keep metadata generously (it’s cheap and it’s your history), but clean up TensorBoard instances you no longer need, since each incurs cost — delete old ones after extracting the insights. The signal to prune: TensorBoard storage line items you can’t attribute to an active project.

9. The Commands/APIs That Actually Matter

Grouped by task. This is the 20% you’ll use constantly.

Initialize and create

aiplatform.init(project=, location=, experiment=, experiment_tensorboard=) — the entry point; creates/selects the experiment and resolves TensorBoard.
aiplatform.Tensorboard.create(display_name=, project=, location=) — explicit TensorBoard instance when you don’t want the default.

Run lifecycle

aiplatform.start_run("name") — begin a run; use as a with block. Add resume=True to append to an existing run.
aiplatform.end_run() — close the run (automatic if you used with).

Logging

aiplatform.log_params({...}) — input knobs. Cheap, no TensorBoard needed.
aiplatform.log_metrics({...}) — final scalar outputs (summary metrics).
aiplatform.log_time_series_metrics({...}, step=) — per-step curves; needs TensorBoard.
aiplatform.log_classification_metrics(...) — confusion matrices and ROC curves.
aiplatform.autolog() — one-line automatic param/metric capture for supported frameworks. Remember: autologging only supports parameter and metric logging.

Lineage

aiplatform.start_execution(schema_title=, display_name=) — record a step.
execution.assign_input_artifacts([...]) / assign_output_artifacts([...]) — draw lineage edges.

Analyze and compare

aiplatform.get_experiment_df("experiment-name") — all runs as a pandas DataFrame (param.*, metric.* columns). Then standard pandas: .sort_values("metric.accuracy"), boolean filtering, thresholding.
Console: the Experiments section for the comparison table and the Open TensorBoard button for curves.

Job integration

job.run(experiment=, experiment_run=) — attach a Vertex custom training job to an experiment. Inside the container, the same log_params/log_metrics calls flow to the run.

10. How It Breaks

For each: symptom → root cause → diagnose → fix.

“TensorBoard instance required” / time-series calls fail. Root cause: no backing TensorBoard (Core Idea 3). Diagnose: check whether init ran with experiment_tensorboard and whether a default was created (older SDK or minimal job env may not). Fix: Tensorboard.create() and pass it, or upgrade the SDK above 1.25.

Comparison DataFrame has scattered, half-empty columns. Root cause: inconsistent param/metric key names across runs (bite #2). Diagnose: inspect df.columns — you’ll see both param.lr and param.learning_rate. Fix: rename historically if needed; enforce a shared key vocabulary going forward.

Duplicate or overwritten runs after re-running a cell. Root cause: resume semantics (bite #6) — default FALSE spawns a new run; a repeated step key overwrites. Diagnose: list runs via get_experiment_df and look for near-identical duplicates. Fix: be explicit with resume=True when you mean to append, and manage step indices deliberately.

Autolog throws cryptic errors or logs nothing. Root cause: MLflow version/framework incompatibility (bite #7). Diagnose: check google-cloud-aiplatform, mlflow, and framework versions; for TF check protobuf. Fix: pin compatible versions; use SDK > 1.24.1 and protobuf < 4.0 for TensorFlow.

Surprise TensorBoard bill. Root cause: auto-created and/or orphaned TensorBoard instances and data (Core Idea 2 + bite #5). Diagnose: check the TensorBoard instances tab in the console for instances you don’t recognize. Fix: delete unused instances; set delete_backing_tensorboard_run=True when deleting runs.

Permission denied reading someone’s experiment. Root cause: IAM, not a tool bug — access is governed by GCP IAM. Diagnose: check the caller’s roles on the project/resource. Fix: grant the appropriate Vertex AI role; note that within a project, per-experiment access restriction to time-series logs requires separate TensorBoard instances.

General debugging workflow. (1) Confirm init project/location match where you’re looking in the console. (2) get_experiment_df to see what actually got recorded. (3) Check the TensorBoard instance exists and is bound, if curves are missing. (4) Verify SDK/MLflow/framework versions if autolog misbehaves. (5) Check IAM if it’s an access issue. (6) Check the TensorBoard instances tab if it’s a cost issue.

11. The Downsides / Disadvantages

Honest accounting. These are structural, not fixable with better config.

1. The comparison UI is thin — you build the good charts yourself. Vertex Experiments records artifacts, parameters, and metrics, but for comparison graphs like MLflow offers natively, you build them via code. Where it comes from: Experiments is a bookkeeping layer (Core Idea 1), not a visualization product — there’s no sophisticated multi-run comparison UI built into Vertex beyond TensorBoard and DataFrame queries. What it costs: engineering time writing plotting code that W&B users get for free, and a worse day-to-day exploration experience. Dealbreaker when: your team’s core loop is visual hyperparameter exploration and you’d feel the absence of parallel-coordinates plots daily. Livable when: you mostly rank runs by a metric and occasionally eyeball a curve.

2. The two-store split (metadata vs. TensorBoard) is permanent cognitive overhead. Where it comes from: Core Idea 3 — time-series lives in a different system joined by a reference. What it costs: every engineer must internalize which call goes where, TensorBoard must be provisioned and bound (stickily), and lifecycle/cost management is split across two systems. This never goes away with experience; it’s the shape of the tool.

3. TensorBoard’s cost history and separate billing. TensorBoard was originally billed at $300/month per active user before the model changed to $10/GiB/month for log storage — no more subscription, you pay for storage used. Where it comes from: TensorBoard is a separate managed resource, not part of the (nearly free) metadata store. What it costs: even at storage pricing, forgotten instances and orphaned data accrue silent charges; the pricing has shifted more than once, so cost models built on old assumptions rot. Dealbreaker when: you’d spin up many short-lived TensorBoard instances at scale without disciplined cleanup.

4. Lock-in to the Google Cloud graph. Where it comes from: Core Idea 2 — everything is a node in Vertex ML Metadata, addressed by GCP resource URIs and governed by GCP IAM. What it costs: migrating off Google Cloud means re-ingesting history into another system; there’s no clean export to MLflow/W&B. Migration between trackers means re-ingesting historical runs, retraining the team, and updating integrations — expensive, not insurmountable. Dealbreaker when: you anticipate leaving GCP or must stay portable.

5. Autolog inherits MLflow’s fragility without owning the fix. Where it comes from: Core Idea 1 — autolog is glue over MLflow (vertex-mlflow-plugin://). What it costs: version-compatibility work to get an environment where everything cooperates, and when it breaks you’re debugging across two abstraction layers you don’t control. It also only supports parameter and metric logging — no artifacts or richer capture through the autolog path.

6. It records only what you tell it (outside autolog). Where it comes from: Core Idea 1 — it’s a scribe. What it costs: reproducibility is only as good as your logging discipline. Forget the seed, the data snapshot, or a preprocessing flag and the run is not reproducible, and you won’t find out until you try to recreate it. Unlike W&B, which automatically records code version, all hyperparameters, system metrics, and even sample predictions, Vertex captures far less automatically.

7. Google Cloud naming and product churn. Where it comes from: it’s part of a fast-moving, frequently-rebranded platform (the docs now nest it under “Gemini Enterprise Agent Platform”). What it costs: URLs, product names, and console navigation shift under you; older tutorials reference paths and prices that no longer exist. Budget for the docs being a moving target.

8. Best experience assumes you’re already in GCP. Where it comes from: deep integration with Pipelines, Training, Model Registry, IAM. What it costs: from outside GCP, the value proposition collapses — you’re better off with a portable tool. This is a strength that is also a boundary: it’s a seamless platform if you’re fully in Google Cloud.

12. The Taste Test

What separates a clean setup from a cargo-culted one.

Experiment and run naming.

Bad: one experiment named test, runs named run1, run2, and a pile of autolog tensorflow-2023-...-86a88 names.
Good: fraud-detection-v2 as the experiment, runs named baseline-logreg, xgb-featureset-b-seed42, production-candidate-2026-07. You can read the intent months later.

What gets logged.

Bad: just the final accuracy.
Good: every hyperparameter, plus the dataset version, preprocessing flags, feature-set ID, random seed, and a run_type tag. The maximalist logs everything because it costs almost nothing to store metadata and you never know which parameter will matter.

Baseline discipline.

Bad: the first tracked run is a fancy tuned ensemble; there’s nothing to compare it against.
Good: run and log a simple baseline first, so every later run has a reference point for evaluating improvement.

Key-name consistency.

Bad: lr in some runs, learning_rate in others; comparisons fracture.
Good: a shared constants module defining canonical param/metric keys, imported everywhere.

TensorBoard usage.

Bad: a TensorBoard instance auto-created for a one-shot scikit-learn model that has no curves, then left running.
Good: TensorBoard attached only where per-step curves matter, and stale instances pruned on a schedule.

Lineage.

Bad: a deployed model with no recorded connection to the dataset or preprocessing that made it.
Good: production candidates carry full input/output artifact lineage, so “where did this come from” is one graph traversal away.

Notebook vs. job.

Bad: the shipped model was trained by re-running notebook cells nobody can reconstruct.
Good: exploration in notebooks, but ship-candidates trained via tracked Vertex jobs or pipeline runs for reproducibility.

13. Operating It in Production

The examples so far get you tracking. This section is the stuff nobody tells you until it hurts: who can see what, what it costs, how to get curves out programmatically, how to get off the platform, and how tracking changes when the thing you’re building is a GenAI/LLM system rather than a classifier.

Access control — IAM, not app-level permissions

Experiments has no permission model of its own; access is governed entirely by Google Cloud IAM at the project/resource level (Core Idea 2 — everything is a Google Cloud resource). The roles that matter:

roles/aiplatform.user — the everyday role. Lets a data scientist create experiments, start runs, log data, and read get_experiment_df. This is what most of your team gets.
roles/aiplatform.viewer — read-only. Good for stakeholders who should see the leaderboard but not write to it.
roles/storage.objectAdmin (or narrower) on the staging bucket — because artifacts, models, and pipeline roots live in GCS, tracking silently depends on bucket access. A run that can’t write its model to GCS will fail in a way that looks like a tracking bug but isn’t.
The training job’s service account needs the above too — remember Example D passed service_account=.... The job runs as that identity, so it (not you) needs aiplatform.user and bucket access. Mis-scoped service-account permissions are the single most common “why did my tracked job fail” cause.

Two governance realities worth internalizing. First, there’s no per-experiment ACL within a project — IAM grants are project-wide (or resource-wide), so you cannot easily say “Alice sees experiment A but not experiment B” without separating projects. For time-series specifically, restricting who sees which logs requires separate TensorBoard instances, since that’s the only access boundary available below the project. Second, everything is captured in Cloud Audit Logs — who created what, who read what — which is exactly why regulated teams tolerate the other rough edges: the governance story is real and it’s the platform’s, not something you bolt on.

What it actually costs — a worked example

Pricing has two very different components, and conflating them is how people get surprised.

Metadata (params, summary metrics, artifacts, lineage) is billed as plain metadata storage — on the order of a few dollars per GiB-month. In practice, for anything short of millions of runs, the metadata cost rounds to zero. Log generously; this is not where the money is.

TensorBoard (time-series curves) is the line item that bites. It was historically billed at $300/month per active user — a flat, painful subscription — before Google changed it to roughly $10 per GiB-month of log storage, i.e. you now pay for stored curve data, not per head. That change made Experiments genuinely cheap, but two traps remain: the default TensorBoard instance auto-created on init keeps existing (and storing) until you delete it, and deleting a run doesn’t reclaim its TensorBoard data unless you pass delete_backing_tensorboard_run=True.

A concrete back-of-envelope for a mid-size effort — say 500 runs, each logging 3 time-series metrics over 80 epochs:

Metadata: 500 runs × (a handful of params + summary metrics each) ≈ single-digit MB total → effectively free.
TensorBoard: 500 × 3 × 80 = 120,000 scalar points, plus overhead. Scalar time-series are tiny; realistically well under 1 GiB → a few dollars a month, if you clean up. Leave three abandoned default TensorBoard instances lying around from earlier experiments, each holding orphaned data, and you’re paying for all three indefinitely.

The discipline that keeps the bill near zero:

from google.cloud import aiplatform

# Audit what exists
for tb in aiplatform.Tensorboard.list():
    print(tb.display_name, tb.resource_name, tb.create_time)

# Delete a run AND its backing time-series data (don't orphan it)
exp = aiplatform.Experiment("iris-pytorch")
run = aiplatform.ExperimentRun("mlp-h8-adam", experiment="iris-pytorch")
run.delete(delete_backing_tensorboard_run=True)

# Delete a whole experiment when a project wraps
aiplatform.Experiment("old-experiment").delete()

# Delete a TensorBoard instance you no longer need
aiplatform.Tensorboard("projects/.../tensorboards/123").delete()

Rule of thumb: attach TensorBoard only when you need curves (deep learning over epochs), skip it entirely for one-shot models like a fitted scikit-learn pipeline, and prune stale instances on a schedule. Storage is cheap; forgotten instances are not.

Getting time-series data out programmatically

The comparison DataFrame gives you params and summary metrics but not the per-epoch curves (Core Idea 3 — they live in TensorBoard). When you need those points in a custom app rather than the TensorBoard UI — a bespoke Streamlit chart, an automated regression report — you read them back through the experiment run object:

from google.cloud import aiplatform

aiplatform.init(project="your-project-id", location="us-central1",
                experiment="iris-pytorch")

run = aiplatform.ExperimentRun("mlp-h16-adam", experiment="iris-pytorch")

# Pull the logged time-series back as a DataFrame of (step, metric) rows
ts = run.get_time_series_data_frame()
# columns include 'step' and one column per time-series metric key
print(ts[["step", "train_loss", "val_loss", "val_acc"]].head())

# now it's just pandas -> feed any plotting library
import matplotlib.pyplot as plt
ts.plot(x="step", y=["train_loss", "val_loss"])
plt.show()

Be honest with yourself about this path: it’s the sharp edge of an otherwise smooth integration story. Summary metrics flow out effortlessly through get_experiment_df; curves require per-run calls and a different shape of data. If your dashboard is fundamentally about training dynamics across many runs, TensorBoard’s own overlay UI is still the path of least resistance, and reaching for the read API is a deliberate choice you make when you need the raw points in your own pipeline.

Migration — getting in, and getting out

The lock-in is real (Downside #4), so know the escape hatches before you need them.

Coming from MLflow. Because Vertex autolog is MLflow autolog under the hood, code that already uses MLflow-supported frameworks needs almost no change — you call aiplatform.autolog() instead of mlflow.autolog() and point at a Google Cloud project. Historical MLflow runs, though, don’t import cleanly; there’s no first-class “load my old MLflow store into Vertex” button. The pragmatic move is to treat the cutover as a line in the sand: keep the old MLflow server read-only for history, start new work in Vertex, and if you truly need the old runs in one place, script a migration that reads the MLflow tracking store and replays each run as start_run + log_params + log_metrics.

Getting out of Vertex. Everything you logged is retrievable as data — get_experiment_df() for the tabular history, get_time_series_data_frame() per run for curves. A dozen lines of pandas will dump your entire experiment history to CSV/Parquet, which you can then re-ingest into MLflow or W&B. What does not travel is the lineage graph and the tight Model Registry / Pipelines integration — those are Google Cloud resources with no portable equivalent, so a move off GCP means rebuilding that connective tissue elsewhere. Budget for re-ingesting history and re-wiring integrations; it’s expensive but not a trap you can’t escape.

# Portable export: dump everything to Parquet before any migration
import pandas as pd
from google.cloud import aiplatform
aiplatform.init(project="your-project-id", location="us-central1")

frames = []
for name in ["iris-sklearn", "iris-pytorch", "iris-xgb-sweep"]:
    df = aiplatform.get_experiment_df(name)
    df["_experiment"] = name
    frames.append(df)
pd.concat(frames, ignore_index=True).to_parquet("experiment_history.parquet")

Tracking GenAI and LLM work

Classical experiment tracking assumes a training loop with numeric metrics. GenAI work often has neither — you’re iterating on prompts, retrieval configs, and models you didn’t train, and your “metrics” are eval scores from a judge model or a rubric. The good news: because Experiments only records what you dictate (Core Idea 1), it doesn’t care whether a “param” is a learning rate or a prompt template, or whether a “metric” came from accuracy_score or an LLM judge. You track a prompt-engineering iteration the same way you track a hyperparameter sweep:

from google.cloud import aiplatform

aiplatform.init(project="your-project-id", location="us-central1",
                experiment="rag-prompt-tuning")

variants = [
    {"prompt_template": "concise-v1", "model": "gemini-2.5-flash",
     "temperature": 0.2, "top_k_chunks": 3},
    {"prompt_template": "detailed-v2", "model": "gemini-2.5-pro",
     "temperature": 0.7, "top_k_chunks": 5},
]

for i, cfg in enumerate(variants):
    with aiplatform.start_run(f"{cfg['prompt_template']}-{cfg['model']}"):
        aiplatform.log_params(cfg)                 # prompt/model/retrieval config
        # ... run your eval set through the RAG chain, score with a judge ...
        aiplatform.log_metrics({
            "faithfulness": 0.88,                  # from Vertex GenAI evaluation
            "answer_relevance": 0.91,
            "groundedness": 0.85,
            "avg_latency_ms": 740,
            "cost_per_1k_queries_usd": 2.3,
        })

Two things worth knowing. Vertex has a Gen AI evaluation service that produces exactly these kinds of metrics (faithfulness, relevance, rubric scores, pairwise/AutoSxS comparisons), and its outputs are ordinary numbers you can pipe straight into log_metrics — so your prompt experiments compare in the same get_experiment_df leaderboard as everything else. And the params that matter shift: for GenAI, log the prompt template version, the model ID, temperature/top-k, retrieval settings, and cost/latency, because those — not weights — are the levers you’re actually pulling. This is where the industry is heading in 2026, and the tracking substrate handles it precisely because it was never about training in the first place — it’s about remembering what you tried.

14. Where to Go Deeper

Introduction to Vertex AI Experiments (official docs) — the canonical concept map (experiment, run, params, metrics, artifacts, lineage). Read first to anchor vocabulary. cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments
“Manually log data to an experiment run” (official docs) — the precise semantics of log_params, log_metrics, log_time_series_metrics, resume, and step. Read when you’re actually writing logging code.
GoogleCloudPlatform/vertex-ai-samples → notebooks/official/experiments — the runnable notebooks this guide is grounded in. Start with get_started_with_vertex_experiments (manual logging + lineage + custom job) and get_started_with_vertex_experiments_autologging (the one-line autolog path, sklearn/TF/PyTorch). The fastest way to see the API in motion.
comparing_local_trained_models.ipynb (same repo) — the run-comparison notebook; source of the parallel-coordinates idiom in §4. Read when you want to build comparison visuals the console doesn’t give you.
comparing_pipeline_runs.ipynb (same repo) — the KFP-component → PipelineJob.submit(experiment=...) → poll-state → compare pattern from Worked Example F. Read when you’re graduating from notebook runs to reproducible pipeline sweeps.
build_model_experimentation_lineage_with_prebuild_code.ipynb (same repo) — the multi-step preprocess→train lineage pattern from Worked Example E, ending in lineage_console_uri and a Model.upload. Read when lineage/audit is the point.
GoogleCloudPlatform/mlops-with-vertex-ai — end-to-end MLOps with TFX, Pipelines, TensorBoard, and ML Metadata together. Read when you’re ready to move from notebook tracking to production pipelines.
Vertex AI TensorBoard introduction (official docs) — the visualization half: curves, histograms, embeddings, graph views. Read when you need per-step debugging. cloud.google.com/vertex-ai/docs/experiments/tensorboard-introduction
The DeepWiki page on python-aiplatform experiments/metadata — the implementation view: how runs map to MLMD contexts, how the MLflow plugin wires up, how get_data_frame works. Read when you want to know what’s happening under the SDK, not just how to call it.
A hands-on project to cement it: take a dataset, log a baseline plus five hyperparameter variants, attach TensorBoard for curves, wire lineage from dataset → model, promote the winner to a tracked custom job, then pull get_experiment_df and a parallel-coordinates plot to pick it. Doing this once teaches more than reading every resource above.

15. The Final Verdict

Vertex AI Experiments is a good, honest, unglamorous tracking layer that quietly does the one thing that matters most: it makes your ML work reproducible and comparable without asking you to run any infrastructure. It is not trying to be Weights & Biases. It will not dazzle you with a comparison UI, and if you came from W&B you’ll spend the first week mildly annoyed that you have to write a few lines of pandas to get a chart you used to get by clicking. That annoyance is real, and it’s also the price of something worth having: tracking that lives inside the same governed, lineage-aware graph as the rest of your Google Cloud ML stack, billed like storage instead of like a subscription.

What it gets profoundly right: serverless economics (nothing to host, you pay for bytes, not seats), framework-agnostic capture through a shared metadata graph (the same substrate that answers lineage questions for Pipelines and Model Registry), and native IAM governance — which sounds boring until you’re the fintech team that has to prove to an auditor which dataset produced a deployed fraud model, and the answer is a graph traversal rather than a Slack archaeology dig.

What it costs you: a permanent split-brain between the metadata store and TensorBoard that you’ll never fully stop thinking about; a comparison experience thin enough that you’ll build your own; real lock-in to the Google Cloud graph; and reproducibility that is only as disciplined as your logging, because the tool records exactly what you dictate and not one field more.

Who should reach for it: teams already committed to Google Cloud, running training and pipelines on Vertex, who value low operational burden and governance over UI polish — especially in regulated domains where lineage is not negotiable. Who shouldn’t: multi-cloud or on-prem teams, teams that need best-in-class visual exploration as their daily driver, and anyone who might leave GCP and would resent re-ingesting their history.

What to believe walking away: believe that the serverless, lineage-integrated design is the right call for GCP-native teams and that it’s the path of least resistance if your data and compute already live there. Don’t believe that turning on autolog() means you’re tracking everything — it captures params and metrics for supported frameworks and nothing more, and your reproducibility still rests on the fields you choose to log. When someone says “we use Vertex Experiments,” what they probably mean is “our runs are recorded in the platform’s metadata graph and we compare them in a DataFrame or the console” — not “we have a beautiful dashboard.”

The hard-won line: the tracker doesn’t make your work reproducible — your logging discipline does, and the tracker just remembers what you had the discipline to write down. Vertex AI Experiments is an excellent place to write it down. It won’t do the remembering for you.

The ideas are mine. The writing is AI assisted

Vertex AI Experiments Deep Intuition

1. One-Sentence Essence

2. The Problem It Solved

3. The Concepts You Need

4. The Distilled Introduction

Worked Example A — scikit-learn (Pipeline), manual logging

Worked Example B — the same task with autologging

Worked Example C — PyTorch, manual time-series logging

Worked Example D — tracking from inside a Vertex custom training job

Worked Example E — multi-step lineage you can actually audit

Worked Example F — Pipelines + Experiments: comparing many pipeline runs

Comparing experiment runs

Pulling experiment data into Streamlit / marimo / any app

The whole tool, in one breath

5. The Mental Model

Core Idea 1: Experiments is a bookkeeping layer over a graph database — it records, it does not compute.

Core Idea 2: Everything you log becomes a typed node or edge in a lineage graph.

Core Idea 3: Scalar metadata and time-series metrics live in two different stores, joined only by a reference.

6. The Architecture in Plain English

7. The Things That Bite You

8. The Judgment Calls

9. The Commands/APIs That Actually Matter

10. How It Breaks

11. The Downsides / Disadvantages

12. The Taste Test

13. Operating It in Production

Access control — IAM, not app-level permissions

What it actually costs — a worked example

Getting time-series data out programmatically

Migration — getting in, and getting out

Tracking GenAI and LLM work

14. Where to Go Deeper

15. The Final Verdict

Related reading

BigQuery Deep Intuition

MLflow Deep Intuition

vLLM Deep Intuition