Tutorial vertex ai custom container model deployment production 2026 vertex ai custom container deployment google vertex ai endpoint production vertex ai model registry tutorial deploy pytorch model vertex ai vertex ai serverless inference vertex ai vpc private endpoint vertex ai terraform deployment

Deploying Custom Container Models on Vertex AI: A Production Guide

A guide on building secure, scalable, and GDPR-compliant ML model deployments on Vertex AI, leveraging Terraform, VPC Service Controls, and advanced monitoring in EU regions.

26 min read
Deploying Custom Container Models on Vertex AI: A Production Guide
TL;DR

A guide on building secure, scalable, and GDPR-compliant ML model deployments on Vertex AI, leveraging Terraform, VPC Service Controls, and advanced monitoring in EU regions.

Building Secure and Scalable ML Endpoints with Vertex AI: An EU Practitioner's Guide

When I build machine learning solutions for production, I don't just think about getting a model to infer; I think about the entire lifecycle: security, scalability, observability, and, increasingly, regulatory compliance. For those of us operating within the European Union, additional considerations like data residency and GDPR compliance are not optional – they're paramount. That's why I've been diving deep into Google Cloud's Vertex AI, a managed platform that effectively addresses these challenges, especially for custom container models.

In this guide, I'll walk you through how I deploy a custom PyTorch or TensorFlow model using Vertex AI Model Registry and Prediction Endpoints. My goal is to show you how to build a production-ready container image, register your model, and deploy it to a private endpoint secured with VPC Service Controls. We'll also cover configuring auto-scaling, traffic splitting for canary releases, and setting up comprehensive monitoring with Cloud Logging and Vertex AI Model Monitoring. I'll emphasize an EU perspective throughout, utilizing regions like europe-west1 and europe-west4 to meet data residency requirements and discussing the implications of GDPR and the EU Chips Act for cloud inference strategies. By the end, you'll have the expertise to deploy and manage your custom models on Vertex AI with confidence, ensuring they are secure, performant, and compliant.

Prerequisites

Before we start, ensure your local environment and Google Cloud project are set up. I assume you have:

  • Google Cloud Project: With billing enabled.
  • gcloud CLI: Installed and configured (version 460.0.0 or higher).
  • Docker: Desktop or Engine installed.
  • Python: 3.12+ and pip.
  • Terraform CLI: Installed (version 1.7.0 or higher).
  • Basic Understanding: Of Machine Learning concepts, Docker, and GCP.

Tools I'll Be Using

  • gcloud CLI
  • Docker
  • Python 3.12+
  • Terraform CLI
  • Google Cloud SDK for Python (google-cloud-aiplatform)

Getting Started: Preparing My Environment

Before diving into deployment, I always ensure my Google Cloud environment is prepared and my local machine has the necessary tools installed. For this guide, I'm working with a Google Cloud Project where billing is enabled, and my gcloud CLI is authenticated. All the resources we'll provision will be in European regions to align with our EU compliance goals.

GCP Project Setup

First, I confirm my gcloud configuration points to my target project and region. I typically standardize on europe-west4 for my EU deployments.

gcloud config set project $(gcloud config get-value project)
gcloud config set region europe-west4
gcloud auth application-default login

Required IAM Permissions

The service account I use for Terraform and gcloud operations needs specific IAM roles. For initial setup, roles/owner is often used, but for production, apply the principle of least privilege, using a combination of the following:

  • `roles/resourcemanager.projectIamAdmin
*

roles/serviceusage.serviceUsageAdmin

*

roles/compute.networkAdmin

*

roles/aiplatform.admin

*

roles/artifactregistry.admin

*

roles/storage.admin

*

roles/logging.configWriter

*

roles/monitoring.editor

*

roles/accesscontextmanager.policyAdmin`

Python Environment

To keep my dependencies clean and isolated, I always create and activate a dedicated Python virtual environment for each project. This prevents conflicts and ensures reproducibility.

python3.12 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install google-cloud-aiplatform==1.42.0 google-cloud-storage==2.10.0 flask==3.0.3 gunicorn==22.0.0 transformers==4.38.2 torch==2.2.1 sentencepiece==0.1.99

Building It Out: The Implementation Steps

Step 1: Bootstrapping GCP Infrastructure with Terraform

My first move is to provision infrastructure using Infrastructure as Code. For this, I reach for Terraform. It helps me ensure reproducibility, maintain version control, and keep everything in my chosen EU region. I'm setting up project API enablement, a custom VPC network, a Serverless VPC Access connector, a dedicated service account for Vertex AI, and an Artifact Registry repository. These components are crucial for a secure and private model deployment.

To provision these, I'll define them in my main.tf, variables.tf, and terraform.tfvars files.

# main.tf
provider "google" {
  project = var.project_id
  region  = var.region
}

resource "google_project_service" "api_services" {
  for_each = toset([
    "artifactregistry.googleapis.com",
    "compute.googleapis.com",
    "containerregistry.googleapis.com", # For backward compatibility with Docker.
    "aiplatform.googleapis.com",
    "vpcaccess.googleapis.com",
    "cloudbuild.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "accesscontextmanager.googleapis.com", # For VPC Service Controls
    "logging.googleapis.com",
    "monitoring.googleapis.com"
  ])
  service = each.key
  project = var.project_id
  disable_on_destroy = false
}

resource "google_compute_network" "vpc_network" {
  project = var.project_id
  name    = "vertex-ai-custom-vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "vpc_subnetwork" {
  project       = var.project_id
  name          = "vertex-ai-custom-subnet"
  ip_cidr_range = "10.10.0.0/20"
  region        = var.region
  network       = google_compute_network.vpc_network.id
}

resource "google_vpc_access_connector" "connector" {
  project = var.project_id
  name    = "vertex-ai-connector"
  region  = var.region
  ip_cidr_range = "10.8.0.0/28"
  network = google_compute_network.vpc_network.id

  depends_on = [
    google_project_service.api_services["vpcaccess.googleapis.com"]
  ]
}

resource "google_service_account" "vertex_sa" {
  project      = var.project_id
  account_id   = "vertex-ai-model-deployer"
  display_name = "Service Account for Vertex AI Model Deployment"
}

resource "google_project_iam_member" "vertex_sa_permissions" {
  for_each = toset([
    "roles/aiplatform.user",
    "roles/artifactregistry.writer",
    "roles/storage.objectAdmin",
    "roles/logging.logWriter",
    "roles/monitoring.metricWriter",
    "roles/compute.networkUser" # Required for VPC access
  ])
  project = var.project_id
  role    = each.key
  member  = "serviceAccount:${google_service_account.vertex_sa.email}"
}

resource "google_artifact_registry_repository" "model_repo" {
  project       = var.project_id
  location      = var.region
  repository_id = "vertex-ai-model-images"
  description   = "Docker images for Vertex AI custom model serving"
  format        = "DOCKER"

  depends_on = [
    google_project_service.api_services["artifactregistry.googleapis.com"]
  ]
}

# variables.tf
variable "project_id" {
  description = "The ID of the Google Cloud project."
  type        = string
}

variable "region" {
  description = "The GCP region for deployments (e.g., europe-west4)."
  type        = string
  default     = "europe-west4"
}

variable "access_policy_number" {
  description = "Your organization's Access Policy number for VPC Service Controls. Find this using `gcloud access-context-manager policies list --organization ORGANIZATION_ID`."
  type        = string
}

variable "model_display_name" {
  description = "Display name for the Vertex AI Model."
  type        = string
  default     = "SentimentAnalysisModel"
}

variable "vertex_model_resource_name" {
  description = "Full Vertex AI Model resource name from Step 3 (gcloud ai models list --format='value(name)')."
  type        = string
}

# terraform.tfvars (example)
project_id           = "my-gcp-project-id"
region               = "europe-west4"
access_policy_number = "123456789"
model_display_name   = "SentimentAnalysisModel"
vertex_model_resource_name = "projects/YOUR_PROJECT_NUMBER/locations/europe-west4/models/YOUR_MODEL_ID"

After setting up these files, I run Terraform to provision the resources:

terraform init
terraform plan
terraform apply --auto-approve

Expected Output:

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Troubleshooting: * Error 403: The caller does not have permission: I'd double-check that my gcloud authenticated user has roles/owner or the specific IAM roles I listed in the Prerequisites. * API not enabled: I'd verify that all required google_project_service resources are correctly listed and enabled in my main.tf.

Step 2: Developing and Containerizing My Custom Model Server

Next, I need to package my model and its serving logic into a lightweight, optimized Docker container. I'm using a pre-trained sentiment analysis model from Hugging Face (distilbert-base-uncased-finetuned-sst-2-english) served via a Flask application. The key here is to keep the image small and ensure quick startup times for efficient scaling.

First, I download the model artifacts locally into a model/ directory:

# download_model.py
from transformers import pipeline

# Load a pre-trained sentiment analysis model
sentiment_pipeline = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

# Save the model and tokenizer
sentiment_pipeline.model.save_pretrained("model/")
sentiment_pipeline.tokenizer.save_pretrained("model/")
print("Model saved to ./model/")

python download_model.py

Then, I create my predict.py Flask application and its Dockerfile:

# predict.py
import os
from flask import Flask, request, jsonify
from transformers import pipeline

app = Flask(__name__)

# Load the model globally to avoid reloading on each request
def load_model():
    model_path = os.environ.get('AIP_MODEL_DIR', './model/')
    print(f"Loading model from: {model_path}")
    return pipeline("sentiment-analysis", model=model_path)

sentiment_pipeline = load_model()

@app.route('/ping', methods=['GET'])
def ping():
    return jsonify({'status': 'healthy'})

@app.route('/predict', methods=['POST'])
def predict():
    try:
        instances = request.json['instances']
        if not isinstance(instances, list):
            return jsonify({'error': 'Instances must be a list of strings'}), 400

        results = sentiment_pipeline(instances)
        return jsonify({'predictions': results})
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    # Use gunicorn in production for robustness
    app.run(host='0.0.0.0', port=os.environ.get('AIP_HTTP_PORT', 8080))

# Dockerfile
# Use a slim Python 3.12 image for reduced size
FROM python:3.12-slim-bookworm AS builder

# Install build dependencies for transformers
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/wheels -r requirements.txt

# Final image
FROM python:3.12-slim-bookworm

# Install runtime dependencies from wheels
WORKDIR /app
COPY --from=builder /app/wheels /app/wheels
COPY --from=builder /usr/lib/python3.12/site-packages /usr/lib/python3.12/site-packages
RUN pip install --no-cache-dir --no-index --find-links=/app/wheels -r requirements.txt

# Copy application code and model
COPY predict.py .
COPY model/ ./model/

# Expose the port that Vertex AI expects
ENV AIP_HTTP_PORT=8080
EXPOSE 8080

# Command to run the prediction server using Gunicorn
ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:8080", "predict:app", "--timeout", "90", "--workers", "2"]

# requirements.txt
flask==3.0.3
gunicorn==22.0.0
transformers==4.38.2
torch==2.2.1
sentencepiece==0.1.99 # Required by some Hugging Face models

Now, I build and push the Docker image to my Artifact Registry repository:

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"
REPO_NAME="vertex-ai-model-images"
IMAGE_NAME="sentiment-model-server:v1.0.0"

docker build -t ${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/${IMAGE_NAME} .

gcloud auth configure-docker ${REGION}-docker.pkg.dev
docker push ${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/${IMAGE_NAME}

export CONTAINER_IMAGE_URI=${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/${IMAGE_NAME}
echo "Container image pushed: ${CONTAINER_IMAGE_URI}"

Expected Output:

The push refers to repository [europe-west4-docker.pkg.dev/your-gcp-project-id/vertex-ai-model-images/sentiment-model-server]
... (layers pushed) ...
v1.0.0: digest: sha256:... size: ...
Container image pushed: europe-west4-docker.pkg.dev/your-gcp-project-id/vertex-ai-model-images/sentiment-model-server:v1.0.0

Troubleshooting: * denied: Permission denied for repository: I'd ensure the service account or authenticated user has the Artifact Registry Writer role. * Docker build fails: I'd check requirements.txt for typos or missing dependencies and ensure build-essential is correctly installed in the builder stage.

Optimizing Container Builds for Cloud

While `FROM python:3.12-slim-bookworm` is a great start for a lightweight image, in larger production cloud environments, I'd often consider using GPU-optimized base images (e.g., from NVIDIA's NGC) or Cloud Build's caching mechanisms for faster, more repeatable builds. The `apt-get install` commands in the Dockerfile should always be followed by `rm -rf /var/lib/apt/lists/*` to minimize final image size by clearing package caches, which is critical for cold start times and cost.

Step 3: Registering the Model with Vertex AI Model Registry

Vertex AI Model Registry is my centralized hub for storing and versioning ML models. Registering my container image and its associated model artifacts here allows Vertex AI to manage the model's lifecycle and simplify deployments. For this example, I've baked the model weights directly into the container image for simplicity. For larger models, I'd typically upload them to Cloud Storage (GCS) and have the container pull them at startup.

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"
MODEL_DISPLAY_NAME="SentimentAnalysisModel"

# Register the model with Vertex AI Model Registry
gcloud ai models upload \
  --project=${PROJECT_ID} \
  --region=${REGION} \
  --display-name=${MODEL_DISPLAY_NAME} \
  --container-image-uri=${CONTAINER_IMAGE_URI} \
  --container-command='["gunicorn", "--bind", "0.0.0.0:8080", "predict:app", "--timeout", "90", "--workers", "2"]' \
  --container-predict-route="/predict" \
  --container-health-route="/ping" \
  --container-ports=8080 \
  --description="Sentiment analysis model deployed via custom container"

# The actual model ID is a long number. I'll need this for Terraform.
# I'll store the full resource name (including project and location) for later use.
export MODEL_RESOURCE_NAME=$(gcloud ai models list --project=${PROJECT_ID} --region=${REGION} --filter="displayName=${MODEL_DISPLAY_NAME}" --format="value(name)")
echo "Model registered with resource name: ${MODEL_RESOURCE_NAME}"

Expected Output:

Model [projects/YOUR_PROJECT_NUMBER/locations/europe-west4/models/YOUR_MODEL_ID] uploaded.
... (details) ...
Model registered with resource name: projects/YOUR_PROJECT_NUMBER/locations/europe-west4/models/YOUR_MODEL_ID

Troubleshooting: * 403 Permission denied: I'd ensure the service account has roles/aiplatform.user. * Invalid image URI: I'd double-check the CONTAINER_IMAGE_URI variable for correctness.

Step 4: Deploying a Private Prediction Endpoint with VPC Service Controls

To ensure maximum security and prevent data exfiltration, I always deploy my models to a Vertex AI endpoint within a VPC Service Controls perimeter. This isolates my data and resources, making them inaccessible from outside my defined perimeter. The endpoint will also leverage a private IP for inference requests, keeping traffic strictly within my VPC. This is a critical step for GDPR compliance and overall data security in the EU.

I'll add the following resources to my main.tf to define the VPC Service Controls perimeter and the Vertex AI endpoint. The model itself is referenced by the full resource name from Step 3 (var.vertex_model_resource_name) when I run gcloud ai endpoints deploy-model after Terraform creates the endpoint.

# main.tf (add to previous Terraform file)
#
# The HashiCorp google provider includes google_vertex_ai_endpoint and VPC Service Controls,
# but there is no data source named google_ai_models and no resource google_vertex_ai_endpoint_model.
# After Terraform creates the endpoint, deploy the model from Step 3 with gcloud (shell block below).

resource "google_access_context_manager_service_perimeter" "vertex_ai_perimeter" {
  name        = "accessPolicies/${var.access_policy_number}/servicePerimeters/vertex_ai_perimeter_v1"
  title       = "vertex-ai-perimeter-v1"
  description = "VPC Service Controls perimeter for Vertex AI"
  perimeter_type = "REGULAR"

  status {
    restricted_services = [
      "aiplatform.googleapis.com",
      "artifactregistry.googleapis.com",
      "storage.googleapis.com",
      "cloudbuild.googleapis.com" # Required if Cloud Build is used within the perimeter
    ]
    vpc_accessible_services {
      enable_restriction = true
      allowed_services   = [
        "RESTRICTED_SERVICES"
      ]
    }
  }

  # Ingress/Egress Policies can be defined here for fine-grained control
  # For instance, to allow specific service accounts or IP ranges to interact with resources inside the perimeter.
  # policy_data {
  #   ingress_policies {
  #     ingress_from {
  #       sources {
  #         resource = "projects/${var.project_id}"
  #       }
  #       identities = [
  #         "serviceAccount:${google_service_account.vertex_sa.email}"
  #       ]
  #     }
  #     ingress_to {
  #       resources = [
  #         "projects/${var.project_id}"
  #       ]
  #       operations {
  #         service_name = "aiplatform.googleapis.com"
  #         method_selectors {
  #           method = "*"
  #         }
  #       }
  #     }
  #   }
  # }

  depends_on = [
    google_project_service.api_services["accesscontextmanager.googleapis.com"]
  ]
}

resource "google_vertex_ai_endpoint" "sentiment_endpoint" {
  project         = var.project_id
  location        = var.region
  display_name    = "sentiment-analysis-endpoint"
  description     = "Endpoint for sentiment analysis model via custom container"
  network         = google_compute_network.vpc_network.id # Associate with the VPC network
  service_account = google_service_account.vertex_sa.email

  depends_on = [
    google_access_context_manager_service_perimeter.vertex_ai_perimeter, # Ensure perimeter exists
    google_project_iam_member.vertex_sa_permissions["roles/compute.networkUser"]
  ]
}

After terraform apply creates the endpoint, deploy the registered model (numeric ID) to that endpoint:

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"
# Set from Step 3 export or from terraform.tfvars (variable vertex_model_resource_name), e.g.:
# export VERTEX_MODEL_RESOURCE_NAME="projects/123/locations/europe-west4/models/4567890123456789012"
: "${VERTEX_MODEL_RESOURCE_NAME:?set to full model name from Step 3}"

ENDPOINT_NUM="$(gcloud ai endpoints list --project="${PROJECT_ID}" --region="${REGION}" \
  --filter="displayName=sentiment-analysis-endpoint" --format="value(name)" | awk -F/ '{print $NF}')"
MODEL_NUM="$(printf '%s' "${VERTEX_MODEL_RESOURCE_NAME}" | awk -F/ '{print $NF}')"
SERVICE_ACCOUNT_EMAIL="vertex-ai-model-deployer@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud ai endpoints deploy-model "${ENDPOINT_NUM}" \
  --project="${PROJECT_ID}" \
  --region="${REGION}" \
  --model="${MODEL_NUM}" \
  --display-name="sentiment-model-v1" \
  --machine-type=n1-standard-4 \
  --min-replica-count=1 \
  --max-replica-count=2 \
  --traffic-split=0=100 \
  --service-account="${SERVICE_ACCOUNT_EMAIL}"

Set VERTEX_MODEL_RESOURCE_NAME to var.vertex_model_resource_name (from terraform.tfvars) or to the MODEL_RESOURCE_NAME you captured in Step 3.

Before running Terraform, ensure the model from Step 3 has successfully uploaded. Then, I apply these changes:

terraform apply --auto-approve

Expected Output:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
... (Output showing endpoint creation) ...

Troubleshooting: * 403 Permission denied by VPC Service Controls: I'd ensure my service account is either within the perimeter or has explicit ingress/egress rules allowing access. I'd also verify all relevant services (aiplatform.googleapis.com, storage.googleapis.com, artifactregistry.googleapis.com) are restricted within the perimeter. * Invalid network configuration: I'd verify google_vertex_ai_endpoint.sentiment_endpoint.network correctly references my VPC network ID. * Endpoint does not exist or Model not found: This usually means the endpoint or model ID used with gcloud ai endpoints deploy-model was incorrect, or the gcloud ai models upload in Step 3 was not completed before deployment.

Navigating VPC Service Controls Complexity

VPC Service Controls significantly enhance security, but they add a layer of complexity. Debugging access issues can be challenging. I recommend starting with a minimal set of restricted services and gradually adding more. Always thoroughly test your application's access patterns after implementing VPC SC, as external services or even `gcloud` commands can be affected if not properly configured with access levels.

Step 5: Configuring Autoscaling and Traffic Management

Managing traffic and scaling efficiently is critical for any production deployment. Vertex AI autoscaling and traffic splitting are configured on the deployed model (via gcloud or the REST API), not through a google_vertex_ai_endpoint_model Terraform resource—that resource type is not part of the HashiCorp google provider. After deployment, I tune replicas (and optionally autoscaling metric targets) with gcloud ai endpoints mutate-deployed-model, and I adjust traffic split when I add a second deployment to the same endpoint.

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"
ENDPOINT_NUM="$(gcloud ai endpoints list --project="${PROJECT_ID}" --region="${REGION}" \
  --filter="displayName=sentiment-analysis-endpoint" --format="value(name)" | awk -F/ '{print $NF}')"
DEPLOYED_MODEL_ID="$(gcloud ai endpoints describe "${ENDPOINT_NUM}" --project="${PROJECT_ID}" --region="${REGION}" \
  --format="value(deployedModels[0].id)")"

gcloud ai endpoints mutate-deployed-model "${ENDPOINT_NUM}" \
  --project="${PROJECT_ID}" \
  --region="${REGION}" \
  --deployed-model-id="${DEPLOYED_MODEL_ID}" \
  --min-replica-count=1 \
  --max-replica-count=10

For canary releases, I deploy another model version to the same endpoint, then set the endpoint's traffic map (for example 90% / 10%) using gcloud ai endpoints update or the API—see Vertex AI docs for traffic split on multi-deployment endpoints. Online prediction autoscaling metrics use names like aiplatform.googleapis.com/prediction/online/cpu/utilization (see Google's autoscaling reference).

Expected outcome: replica counts update without Terraform; verify in the Cloud Console or with gcloud ai endpoints describe.

Step 6: Implementing Comprehensive Monitoring and Logging

Robust monitoring and logging are non-negotiable for operational excellence, and even more so for GDPR compliance. I'll configure Cloud Logging to capture all prediction requests and responses, directing these logs to a regional sink. Additionally, I'll enable Vertex AI Model Monitoring to detect data drift, concept drift, and attribution drift, ensuring my model remains accurate and performs as expected over time. For GDPR, it's vital that my logs are retained in EU regions with appropriate policies.

# main.tf (add to previous Terraform file)
# Model Monitoring jobs are created via Console, gcloud, or the Vertex AI API. The HashiCorp
# google provider does not define google_vertex_ai_model_monitoring_job. Below: EU log bucket + sink.

resource "google_logging_project_sink" "vertex_ai_log_sink" {
  project = var.project_id
  name    = "vertex-ai-audit-log-sink"
  # I'm directing logs to a GCS bucket, ensuring it's in an EU region.
  destination = "storage.googleapis.com/${google_storage_bucket.log_bucket.name}"
  filter      = "resource.type=(\"aiplatform.googleapis.com/Endpoint\" OR \"aiplatform.googleapis.com/Model\")"

  depends_on = [
    google_project_service.api_services["logging.googleapis.com"],
    google_project_service.api_services["storage.googleapis.com"]
  ]
}

resource "google_storage_bucket" "log_bucket" {
  project       = var.project_id
  name          = "vertex-ai-log-bucket-${var.project_id}"
  location      = upper(var.region) # e.g., EUROPE-WEST4
  force_destroy = false
  uniform_bucket_level_access = true

  # I set a retention policy for GDPR compliance, for example, 365 days.
  retention_policy {
    is_locked        = false
    retention_period = 31536000 # 365 days in seconds
  }
}

data "google_project" "project" {
  project_id = var.project_id
}

# Grant the logging service account permission to write to the bucket
resource "google_storage_bucket_iam_member" "log_bucket_iam" {
  bucket = google_storage_bucket.log_bucket.name
  role   = "roles/storage.objectCreator"
  member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-logging.iam.gserviceaccount.com"
}

Model monitoring: I create a Model Monitoring job in the Google Cloud console (Vertex AI → Endpoints → Monitoring) or with gcloud ai model-monitoring-jobs create, using baseline data and schema in EU GCS buckets. I pass the deployed model ID from gcloud ai endpoints describe. The removed Terraform block used a resource type that does not exist in the HashiCorp provider.

Note on schemas and baselines: Monitoring compares live traffic to a baseline dataset and schema. Keep both in an EU bucket for residency; do not rely on US-only sample URIs for production baselines.

Now, I run Terraform again:

terraform apply --auto-approve

Expected Output:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Troubleshooting: * Model monitoring job failed: I'd verify the gcs_path for the schema and baseline data is correct and accessible by the Vertex AI service account. I'd also ensure the deployed_model_id matches an active deployment on the endpoint. * Permission denied for logging sink: I'd ensure the Cloud Logging service account (service-PROJECT_NUMBER@gcp-sa-logging.iam.gserviceaccount.com) has Storage Object Creator permissions on the destination bucket.

Step 7: Validating EU Data Residency and Compliance

For any EU-based organization, ensuring data residency is crucial for GDPR compliance and adherence to initiatives like the EU Chips Act. The EU Chips Act emphasizes strengthening the EU's semiconductor ecosystem, impacting how organizations might consider on-premises versus cloud inference for sensitive data. By deploying resources exclusively in EU regions, I maintain control over data location. I've configured everything in europe-west4, but europe-west1 would also work. I'll quickly verify the locations using gcloud commands:

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"

echo "Verifying Artifact Registry location..."
gcloud artifacts repositories describe vertex-ai-model-images --location=${REGION} --project=${PROJECT_ID} --format="value(location)"

echo "Verifying Vertex AI Endpoint location..."
gcloud ai endpoints describe sentiment-analysis-endpoint --region=${REGION} --project=${PROJECT_ID} --format="value(location)"

echo "Verifying Log Bucket location..."
gcloud storage buckets describe gs://vertex-ai-log-bucket-${PROJECT_ID} --format="value(location)"

echo "Verifying VPC Access Connector location..."
gcloud vpc-access connectors describe vertex-ai-connector --region=${REGION} --project=${PROJECT_ID} --format="value(location)"

Expected Output:

Verifying Artifact Registry location...
europe-west4
Verifying Vertex AI Endpoint location...
europe-west4
Verifying Log Bucket location...
EUROPE-WEST4
Verifying VPC Access Connector location...
europe-west4

This verification confirms my infrastructure adheres to EU data residency requirements, positioning my ML deployments for GDPR compliance.

Step 8: Performing a Prediction and Observing Results

With the model deployed and monitoring configured, it's time to initiate a sample prediction request to test the endpoint and observe the results in Cloud Logging. This demonstrates the end-to-end inference flow.

# predict_client.py
import os
from google.cloud import aiplatform

PROJECT_ID = os.environ.get('PROJECT_ID')
REGION = os.environ.get('REGION', 'europe-west4')
ENDPOINT_NAME = os.environ.get('ENDPOINT_NAME', 'sentiment-analysis-endpoint')

def predict_text_sentiment(project: str, location: str, endpoint_name: str, text_instances: list):
    aiplatform.init(project=project, location=location)

    # I filter by display_name to ensure I get the correct endpoint
    endpoints = aiplatform.Endpoint.list(filter=f"display_name=\"{endpoint_name}"")
    if not endpoints:
        raise ValueError(f"Endpoint with display_name {endpoint_name} not found.")
    endpoint = endpoints[0]

    # The request format depends on my container's /predict route implementation
    # For my Flask app, it expects {'instances': ["text1", "text2"]}
    instances_payload = [{"instances": text_instances}]

    response = endpoint.predict(instances=instances_payload)

    print("Prediction response:")
    for prediction in response.predictions:
        print(prediction)

if __name__ == '__main__':
    # Example sentences for sentiment analysis
    sample_texts = [
        "This is an amazing product, I love it!",
        "The service was terrible and I am very disappointed.",
        "It was an okay experience, neither good nor bad."
    ]

    print(f"Sending prediction request to {ENDPOINT_NAME} in {REGION} for project {PROJECT_ID}")
    predict_text_sentiment(PROJECT_ID, REGION, ENDPOINT_NAME, sample_texts)

I'll run the prediction script:

PROJECT_ID=$(gcloud config get-value project)
REGION="europe-west4"
ENDPOINT_NAME="sentiment-analysis-endpoint"

export PROJECT_ID
export REGION
export ENDPOINT_NAME

python predict_client.py

Expected Output:

Prediction response:
{'label': 'POSITIVE', 'score': 0.9998...}
{'label': 'NEGATIVE', 'score': 0.9996...}
{'label': 'POSITIVE', 'score': 0.8876...} # (Neutral might be classified as slightly positive or negative depending on model)

After running the prediction, I navigate to Cloud Logging in my GCP Console. I filter logs by resource.type="aiplatform.googleapis.com/Endpoint" and resource.labels.endpoint_id="YOUR_ENDPOINT_ID_FROM_TERRAFORM_OUTPUT" to see my prediction requests and responses. This verifies that my logging sink is actively capturing inference data.

Troubleshooting: * 403 Permission denied: I'd ensure the service account running predict_client.py (my user account if using gcloud auth application-default login) has roles/aiplatform.user and necessary network access if not running within the VPC. * ValueError: Endpoint with display_name... not found.: I'd double-check the ENDPOINT_NAME and REGION variables and ensure my endpoint was successfully deployed.

Testing My Implementation

Beyond a single prediction, I always thoroughly test my deployment to ensure stability, performance, and scalability.

  1. Load Testing: I use tools like ApacheBench (ab), Locust, or JMeter to simulate concurrent users and verify autoscaling behavior. I always monitor CPU utilization and replica counts in Cloud Monitoring.
# Example with ApacheBench
# Install: sudo apt-get install apache2-utils
# Note: Requires external access or a client within your VPC to reach the private endpoint.
# For private endpoints, this would typically be run from a VM inside the same VPC.
# The URL below is for public endpoints. For private, you'd address it via its internal IP.
# If you expose it via a Load Balancer or similar, that URL would be used.
# For direct testing from a VM within the VPC, you'd use a different approach for endpoint discovery.
# For this example, I'm assuming temporary external access for demonstration, or testing from inside the VPC.
# Replace ENDPOINT_ID with your Vertex AI endpoint's ID.
# You can get the endpoint's public DNS if you hadn't restricted external access:
# gcloud ai endpoints describe sentiment-analysis-endpoint --region=europe-west4 --format="value(publicEndpoint.dnsName)"
ENDPOINT_ID="$(gcloud ai endpoints list --project=${PROJECT_ID} --region=${REGION} --filter="display_name=sentiment-analysis-endpoint" --format="value(endpoints[0].id)")"
ENDPOINT_URL="https://europe-west4-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:predict"
PAYLOAD='{"instances": ["This is a test sentence."]}'

# Example: 100 requests, 10 concurrent.
# Note: This requires the endpoint to be publicly accessible or accessed from within a VPC-connected environment.
# If VPC SC restricts external access, this `ab` command would likely fail unless run from within the perimeter.
ab -n 100 -c 10 -T 'application/json' -p <(echo "${PAYLOAD}") -H "Authorization: Bearer $(gcloud auth print-access-token)" "${ENDPOINT_URL}"

  1. Canary Rollout Verification: If I introduce a new model version with a small traffic split, I ensure only the specified percentage of requests are routed to the new version. I typically observe this by logging unique identifiers from each model version within my predict.py or by examining prediction outcomes if the models produce distinct results.

  2. Monitoring Alerts: I intentionally trigger conditions for my Vertex AI Model Monitoring (e.g., by sending data that would cause drift) and confirm that email or Pub/Sub alerts are received as configured.

  3. Log Review: I regularly inspect Cloud Logging for errors, latency spikes, and correct prediction outputs. I make sure sensitive data is handled according to my GDPR policies, and that logging configuration is correctly filtering or redacting what's necessary.

Production Considerations

Deploying to production involves more than just getting the code to run. Here are some key areas I always focus on:

Security Best Practices

  • VPC Service Controls: As I've demonstrated, always use VPC Service Controls for sensitive workloads to prevent data exfiltration. This creates a strong security perimeter around your Vertex AI resources.
  • Least Privilege: Grant the Vertex AI service account only the absolutely necessary IAM roles. I avoid roles/editor or roles/owner for production deployments.
  • Secrets Management: Store API keys, credentials, or sensitive model parameters in Secret Manager, not directly in your container image or code.
  • Container Security: Regularly scan your Docker images for vulnerabilities using Artifact Analysis or third-party tools. Use minimal base images like python:3.12-slim to reduce the attack surface.
  • Model Governance: Implement model signing (e.g., using Sigstore/Cosign) to verify the integrity and origin of your model artifacts. Ensure comprehensive audit logging is enabled across all stages of the ML lifecycle.

Performance Optimization

  • Machine Types: Select appropriate machine types (n1-standard, a2-highgpu, etc.) for your workload. GPU instances significantly accelerate inference for large deep learning models.
  • Container Optimization: Optimize your Dockerfile for size and cold start performance. Use multi-stage builds, clean up unnecessary files, and pre-load models into memory.
  • Batching: For high-throughput scenarios, implement request batching within your predict.py to process multiple inferences in a single call, reducing overhead.
  • Horizontal Scaling: Configure min_replica_count and max_replica_count with aggressive autoscaling_metric_specs to handle traffic fluctuations effectively.

Scalability Considerations

  • Regional Deployment: Deploy models in regions geographically close to your users to minimize latency, particularly in europe-west1 or europe-west4 for EU users.
  • Managed Service: Vertex AI is a managed service, handling underlying infrastructure scaling. My focus shifts to optimizing my container and model for efficiency rather than managing Kubernetes clusters.
  • Quota Management: I'm always aware of Vertex AI and Compute Engine quotas. I proactively request increases if I anticipate very high traffic volumes.

Monitoring Recommendations

  • Dashboards: I create custom dashboards in Cloud Monitoring to visualize key metrics like request latency, error rates, CPU/GPU utilization, and replica count.
  • Alerting Policies: I set up granular alerting policies for critical thresholds (e.g., P99 latency exceeding 500ms, error rate above 1%).
  • Logging Analysis: I leverage Cloud Logging with BigQuery exports for advanced log analysis and auditing.
  • Model Monitoring: Continuously using Vertex AI Model Monitoring to detect data drift, concept drift, and prediction quality degradation is essential. Proactive model monitoring can reduce the mean time to detect model degradation by 70%, which is a significant operational win.

Common Questions

Q: How can I update my deployed model without downtime?

A: I achieve this using traffic splitting on my Vertex AI Endpoint. I deploy the new model version (e.g., version 1) to the same endpoint with 0% traffic. Once verified, I gradually shift traffic (e.g., 5%, then 25%, then 100%) to the new version. This allows for canary releases and easy rollback if issues arise.

Q: What's the cost of deploying a custom model on Vertex AI?

A: Vertex AI pricing for custom container models is primarily based on machine type, accelerator usage, and replica count. For instance, an n1-standard-4 machine type without GPUs costs approximately €0.15/hour ($0.16/hour) in europe-west4. Model monitoring incurs additional costs, roughly €0.01/1000 prediction requests ($0.01/1000) sampled. With a conversion rate of $1 \approx €0.92$, a small n1-standard-4 endpoint running 24/7 with 1 replica would be around €109.50/month ($119.00/month) for the instance, plus request costs. I always refer to the Vertex AI pricing page for the most current figures and specific regional variations.

Q: Can I use GPUs with my custom container model?

A: Yes, Vertex AI supports GPU-enabled machine types. I would specify accelerator_type (e.g., NVIDIA_TESLA_T4) and accelerator_count in my machine_spec within the dedicated_resources block of my google_vertex_ai_endpoint_model Terraform resource. It's crucial that my Docker image has the necessary GPU drivers and libraries (e.g., CUDA, cuDNN, NVIDIA Container Toolkit) baked in.

Q: How do I handle large model artifacts that don't fit in the container image?

A: For models larger than a few GB, I store the model artifacts in a GCS bucket. My custom container predict.py then downloads these artifacts from GCS into the container's /tmp directory during container startup. Vertex AI will automatically provide credentials for the deployed model's service account to access GCS, simplifying access management.

Q: How does VPC Service Controls impact model deployment and access?

A: VPC Service Controls creates a security perimeter that restricts data movement. Resources inside the perimeter (like my Vertex AI endpoint and GCS buckets) can only communicate with other resources within the same perimeter or explicitly allowed services. This means clients accessing the endpoint must either be within the perimeter (e.g., a VM in my VPC) or explicitly granted ingress access via an Access Level. It significantly enhances data security by preventing unauthorized access and data exfiltration.

Q: What are the implications of the EU Chips Act on cloud ML deployments?

A: The EU Chips Act aims to bolster the EU's semiconductor industry. While primarily focused on manufacturing, it has implications for data processing and supply chain resilience. For my ML deployments, it encourages me to consider where my inference processing occurs. Using cloud regions within the EU, such as europe-west1 or europe-west4, ensures data remains within the EU jurisdiction, which is advantageous for compliance and reducing geopolitical risks associated with data processing locations, even if the underlying chips are sourced globally. It reinforces the need for clear data residency strategies in cloud operations.

Q: What is the recommended Python version for custom containers on Vertex AI in 2026?

A: As of early 2026, Python 3.12+ is the recommended version for new developments. It offers performance improvements and new features that contribute to more efficient model serving. I always aim to use the latest stable Python version supported by my core ML libraries (PyTorch, TensorFlow, Transformers) to benefit from the latest optimizations and security patches.

Conclusion

Deploying custom machine learning models to production, especially within the stringent regulatory landscape of the EU, is a multi-faceted challenge. Throughout this guide, I've walked you through my process for building a secure, scalable, and compliant Vertex AI deployment using Infrastructure as Code. The combination of Artifact Registry for container management, Vertex AI Model Registry for versioning, private Prediction Endpoints with VPC Service Controls for data isolation, and robust monitoring with Cloud Logging and Vertex AI Model Monitoring creates a solid foundation for any production ML workload.

One significant trade-off to consider is the initial complexity introduced by VPC Service Controls. While incredibly powerful for security, it requires careful planning and can complicate debugging. My recommendation is to invest the time upfront to design your network and access policies correctly, as retrofitting can be much more painful. For FinOps, always keep a close eye on your replica counts and machine types; idle GPU instances, in particular, can lead to unexpected costs if autoscaling isn't tuned correctly.

My actionable next step after setting up such a pipeline is always to refine the model monitoring. I prioritize establishing a robust baseline dataset and a detailed schema that accurately reflects the model's inputs and outputs. This proactive approach allows me to catch data drift or performance degradation early, ensuring my models continue to deliver value in production, not just in development.

Last updated:

This article was produced using an AI-assisted research and writing pipeline. Learn how we create content →