Triage and Remediation
- Remediation
Remediation
Using Console
Using Console
Below are the concise, console-based steps to ensure Cloud Monitoring is enabled for Vertex AI Workbench notebook instances.
1. Verify Cloud Monitoring & APIs
- Go to Google Cloud console: https://console.cloud.google.com
- In the top-left, select the correct Project.
- In the left menu, go to APIs & Services → Library.
- Confirm the following APIs are Enabled (enable if they are not):
- Cloud Monitoring API
- Vertex AI API
2. Enable Monitoring on a New Vertex AI Workbench Notebook
- Go to Vertex AI → Workbench in the console.
- Click New notebook → choose the type (e.g., Managed notebooks).
- Fill in the basic details (name, region, machine type, etc.).
- Expand Advanced Settings (or similar section, wording may slightly differ).
- Find the Monitoring or Operations section:
- Check/enable Cloud Monitoring / Ops Agent / Enable monitoring option.
- Complete the rest of the configuration as desired.
- Click Create.
3. Enable Monitoring on an Existing Notebook Instance
You typically need to stop the instance to edit some settings.
- Go to Vertex AI → Workbench.
- Locate your notebook instance in the list.
- If it’s Running, click the three‑dot menu (⋮) → Stop and wait until status is Stopped.
- Again click the three‑dot menu (⋮) → Edit.
- In the edit page, expand Advanced Settings.
- Locate the Monitoring / Ops Agent / Enable Cloud Monitoring section and:
- Check/enable the monitoring option.
- Save the changes (e.g., Save, Update, or Save & Continue depending on UI).
- Once updated, click Start to start the notebook instance again.
4. Confirm Metrics in Cloud Monitoring
- Go to Monitoring → Metrics explorer in the console.
- In Resource type, select:
VM InstanceorVertex AI Workbench(depending on what appears for your setup).
- Check that you see metrics (CPU, memory, disk, etc.) for your notebook VM.
Using CLI
Using CLI
Below is a practical CLI-based way to ensure Vertex AI notebook instances are sending metrics to Cloud Monitoring. The approach is different for:
If you are only using managed notebooks, there is nothing more to “turn on” for Monitoring via CLI.
Note: For user-managed notebooks, the instance name is typically the same as the underlying GCE VM name and uses the same zone.Replace RHEL/CentOS:The agent will automatically send system metrics and logs to Cloud Monitoring/Logging.Exit the VM when done:You should see the service as
Then check metrics in the console:
- Managed Notebooks (Vertex AI Workbench managed) – monitoring is on by default; you mainly need APIs enabled.
- User-Managed Notebooks (Vertex AI Workbench user-managed / legacy AI Platform Notebooks) – these are GCE VMs; you enable monitoring by installing the Ops Agent on the VM.
1. Prerequisites (all Vertex AI notebooks)
1.1. Set default project and region/zone
1.2. Enable required APIs
2. Managed Notebooks (Vertex AI Workbench – Managed)
For managed notebooks, Google manages the underlying infra and Cloud Monitoring integration is enabled by default. The main remediation is just ensuring the APIs are on (done above).To verify instances:3. User-Managed Notebooks (VM-based) – Install Ops Agent
For user-managed notebooks, each notebook is a Compute Engine VM. To enable Cloud Monitoring, install the Ops Agent on those VMs.3.1. List user-managed notebook instances
3.2. Enable OS Login (recommended, if not already)
INSTANCE_NAME with your notebook VM name.3.3. SSH into the notebook VM
3.4. Install the Ops Agent (Cloud Monitoring + Logging)
Once logged into the VM:Debian/Ubuntu (most Vertex AI notebook images are Debian-based):3.5. (Optional) Verify the Ops Agent status
From the VM:active (running).4. Verify in Cloud Monitoring
After a few minutes, verify metrics:- Go to Monitoring → Metrics explorer
- Resource type:
gce_instance - Filter by instance name of your notebook VM.
Using Python
Using Python
Below is a practical way to ensure Vertex AI Workbench (user‑managed) notebook VMs are correctly configured for Cloud Monitoring using Python.For Vertex AI Workbench user‑managed notebooks, monitoring is essentially:
Ensure your default credentials have
Any instance printed as
and that you’re using a recent Vertex AI Workbench image (which includes the Ops Agent).If you want to recreate an existing instance:
If you can share whether you’re using user‑managed or managed notebooks and what your current instance config looks like (or a sample
- The VM must have the Cloud Monitoring and Cloud Logging OAuth scopes.
- The Ops Agent / Monitoring agent must be installed (on current images it usually is by default).
- Check scopes for existing notebook VMs.
- Create/recreate a notebook instance with Monitoring & Logging enabled.
1. Setup
roles/notebooks.admin and roles/compute.viewer at least.2. Check if existing notebook instances have Monitoring enabled
Vertex AI Workbench user-managed notebooks are backed by Compute Engine VMs. We’ll:- List notebook instances via the Notebooks API.
- For each, find the underlying VM and inspect its OAuth scopes.
- Flag instances missing
monitoring.writeandlogging.write.
missing scopes should be recreated with the correct scopes.3. Create (or recreate) a notebook with Cloud Monitoring enabled
The simplest way to “enable Cloud Monitoring” is to ensure the notebook’s underlying VM has:- Export current settings (machine type, disks, network, etc.).
- Delete the old instance.
- Call
create_instancewith the same settings plus the Monitoring & Logging scopes as above.
4. (Optional) Verify in Cloud Monitoring
After the instance is running:- In the console: Monitoring → Metrics Explorer → find
agent.googleapis.commetrics for the VM. - Or via API, list time series for
agent.googleapis.com/agent/uptimefor that VM.
If you can share whether you’re using user‑managed or managed notebooks and what your current instance config looks like (or a sample
Instance JSON), I can adjust the Python exactly to your environment.Using Terraform
Using Terraform
terraform plan should show no changes related to Cloud Monitoring for the notebook instances, because the setting is not managed by the provider.
