Google.monitoring.v3.metricservice.createmetricdescriptor, remediation

Event Information

What the event means
google.monitoring.v3.MetricService.CreateMetricDescriptor is an API call to create a new metric descriptor in Cloud Monitoring. This defines a new metric type (usually a custom metric), including its name, labels, unit, value type, and description, which Cloud Monitoring will then accept time series data for.
When/why it occurs operationally
It is triggered when:
- You or a service run gcloud monitoring metrics descriptors create, use the REST/Client API, or deploy something (e.g., code, Terraform, Deployment Manager) that defines a new custom metric.
- Applications or agents start exporting metrics with a previously unknown type, causing automated creation (if implemented that way).
Security & compliance relevance
- Creation of new metrics can expose new data categories (e.g., PII, system internals), so this event should be logged and monitored for change control (ISO 27001 A.12, SOC 2 CC8, PCI-DSS 10).
- Repeated or unexpected metric descriptor creation can indicate misconfiguration or abuse (e.g., noisy or data-exfil-like metrics), so consider alerting on anomalous patterns and enforcing least privilege (monitoring.metricDescriptors.create).

Examples

Creation of metrics exposing sensitive data (PII/secrets/keys)
- Example: A custom metric descriptor custom.googleapis.com/user_email or custom.googleapis.com/jwt_token is created, causing PII/tokens to be exported to logs, dashboards, and external sinks (e.g., Pub/Sub, BigQuery, Splunk), violating GDPR/CCPA or internal data-handling policies.
- Mitigation: Restrict who can call CreateMetricDescriptor via IAM, enforce naming/labeling conventions with org policies, and use DLP scans on Monitoring exports.
Stealthy exfiltration or C2 channel via custom metrics
- Example: An attacker with limited permissions uses CreateMetricDescriptor plus metric writes to encode and exfiltrate database query results or secrets into time-series labels/values, which then get exported to external monitoring backends, breaching data residency and confidentiality (e.g., SOC 2, ISO 27001).
- Mitigation: Monitor and alert on anomalous metric descriptor creations (unused services, odd names, high-cardinality labels), review export sinks, and apply least-privilege on metric write roles.
Compliance and logging blind spots through unreviewed security metrics
- Example: A team creates a metric custom.googleapis.com/firewall_denies but misconfigures labels/types, leading to missing or inaccurate security telemetry required for PCI-DSS/ISO 27001 control evidence (e.g., incomplete logging of access denials).
- Mitigation: Govern metric descriptor creation via change management, standardize security-related metric schemas, and periodically audit descriptors for correctness and alignment with compliance logging requirements.

Remediation

Using Console

Lock down who can create / write custom metrics (prevent PII & exfiltration)
- In GCP Console: go to IAM & Admin → IAM, filter for roles containing Monitoring Metric Writer or Monitoring Admin.
- For each project/folder/org, remove broad roles like roles/monitoring.editor, roles/owner, or roles/editor from general groups; instead grant:
  - roles/monitoring.metricWriter only to service accounts that truly need to push metrics.
  - roles/monitoring.admin or roles/monitoring.metricDescriptors.writer only to a small SRE/Platform group.
- Under IAM & Admin → Roles, create custom roles that exclude monitoring.metricDescriptors.create and assign them to regular developers; reserve metric descriptor creation for a controlled group (supports least-privilege for SOC 2, ISO 27001).
Review, clean up, and govern metric descriptors & exports (PII, C2, and compliance)
- In Monitoring → Metrics explorer → Query tab → Metric, type custom.googleapis.com/ to list custom metrics; look for:
  - PII/secrets (e.g., user_email, jwt_token, session_id, api_key in names/labels).
  - Suspicious / high-cardinality patterns (query_result_*, *_dump, long random strings in labels).
  - Misconfigured security metrics (e.g., firewall_denies missing key labels such as source_ip, action).
- For each problematic metric:
  - Open Monitoring → Metrics scope → Metric descriptors, select the metric, and document and deprecate it (you cannot delete data, but you can:
    - Stop writes by updating applications, removing the writer’s IAM, or disabling the involved service account under IAM & Admin → Service Accounts.
    - Update dashboards/alerts to stop using the bad metric and migrate to a corrected schema.)
- In Monitoring → Notifications & integrations → Sinks / Export (or Logging → Logs Router if you export Monitoring via logs), review all exports (Pub/Sub, BigQuery, Splunk, etc.):
  - Ensure exports with Monitoring/timeSeries data only go to approved locations (data residency, GDPR/CCPA).
  - Restrict sink service account IAM on targets (BigQuery dataset, Pub/Sub topic, bucket) to least-privilege.
Standardize metrics & add monitoring for abuse / gaps (governance & detection)
- Define a central metrics standard (name prefix, allowed labels, and data types) in your security/platform docs, e.g.:
  - Names: custom.googleapis.com/security/firewall_denies with fixed labels (source_ip, dest_ip, action, rule_name).
  - Explicit rule: no PII or secrets in metric names, labels, or values (GDPR/CCPA).
- Implement change control: require metric descriptor changes to go through a ticket/PR reviewed by security/compliance; in Console, restrict roles/monitoring.admin to that change-control group.
- In Monitoring → Alerting → Create policy:
  - Add conditions on Metric: “API request count” filtered on method="google.monitoring.v3.MetricService.CreateMetricDescriptor" to alert on:
    - New custom metrics in unusual projects/services.
    - Metrics with suspicious names (match filters like metric.type=starts_with("custom.googleapis.com/") AND metric.type:("token" OR "email" OR "jwt" OR "secret" OR "key") via logs-based metrics + alerting).
  - Periodically export metric descriptors (via script/CSPM tool) and use DLP or regex scans on names/labels in your CI/CD or security review to catch PII/secrets and ensure key security metrics meet PCI-DSS / ISO 27001 evidence requirements.

Using CLI

Lock down who can create/write sensitive custom metrics (PII/exfil paths)

Identify and review who currently can create descriptors and write metrics:

List IAM on the project / folder / org for Monitoring roles:

gcloud projects get-iam-policy PROJECT_ID \
  --filter="bindings.role:roles/monitoring.metricWriter OR bindings.role:roles/monitoring.admin" \
  --format="table(bindings.role, bindings.members)"

Remove broad monitoring.admin / monitoring.metricWriter from non-platform/service principals; replace with least-privilege roles or custom roles without monitoring.metricDescriptors.create and monitoring.timeSeries.create:
gcloud projects remove-iam-policy-binding PROJECT_ID \ --member="user:someone@example.com" \ --role="roles/monitoring.admin"

For sensitive environments (GDPR/CCPA/SOC 2), restrict metric descriptor creation to a small admin group and enforce approval via change management (e.g., PR + ticket) before granting:

Create a custom role without descriptor-creation permissions:

gcloud iam roles create limitedMetricWriter \
  --project=PROJECT_ID \
  --title="Limited Metric Writer" \
  --permissions="monitoring.timeSeries.create" \
  --stage="GA"

Bind only this role to app SAs:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:APP_SA@PROJECT_ID.iam.gserviceaccount.com" \
  --role="projects/PROJECT_ID/roles/limitedMetricWriter"

Detect/clean up PII / C2-like metrics and secure exports

Enumerate and review custom metric descriptors for PII/secrets and suspicious C2-style naming or high-cardinality labels:

gcloud monitoring metric-descriptors list \
  --project=PROJECT_ID \
  --filter='metric.type=starts_with("custom.googleapis.com/")' \
  --format="table(name, type, displayName, labels)"

Delete offending metrics (break exfil path; document for audit):

gcloud monitoring metric-descriptors delete \
  custom.googleapis.com/user_email \
  --project=PROJECT_ID

Audit and harden export sinks that could receive sensitive metric data (BigQuery, Pub/Sub, Logging):

# Logging sinks that might receive Monitoring data
gcloud logging sinks list --project=PROJECT_ID
gcloud logging sinks describe SINK_NAME --project=PROJECT_ID

# Pub/Sub topics / subscriptions used by monitoring exports
gcloud pubsub topics list --project=PROJECT_ID
gcloud pubsub subscriptions list --project=PROJECT_ID

Restrict IAM on these sinks to least privilege and ensure data residency controls (e.g., EU-only BigQuery datasets).

Implement anomaly detection using Monitoring or Logging for:
- Unusual descriptor creation events (names like jwt, token, secret, query_result, high-label-cardinality, created by unexpected principals).
- Use Cloud Logging queries on protoPayload.methodName="google.monitoring.v3.MetricService.CreateMetricDescriptor" and set alerts:
  gcloud logging read \ 'protoPayload.methodName="google.monitoring.v3.MetricService.CreateMetricDescriptor"' \ --project=PROJECT_ID \ --limit=50 \ --format="table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)"

Standardize & continuously audit security/compliance metrics
- Define a catalog of approved security metrics (e.g., custom.googleapis.com/security/firewall_denies) with fixed label schemas; store definitions in source control and require change-approval before creation. Create descriptors only via automated pipelines, not ad hoc:
  gcloud monitoring metric-descriptors create \ --project=PROJECT_ID \ --descriptor-from-file=firewall_denies_descriptor.json
- Periodically validate existing descriptors against your standard and compliance requirements (PCI-DSS, ISO 27001 logging controls):
  gcloud monitoring metric-descriptors list \ --project=PROJECT_ID \ --filter='metric.type=starts_with("custom.googleapis.com/security/")' \ --format="json" > current_security_metrics.json # Compare current_security_metrics.json to baseline in code repo
- For exports used as evidence (e.g., SOC 2/PCI reporting pipelines), ensure completeness/accuracy:
  - Verify that required labels (e.g., source_ip, dest_ip, action) exist and have correct types; fix by updating descriptor (delete + recreate) and updating producers.
  - Run scheduled DLP scans on BigQuery tables / logs receiving metrics containing user-related or network fields to ensure no PII or secrets leak into monitoring datasets.

Using Python

Lock down who can create/write custom metrics (IAM + org policy)

Restrict roles like roles/monitoring.metricWriter and especially roles/monitoring.admin to tightly controlled service accounts / groups; remove broad grants on projects/folders/org.
Use org policies / policy constraints (e.g. label/namespace conventions enforced via CI/CD) to prevent PII in metric names/labels; add pre-commit / pre-merge checks for custom.googleapis.com/*.

Example Python helper to list and then remove overly broad IAM bindings on a project for Monitoring roles (run from an admin workstation / pipeline):

from google.cloud import resourcemanager_v3

PROJECT_ID = "my-project-id"
ROLES_TO_RESTRICT = {
    "roles/monitoring.metricWriter",
    "roles/monitoring.editor",
    "roles/monitoring.admin",
}

client = resourcemanager_v3.ProjectsClient()
project_name = f"projects/{PROJECT_ID}"
policy = client.get_iam_policy(request={"resource": project_name})

new_bindings = []
for b in policy.bindings:
    if b.role in ROLES_TO_RESTRICT and any(
        m in ("allUsers", "allAuthenticatedUsers") for m in b.members
    ):
        # Drop public bindings; adjust logic to match your policy
        continue
    new_bindings.append(b)

policy.bindings[:] = new_bindings
client.set_iam_policy(request={"resource": project_name, "policy": policy})

Detect and respond to suspicious or non-compliant custom metrics (DLP, audits, alerts)

Periodically scan metric descriptors for PII/secrets patterns in names and labels (GDPR/CCPA) and for stealth exfil channels (weird names, high-cardinality labels, unused services).
Combine this with reviews of Monitoring export sinks (Pub/Sub, BigQuery, Splunk) to ensure they are approved, encrypted, and conform to data residency (SOC 2 / ISO 27001).

Example Python script to list custom descriptors and flag risky ones (for review / auto-create tickets or break build):

import re
from google.cloud import monitoring_v3

PROJECT_ID = "my-project-id"
RISKY_PATTERNS = [
    r"email",
    r"e-?mail",
    r"jwt",
    r"token",
    r"secret",
    r"password",
    r"apikey",
    r"api_key",
    r"ssn",
]
risky_re = re.compile("|".join(RISKY_PATTERNS), re.IGNORECASE)

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{PROJECT_ID}"

def is_risky(text: str) -> bool:
    return bool(text and risky_re.search(text))

for md in client.list_metric_descriptors(name=project_name):
    if not md.type.startswith("custom.googleapis.com/"):
        continue
    flagged = []
    if is_risky(md.type):
        flagged.append("type")
    for label in md.labels:
        if is_risky(label.key) or is_risky(label.description):
            flagged.append(f"label:{label.key}")
    if flagged:
        print(f"[RISK] {md.name} ({md.type}) fields={flagged}")

Standardize and validate security/compliance metrics before use (governance + linting)

Maintain approved schemas for security metrics (e.g. custom.googleapis.com/security/firewall_denies) with mandatory labels and types aligned to PCI-DSS/ISO 27001 evidence requirements.
Enforce that all new metric descriptors are created only via CI/CD (not ad-hoc) and pass a validation step (no PII, correct label sets, no high-cardinality or free-form text).

Example Python “lint” for metric descriptors (run in CI before CreateMetricDescriptor):

from google.cloud import monitoring_v3

ALLOWED_PREFIXES = ["custom.googleapis.com/security/", "custom.googleapis.com/ops/"]
FORBIDDEN_LABEL_TYPES = {"STRING"}  # e.g. disallow free-text for some classes
REQUIRED_FIREWALL_LABELS = {"source_ip", "destination_ip", "action"}

def validate_descriptor(md: monitoring_v3.MetricDescriptor):
    if not any(md.type.startswith(p) for p in ALLOWED_PREFIXES):
        raise ValueError(f"Disallowed metric prefix: {md.type}")

    # Example: enforce required labels for firewall metrics
    if md.type == "custom.googleapis.com/security/firewall_denies":
        label_keys = {l.key for l in md.labels}
        missing = REQUIRED_FIREWALL_LABELS - label_keys
        if missing:
            raise ValueError(f"Missing labels for firewall metric: {missing}")

    # Optionally restrict label value types
    for l in md.labels:
        if l.value_type.name in FORBIDDEN_LABEL_TYPES:
            raise ValueError(f"Forbidden label type {l.value_type.name} on {l.key}")

# Use in your pipeline before creating:
# client.create_metric_descriptor(name=project_name, metric_descriptor=md)

GCP Introduction

GCP Pricing

GCP Threats

GCP Misconfigurations

Resources

GCP Threats

Google.monitoring.v3.metricservice.createmetricdescriptor, remediation

Event Information

Examples

Remediation

Using Console

Using CLI

Using Python

​Event Information

​Examples

​Remediation

​Using Console

​Using CLI

​Using Python

Event Information

Examples

Remediation

Using Console

Using CLI

Using Python