Service limits remediation

Using Console

Here’s how to remediate GCP IAM “Service Limits” (quota) issues using the GCP Console, focusing on IAM-related quotas (service accounts, roles, policy bindings, etc.).

1. Identify Which IAM Limit You’re Hitting

Go to IAM & Admin → Quotas in the GCP Console
- URL: https://console.cloud.google.com/iam-admin/quotas
In the Services filter, select:
- IAM Service Account Credentials API, Identity and Access Management (IAM) API, and/or Cloud Resource Manager API, depending on what you’re working with.
Use the Metric filter to locate relevant quotas, for example:
- Service accounts per project
- Custom roles per organization/project
- Policy size / policy bindings per resource
- API requests per minute/hour
Check which quota shows Usage close to or at Limit or is marked as Blocked.

2. Short-Term Remediation: Reduce Usage (Clean Up IAM Resources)

Depending on which limit is impacted:

A. If you hit service account limits

Go to IAM & Admin → Service Accounts
- https://console.cloud.google.com/iam-admin/serviceaccounts
Sort or filter by Last used (if available) or by Name/Description to identify unused accounts.
For each unused service account:
- Click the three dots (⋮) → Delete.
If apps can share accounts:
- Consolidate multiple similar service accounts into fewer, well-scoped service accounts.
- Update workloads (GCE, GKE, Cloud Run, etc.) to use shared accounts with least-privilege roles.

B. If you hit custom role limits

Go to IAM & Admin → Roles
- https://console.cloud.google.com/iam-admin/roles
Filter for Custom roles.
Identify:
- Roles that are deprecated, not used, or duplicate.
For unused custom roles:
- Click the role → Delete (or Disable first if you want a safe test).
Merge overlapping custom roles where possible so fewer custom roles are needed.

C. If you hit policy size / bindings limits

Go to IAM & Admin → IAM
- https://console.cloud.google.com/iam-admin/iam
Look for:
- Members with many roles, or roles with many members.
Reduce bindings:
- Replace many individual users with Google Groups and assign roles to the group instead.
- Remove obsolete members or roles that are no longer required.
- Prefer predefined roles instead of several overlapping custom roles.

D. If you hit IAM API request rate limits

Reduce automation chattiness:
- Batch IAM changes where possible.
- Avoid scripts that repeatedly call setIamPolicy / getIamPolicy in tight loops.
Space out calls:
- Implement backoff and retry in tooling, if you control it.

3. Long-Term Remediation: Request a Quota Increase

Go to IAM & Admin → Quotas
- https://console.cloud.google.com/iam-admin/quotas
Filter by Service and Metric to locate the specific quota you’re exceeding.
Check the box next to the quota you want to increase.
At the top, click EDIT QUOTAS.
In the side panel:
- Select a Support case channel (you may need a support plan).
- Fill in:
  - New limit you’re requesting.
  - Justification (describe workloads, growth, why cleanup alone isn’t enough).
Submit the request and monitor the status in your Support section.

4. Validate After Remediation

Re-run the action that previously failed (e.g., create service account, add IAM binding).
Revisit IAM & Admin → Quotas to confirm:
- Usage is below limit, or
- New higher limit is applied.
Optionally, set up internal processes:
- Periodic review of service accounts, roles, and bindings.
- Use Groups and standardized roles to control IAM sprawl.

If you share the exact quota metric you’re hitting (e.g., “Service accounts per project” or “Policy size”), I can give exact click-by-click steps tailored to that limit.

Using CLI

For IAM on GCP, “Service Limits” (quotas) can’t actually be raised directly via gcloud; increases are requested via the web console or a support case. With the CLI you can:

Identify what limit you’re hitting
Reduce usage (cleanup / refactor IAM)
Then formally request a quota increase in the console

Below are the concrete CLI steps to diagnose and remediate IAM-related service limits.

1. Identify which IAM limit you’re hitting

Common IAM limits include:

Max bindings per policy (e.g., 1,500 bindings on a project)
Max roles per principal
Max custom roles per project/org
Permissions per custom role

Use logs or error messages from failed commands (such as gcloud projects add-iam-policy-binding, gcloud iam roles create, etc.) and note the exact error text, e.g.:

RESOURCE_EXHAUSTED
Quota exceeded
exceeds the maximum number of bindings
Maximum number of custom roles reached

There isn’t a direct gcloud command to show IAM policy limits, but you can:

# Show current project IAM policy size (bindings count, etc.)
gcloud projects get-iam-policy PROJECT_ID --format=json > policy.json

# Count bindings
jq '.bindings | length' policy.json

If the bindings count is near/above 1500, you’re hitting the IAM policy bindings limit.

2. Remediate by reducing IAM policy size (clean up via CLI)

2.1 List IAM policy bindings

gcloud projects get-iam-policy PROJECT_ID \
  --format="table(bindings.role, bindings.members)"

This lets you see:

Roles that are overused or duplicated
Direct user/service account bindings that can be replaced by groups

2.2 Remove unused or redundant bindings

Get current policy:

gcloud projects get-iam-policy PROJECT_ID --format=json > policy.json

Edit policy.json:
- Remove bindings you no longer need.
- Consolidate multiple bindings where possible (e.g., same role with multiple members → one binding with combined members).

Example binding to remove (in policy.json):

{
  "role": "roles/viewer",
  "members": [
    "user:olduser@example.com"
  ]
}

Delete that block or remove individual members.

Set the updated policy:

gcloud projects set-iam-policy PROJECT_ID policy.json

Repeat the same pattern for:

gcloud organizations get-iam-policy / set-iam-policy
gcloud folders get-iam-policy / set-iam-policy

2.3 Replace individual bindings with group-based bindings

Create or use an existing Google Group in your org (done in Admin Console, not gcloud).
Replace many individual user bindings with a single group binding.

Example: Replace:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:alice@example.com" \
  --role="roles/viewer"

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:bob@example.com" \
  --role="roles/viewer"

With one group binding:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="group:viewers@example.com" \
  --role="roles/viewer"

Then remove the per-user bindings as in 2.2.

2.4 Use custom roles to reduce per-binding permission sprawl

If you’re hitting “permissions per custom role” or similar:

Inspect a custom role:

gcloud iam roles describe ROLE_ID --project=PROJECT_ID

Reduce permissions if there are many rarely-used ones:

gcloud iam roles update ROLE_ID \
  --project=PROJECT_ID \
  --permissions=perm1,perm2,perm3

If you have many near-identical custom roles, consolidate them into fewer roles with shared permissions.

3. Remediate “too many custom roles” limit

If error mentions maximum custom roles per project/org:

List all custom roles:

# Project-level
gcloud iam roles list --project=PROJECT_ID --filter="stage=GA"

# Org-level
gcloud iam roles list --organization=ORG_ID --filter="stage=GA"

Identify unused ones (for example, by last used in your own CMDB / internal docs; GCP doesn’t give per-role usage counts directly via gcloud).
Disable roles you no longer need:

gcloud iam roles update ROLE_ID \
  --project=PROJECT_ID \
  --stage=DISABLED

After validation, delete them:

gcloud iam roles delete ROLE_ID --project=PROJECT_ID

Repeat at org level with --organization=ORG_ID.

4. Request a quota increase (cannot be done via gcloud)

Once you’ve optimized, if you still hit limits you must request a quota increase:

Go to:
https://console.cloud.google.com/iam-admin/quotas
Filter for:
- “IAM”
- Or the specific quota mentioned in your error (e.g., “Policy size”, “Custom roles per project”).
Select the quota → “EDIT QUOTAS” → fill out:
- Project
- Contact details
- New requested limit
- Justification
Submit and wait for approval.

If you have a support plan, you can also open a support case and reference the error and project ID.

Summary (CLI-only actions)

Use gcloud projects/organizations/folders get-iam-policy to inspect policy size and bindings.
Clean up and consolidate IAM bindings by editing JSON and applying via set-iam-policy.
Use group-based IAM instead of many per-user bindings.
Consolidate and clean up custom roles via gcloud iam roles list/update/delete.
Any actual increase in the service limit must be requested via the GCP Console or support, not gcloud.

Using Python

“Service limits” for GCP IAM usually means you’re hitting or about to hit hard limits like:

Max role bindings per policy (e.g., 1,500 bindings)
Max members per binding
API rate limits / quotas for iam.googleapis.com

You can’t change these limits with code, but you can remediate by:

detecting when you’re close to limits
cleaning up unused / duplicate IAM bindings
batching / optimizing calls to stay under API quota

Below is a practical, step‑by‑step approach using Python.

1. Set up Python environment

pip install google-api-python-client google-auth google-auth-httplib2 google-cloud-monitoring
gcloud auth application-default login

Make sure your ADC (Application Default Credentials) has permission:

roles/resourcemanager.projectIamAdmin for modifying project IAM
roles/monitoring.viewer (optional for monitoring quotas)

2. Detect IAM policy “bloat” (bindings/members near limits)

Example: check all IAM bindings on a project and warn if near a threshold.

from google.oauth2 import service_account
from googleapiclient import discovery
import google.auth

# Use ADC
credentials, project_id = google.auth.default()

service = discovery.build('cloudresourcemanager', 'v1', credentials=credentials)

def get_iam_policy(project_id):
    request = service.projects().getIamPolicy(
        resource=project_id,
        body={"options": {"requestedPolicyVersion": 3}}
    )
    return request.execute()

def analyze_policy(policy, binding_threshold=1200, members_threshold=100):
    bindings = policy.get("bindings", [])
    num_bindings = len(bindings)
    print(f"Total bindings: {num_bindings}")

    if num_bindings > binding_threshold:
        print(f"WARNING: Total bindings ({num_bindings}) close to limit (e.g. 1500).")

    for b in bindings:
        role = b.get("role")
        members = b.get("members", [])
        if len(members) > members_threshold:
            print(f"Role {role} has {len(members)} members (high, may hit member limits).")

policy = get_iam_policy(project_id)
analyze_policy(policy)

This doesn’t fix anything yet; it just tells you where you’re likely to hit service limits.

3. Remediate by cleaning up IAM bindings

Typical cleanups:

Remove members that no longer exist (users, service accounts)
Remove duplicate members
Remove roles that are no longer needed

Below is a pattern to:

remove specific members from specific roles
then write back the cleaned policy

from googleapiclient.errors import HttpError

def remove_members_from_roles(policy, members_to_remove_by_role):
    """
    members_to_remove_by_role example:
    {
      "roles/editor": [
        "user:old.user@example.com",
        "serviceAccount:old-sa@PROJECT_ID.iam.gserviceaccount.com",
      ]
    }
    """
    new_bindings = []

    for b in policy.get("bindings", []):
        role = b["role"]
        members = b.get("members", [])

        if role in members_to_remove_by_role:
            to_remove = set(members_to_remove_by_role[role])
            new_members = [m for m in members if m not in to_remove]
            if new_members:
                new_bindings.append({"role": role, "members": new_members})
            else:
                # Entire binding removed (no members left)
                continue
        else:
            new_bindings.append(b)

    policy["bindings"] = new_bindings
    return policy

def set_iam_policy(project_id, policy):
    request = service.projects().setIamPolicy(
        resource=project_id,
        body={"policy": policy}
    )
    return request.execute()

# EXAMPLE USAGE:
members_to_remove_by_role = {
    "roles/editor": [
        "user:old.user@example.com",
        "serviceAccount:old-sa@%s.iam.gserviceaccount.com" % project_id
    ]
}

policy = get_iam_policy(project_id)
clean_policy = remove_members_from_roles(policy, members_to_remove_by_role)

try:
    updated = set_iam_policy(project_id, clean_policy)
    print("Updated IAM policy written.")
except HttpError as e:
    print(f"Error updating IAM policy: {e}")

You can extend this logic to:

drop bindings where the role is no longer used in your org
move many individual users to a Google group and grant the role to the group instead (reduces members per binding)

4. Avoid IAM API quota limits (rate limits)

If your scanner / automation is hitting IAM rate limits, you need to:

Batch operations when possible
Use exponential backoff on 429 / 503 responses

Example of a simple exponential backoff wrapper:

import time
from googleapiclient.errors import HttpError

def with_backoff(callable_fn, max_retries=5, initial_delay=1.0, multiplier=2.0):
    delay = initial_delay
    for i in range(max_retries):
        try:
            return callable_fn()
        except HttpError as e:
            if e.resp.status in (429, 500, 503):
                if i == max_retries - 1:
                    raise
                time.sleep(delay)
                delay *= multiplier
            else:
                raise

# Example: safe getIamPolicy
def safe_get_iam_policy(project_id):
    return with_backoff(
        lambda: service.projects().getIamPolicy(
            resource=project_id,
            body={"options": {"requestedPolicyVersion": 3}}
        ).execute()
    )

5. Monitor IAM quotas via Cloud Monitoring (optional)

You can track metrics like iam.googleapis.com/quota/allocation/usage and alert when near quota.Basic example to list IAM quota metrics:

from google.cloud import monitoring_v3

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{project_id}"

for descriptor in client.list_metric_descriptors(name=project_name):
    if "iam.googleapis.com" in descriptor.type:
        print(descriptor.type)

You would typically configure alerts in the console, but you can automate that with the Monitoring API if desired.

6. When you truly hit a hard service limit

If you hit hard IAM structural limits (e.g., max bindings per policy):
- Consolidate: use groups instead of many individual members
- Split: move resources into additional projects / folders so policy is distributed
If you hit quota limits (API calls, QPS):
- Optimize code (batch, cache, backoff)
- Then request a quota increase in the console (IAM & Admin → Quotas), which cannot be done purely via Python.

If you can share:

which exact IAM service limit you’re hitting (bindings, members, API QPS, etc.) I can give you a more targeted Python snippet tailored to that specific limit.

Using Terraform

# There is no Terraform resource that changes IAM-related service limits (quotas) in GCP.
# Quota increases must be requested via the Google Cloud Console or gcloud, not Terraform.

# Example: you can define IAM policies and bindings in Terraform, but not the IAM service limits.

resource "google_project_iam_binding" "example" {
  project = "YOUR_PROJECT_ID"   # Replace with your GCP project ID
  role    = "roles/viewer"

  members = [
    "user:USER_EMAIL@example.com",  # Replace with the user/service account/etc.
  ]
}

GCP IAM service limits (e.g., number of roles per principal, policy size, etc.) are enforced by Google and are not configurable via Terraform or the Google provider; to remediate limit issues you must request a quota/limit increase in the Cloud Console (IAM & Admin → Quotas) or via gcloud services quotas where available. terraform plan cannot show a quota change because it is outside Terraform’s control.

GCP Introduction

GCP Pricing

GCP Threats

GCP Misconfigurations

Resources

GCP Threats

Service limits remediation

Triage and Remediation

Remediation

1. Identify Which IAM Limit You’re Hitting

2. Short-Term Remediation: Reduce Usage (Clean Up IAM Resources)

A. If you hit service account limits

B. If you hit custom role limits

C. If you hit policy size / bindings limits

D. If you hit IAM API request rate limits

3. Long-Term Remediation: Request a Quota Increase

4. Validate After Remediation

1. Identify which IAM limit you’re hitting

2. Remediate by reducing IAM policy size (clean up via CLI)

2.1 List IAM policy bindings

2.2 Remove unused or redundant bindings

2.3 Replace individual bindings with group-based bindings

2.4 Use custom roles to reduce per-binding permission sprawl

3. Remediate “too many custom roles” limit

4. Request a quota increase (cannot be done via gcloud)

Summary (CLI-only actions)

1. Set up Python environment

2. Detect IAM policy “bloat” (bindings/members near limits)

3. Remediate by cleaning up IAM bindings

4. Avoid IAM API quota limits (rate limits)

5. Monitor IAM quotas via Cloud Monitoring (optional)

6. When you truly hit a hard service limit

​Triage and Remediation

​Remediation

​1. Identify Which IAM Limit You’re Hitting

​2. Short-Term Remediation: Reduce Usage (Clean Up IAM Resources)

​A. If you hit service account limits

​B. If you hit custom role limits

​C. If you hit policy size / bindings limits

​D. If you hit IAM API request rate limits

​3. Long-Term Remediation: Request a Quota Increase

​4. Validate After Remediation

​1. Identify which IAM limit you’re hitting

​2. Remediate by reducing IAM policy size (clean up via CLI)

​2.1 List IAM policy bindings

​2.2 Remove unused or redundant bindings

​2.3 Replace individual bindings with group-based bindings

​2.4 Use custom roles to reduce per-binding permission sprawl

​3. Remediate “too many custom roles” limit

​4. Request a quota increase (cannot be done via gcloud)

​Summary (CLI-only actions)

​1. Set up Python environment

​2. Detect IAM policy “bloat” (bindings/members near limits)

​3. Remediate by cleaning up IAM bindings

​4. Avoid IAM API quota limits (rate limits)

​5. Monitor IAM quotas via Cloud Monitoring (optional)

​6. When you truly hit a hard service limit

Triage and Remediation

Remediation

1. Identify Which IAM Limit You’re Hitting

2. Short-Term Remediation: Reduce Usage (Clean Up IAM Resources)

A. If you hit service account limits

B. If you hit custom role limits

C. If you hit policy size / bindings limits

D. If you hit IAM API request rate limits

3. Long-Term Remediation: Request a Quota Increase

4. Validate After Remediation

1. Identify which IAM limit you’re hitting

2. Remediate by reducing IAM policy size (clean up via CLI)

2.1 List IAM policy bindings

2.2 Remove unused or redundant bindings

2.3 Replace individual bindings with group-based bindings

2.4 Use custom roles to reduce per-binding permission sprawl

3. Remediate “too many custom roles” limit

4. Request a quota increase (cannot be done via gcloud)

Summary (CLI-only actions)

1. Set up Python environment

2. Detect IAM policy “bloat” (bindings/members near limits)

3. Remediate by cleaning up IAM bindings

4. Avoid IAM API quota limits (rate limits)

5. Monitor IAM quotas via Cloud Monitoring (optional)

6. When you truly hit a hard service limit