Sagemaker Training Jobs Should Have Network Isolation Enabled

Using Console

Here’s how to ensure SageMaker Training Jobs have Network Isolation enabled using the AWS Management Console:

1. Understand the Limitation

You cannot edit an existing completed/running training job to enable network isolation.
You must create a new training job (or update the template/pipeline/notebook code that creates it) with Network isolation turned on.

2. Enable Network Isolation When Creating a Training Job

Sign in to AWS Console
- Go to: https://console.aws.amazon.com/
- Switch to the region where you run SageMaker.
Open SageMaker
- In the Services menu, choose Amazon SageMaker.
Go to Training Jobs
- In the left navigation pane, under Training, click Training jobs.
- To recreate an existing job with isolation, select that job and click Create similar (if available in your console version), or note down its configuration to re-enter.
Create a New Training Job
- Click Create training job (or Create → Training job, depending on UI version).
Fill in Basic Settings
- Job name: Provide a unique name.
- IAM role: Select a role with needed permissions (S3, logs, etc.).
- Configure Algorithm / Container, Input data configuration, Output data configuration, etc. as you normally would.
Enable Network Isolation
- Scroll to the Network or Security section (label may vary slightly by console version).
- Find the option usually named:
  - Network isolation, or
  - Enable network isolation
- Check the box or choose Enabled.
- This corresponds to setting EnableNetworkIsolation = true for the training job.
(Optional) Configure VPC Settings
- In the same Network section, you can also:
  - Select a VPC, Subnets, and Security groups if you need training to run in a private subnet.
- This is separate from network isolation, but commonly used together.
Review and Create
- Review all configurations.
- Click Create training job.

This new training job will now run with network isolation: the container won’t have outbound network access except for what SageMaker itself needs for the job lifecycle.

3. Ongoing Remediation

For any place you define training jobs (SageMaker Studio UI, notebook code, CI/CD pipelines, SageMaker Pipelines):
- Ensure EnableNetworkIsolation = true is always set.
Over time, deprecate old jobs / code that created training jobs without network isolation.

Using CLI

For SageMaker training jobs, network isolation is enabled per training job and cannot be turned on for an already-running/completed job. You must submit new jobs with network isolation enabled.Below are the key steps using AWS CLI.

1. Create a training job with network isolation enabled

You can either use a JSON config file or pass parameters inline. The important flag is:

--enable-network-isolation

Option A: Using a JSON config file

Create a file training-job-config.json:

{
  "TrainingJobName": "my-training-job-net-isolated",
  "AlgorithmSpecification": {
    "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
    "TrainingInputMode": "File"
  },
  "RoleArn": "arn:aws:iam::123456789012:role/SageMakerExecutionRole",
  "InputDataConfig": [
    {
      "ChannelName": "training",
      "DataSource": {
        "S3DataSource": {
          "S3DataType": "S3Prefix",
          "S3Uri": "s3://my-bucket/training-data/",
          "S3DataDistributionType": "FullyReplicated"
        }
      },
      "ContentType": "text/csv"
    }
  ],
  "OutputDataConfig": {
    "S3OutputPath": "s3://my-bucket/output/"
  },
  "ResourceConfig": {
    "InstanceType": "ml.m5.large",
    "InstanceCount": 1,
    "VolumeSizeInGB": 50
  },
  "StoppingCondition": {
    "MaxRuntimeInSeconds": 3600
  },
  "EnableNetworkIsolation": true
}

Create the job via CLI:

aws sagemaker create-training-job \
  --cli-input-json file://training-job-config.json

"EnableNetworkIsolation": true in the JSON is equivalent to --enable-network-isolation on the CLI.

Option B: Passing parameters inline

aws sagemaker create-training-job \
  --training-job-name my-training-job-net-isolated \
  --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest,TrainingInputMode=File \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --input-data-config '[{
      "ChannelName":"training",
      "DataSource":{
        "S3DataSource":{
          "S3DataType":"S3Prefix",
          "S3Uri":"s3://my-bucket/training-data/",
          "S3DataDistributionType":"FullyReplicated"
        }
      }
  }]' \
  --output-data-config S3OutputPath=s3://my-bucket/output/ \
  --resource-config InstanceType=ml.m5.large,InstanceCount=1,VolumeSizeInGB=50 \
  --stopping-condition MaxRuntimeInSeconds=3600 \
  --enable-network-isolation

2. Ensure pipelines / automations use the flag

Wherever training jobs are created (scripts, CI/CD, Step Functions, SageMaker Pipelines, etc.), confirm:

They use enable_network_isolation=True (SDK)
or
--enable-network-isolation / "EnableNetworkIsolation": true (CLI/JSON).

This makes all future training jobs compliant.

Using Python

To remediate this, you must (a) ensure all new training jobs are created with network isolation, and (b) stop/recreate any existing non‑isolated jobs (you cannot “flip” isolation on a running job).Below are step‑by‑step instructions using Python.

1. Using the low-level `boto3` SageMaker client

a. Create training jobs with network isolation

import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")  # adjust region

training_job_name = "my-training-job-with-network-isolation"

response = sm.create_training_job(
    TrainingJobName=training_job_name,
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
        "TrainingInputMode": "File",
    },
    RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    InputDataConfig=[
        {
            "ChannelName": "training",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://my-bucket/training-data/",
                    "S3DataDistributionType": "FullyReplicated",
                }
            },
            "ContentType": "text/csv",
        }
    ],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/output/",
    },
    ResourceConfig={
        "InstanceType": "ml.m5.xlarge",
        "InstanceCount": 1,
        "VolumeSizeInGB": 50,
    },
    StoppingCondition={
        "MaxRuntimeInSeconds": 3600,
    },
    EnableNetworkIsolation=True,  # <<< THIS IS THE KEY SETTING
)
print("Created training job:", response["TrainingJobArn"])

Key field:
EnableNetworkIsolation=True must be set on every create_training_job call.

b. Recreate existing training jobs with network isolation

import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")

old_job_name = "my-old-training-job"

# 1. Describe old job
desc = sm.describe_training_job(TrainingJobName=old_job_name)

# 2. Build new job request with EnableNetworkIsolation=True
new_job_name = old_job_name + "-isolated"

create_args = {
    "TrainingJobName": new_job_name,
    "AlgorithmSpecification": desc["AlgorithmSpecification"],
    "RoleArn": desc["RoleArn"],
    "InputDataConfig": desc.get("InputDataConfig", []),
    "OutputDataConfig": desc["OutputDataConfig"],
    "ResourceConfig": desc["ResourceConfig"],
    "StoppingCondition": desc["StoppingCondition"],
    "EnableNetworkIsolation": True,
}

# Optional fields to carry over if present
for k in [
    "HyperParameters",
    "VpcConfig",
    "Tags",
    "EnableManagedSpotTraining",
    "CheckpointConfig",
    "DebugHookConfig",
    "DebugRuleConfigurations",
    "ProfilerConfig",
    "ProfilerRuleConfigurations",
    "Environment",
    "RetryStrategy",
]:
    if k in desc:
        create_args[k] = desc[k]

resp = sm.create_training_job(**create_args)
print("Created new isolated job:", resp["TrainingJobArn"])

Then stop or let the original non‑isolated job complete and only use the new isolated one going forward.

2. Using the high-level `sagemaker` Python SDK

If you use the SageMaker Python SDK (pip install sagemaker):

import sagemaker
from sagemaker.estimator import Estimator

session = sagemaker.Session()
role = "arn:aws:iam::123456789012:role/SageMakerExecutionRole"

estimator = Estimator(
    image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size=50,
    output_path="s3://my-bucket/output/",
    sagemaker_session=session,
    enable_network_isolation=True,  # <<< REQUIRED
)

estimator.fit(
    inputs={
        "training": "s3://my-bucket/training-data/"
    }
)

Here enable_network_isolation=True ensures EnableNetworkIsolation=True in the underlying training job.

3. Policy / guardrail (optional)

To enforce this org‑wide, use:

AWS Config rule sagemaker-training-job-network-isolation-enabled
Or a custom rule that checks EnableNetworkIsolation on training jobs
and alerts/blocks when it’s not True.

But from a Python remediation standpoint, the essential change is always: set EnableNetworkIsolation=True on all training job creations and recreate any existing ones without it.

AWS Introduction

AWS Pricing

AWS Threats

AWS Misconfigurations

Sagemaker Training Jobs Should Have Network Isolation Enabled

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Understand the Limitation

2. Enable Network Isolation When Creating a Training Job

3. Ongoing Remediation

1. Create a training job with network isolation enabled

Option A: Using a JSON config file

Option B: Passing parameters inline

2. Ensure pipelines / automations use the flag

1. Using the low-level `boto3` SageMaker client

a. Create training jobs with network isolation

b. Recreate existing training jobs with network isolation

2. Using the high-level `sagemaker` Python SDK

3. Policy / guardrail (optional)

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation

​1. Understand the Limitation

​2. Enable Network Isolation When Creating a Training Job

​3. Ongoing Remediation

​1. Create a training job with network isolation enabled

​Option A: Using a JSON config file

​Option B: Passing parameters inline

​2. Ensure pipelines / automations use the flag

​1. Using the low-level boto3 SageMaker client

​a. Create training jobs with network isolation

​b. Recreate existing training jobs with network isolation

​2. Using the high-level sagemaker Python SDK

​3. Policy / guardrail (optional)

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Understand the Limitation

2. Enable Network Isolation When Creating a Training Job

3. Ongoing Remediation

1. Create a training job with network isolation enabled

Option A: Using a JSON config file

Option B: Passing parameters inline

2. Ensure pipelines / automations use the flag

1. Using the low-level `boto3` SageMaker client

a. Create training jobs with network isolation

b. Recreate existing training jobs with network isolation

2. Using the high-level `sagemaker` Python SDK

3. Policy / guardrail (optional)