Skip to content

Commit b0986fb

Browse files
authored
added AWS Batch and SM notebook examples (#4856)
1 parent 9c7f77c commit b0986fb

File tree

20 files changed

+80008
-0
lines changed

20 files changed

+80008
-0
lines changed
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Amazon SageMaker Job Queuing, powered by AWS Batch
2+
3+
AWS Batch enables efficient queuing and resource management for your SageMaker Training Jobs.
4+
5+
## Getting Started
6+
7+
The instructions below are designed to get you going with this feature quickly.
8+
9+
### Setup IAM permissions
10+
11+
The example notebooks require permissions to invoke AWS Batch APIs.
12+
Below is a sample IAM policy granting these permissions - this should be added to the role being used to execute the notebooks (which the notebooks use as both the role to invoke AWS Batch and the role passed to SageMaker for training execution).
13+
14+
```json
15+
{
16+
"Version": "2012-10-17",
17+
"Statement": [
18+
{
19+
"Effect": "Allow",
20+
"Action": ["batch:*"],
21+
"Resource": "*"
22+
},
23+
{
24+
"Effect": "Allow",
25+
"Action": ["iam:CreateServiceLinkedRole"],
26+
"Resource": "arn:aws:iam::*:role/*AWSServiceRoleForAWSBatchWithSagemaker",
27+
"Condition": {
28+
"StringEquals": {
29+
"iam:AWSServiceName": "sagemaker-queuing.batch.amazonaws.com"
30+
}
31+
}
32+
},
33+
{
34+
"Effect": "Allow",
35+
"Action": "iam:PassRole",
36+
"Resource": "*",
37+
"Condition": {
38+
"StringEquals": {
39+
"iam:PassedToService": ["sagemaker.amazonaws.com"]
40+
}
41+
}
42+
}
43+
]
44+
}
45+
```
46+
47+
### Python setup
48+
49+
In order to use the feature, the python `boto3` library needs to be installed.
50+
51+
```
52+
pip install -U boto3 sagemaker
53+
```
54+
55+
### Create AWS Batch queues
56+
57+
To run the [examples](./examples) provided, the Batch queue need to be created. Refer to [smtj_batch_utils](./smtj_batch_utils/README.md) for additional information.

build_and_train_models/sm-training-queues-pytorch/examples/estimator-pytorch-cpu/data/data.csv

Lines changed: 37153 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)