Spot Event Plugin - Multiple Region Support¶
Overview¶
Originally, the Spot Event Plugin only allowed Spot Fleets to be launched and managed in one AWS Region.
In Deadline v10.2.1 and higher an option has been added to launch and manage Spot Fleets in multiple AWS Regions from the same Spot Event Plugin.
Prerequisites¶
You may need to increase the House Cleaning Interval in Configure Repository Options to be longer than the default 60 seconds to ensure that there is enough time for the Spot Event Plugin to manage all configured AWS regions before the next iteration of OnHouseCleaning.
Warning
Depending on the setup of your render farm there may be additional costs for cross AWS Region data transfers and communications when using Multi-Region mode.
Configuring Multi-Region Mode¶
Warning
Before switching to Multi-Region mode ensure any active Spot Fleet Requests are terminated in the AWS console. You should ensure that there are no Spot Fleet Requests or Spot Event Plugin managed EC2 Instances running before switching to Multi-Region mode.
To switch the Spot Event Plugin to run in Multi-Region mode you need to do the following:
Enable Tools > Power User mode in the Monitor’s menu.
Select Tools > Configure Events from the Monitor’s menu.
Select the Spot entry from the list on the left.
Change the Region setting to Multi-Region
EC2 Instance and Worker Naming¶
Warning
Deadline Workers should always have unique names. The name applied to a Worker launched by the Spot Event Plugin will by default be the hostname of the EC2 instance (defaults to the EC2 instance’s private IP address). We highly recommend that you use one of the following options to ensure that all launched Deadline Workers have unique names.
Option 1: Set unique/non-overlapping CIDR blocks across AWS Regions for all subnets that the Spot Event Plugin will launch EC2 instances in. Setting unique CIDR blocks will ensure that the private IP addresses/hostnames of all launched EC2 instances will be unique across AWS Regions and therefore, all Deadline Worker names. Please note by default, CIDR blocks are identical in the default VPC wizard in each AWS region; which will cause a clash in Deadline Worker names if ignored.
OR
Option 2: Use Launch Templates in your Spot Fleet Request configuration and set the Hostname type to Resource name. This will make it so that all EC2 instances launched with the Launch Template use the resource name of the EC2 instance as the hostname instead of the private IP address. EC2 instance resource names are unique across AWS Regions.
Warning
If using Launch Templates you must add the following key/value pair as a Tag in the Launch Template.
DeadlineTrackedAWSResource : SpotEventPluginFailure to add this Tag will result in EC2 Instances being launched that won’t be managed/monitored by the Resource Tracker if it’s enabled.
It’s possible that if this Tag is not included, EC2 Instances that can’t connect to the repository may become stuck and won’t shut themselves down.
It’s crucial that this Tag is added to prevent large overspends in your AWS Account that could otherwise be prevented by the Resource Tracker and other mechanisms.
Note
See here for more information on EC2 instance hostnames.
Multi Region Spot Fleet Configuration¶
Using Multi-Region mode requires that the Spot Fleet Request configuration JSON is in a different format than what is used for Single-Region mode. See here for Spot Fleet Request configuration examples when using Single-Region mode.
The Spot Fleet Request configuration for Multi-Region mode is still a JSON dictionary that is similar to the format for Single-Region mode, but with additional JSON keys.
In Multi-Region mode the Spot Fleet configuration is still a one-to-one mapping between Deadline Groups and Spot Fleet Request Configuration, with the additional requirement that a Deadline Group can only be used in one AWS Region.
To create a Multi-Region Spot Fleet configuration, you will need one or more previously created Spot Fleet Requests.
A Multi-Region Spot Fleet configuration has the form:
[
{
"region": "AWS_REGION_NAME",
"group_configs": {
"DEADLINE_GROUP_NAME": <Replace with Spot Fleet Request>,
"DEADLINE_GROUP_NAME_2": <Replace with Spot Fleet Request 2>
}
},
{
"region": "AWS_REGION_NAME_2",
"group_configs": {
"DEADLINE_GROUP_NAME_3": <Replace with Spot Fleet Request 3>,
"DEADLINE_GROUP_NAME_4": <Replace with Spot Fleet Request 4>,
"DEADLINE_GROUP_NAME_5": <Replace with Spot Fleet Request 5>
}
}
]
Note
The order of the AWS regions in the Spot Fleet Request configuration JSON is the order in which they will be managed by the Spot Event Plugin. This means that Workers in the AWS regions will have access to resources like License Limits with priority based on the order that the AWS region appears in the JSON configuration.
Example Multi Region Spot Fleet Configurations¶
Example Multi-Region Configuration¶
[
{
"region": "us-west-2",
"group_configs": {
"sep_group_1_us_west_2": {
"IamFleetRole": "arn:aws:iam::385970627561:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-09T23:26:23.000Z",
"ValidUntil": "2024-02-09T23:26:23.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"SpotPrice": "0.1591",
"LaunchSpecifications": [
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "c5.4xlarge",
"SpotPrice": "0.105"
},
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "m5.4xlarge",
"SpotPrice": "0.096"
},
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "r5.4xlarge",
"SpotPrice": "0.096"
}
]
},
"sep_group_2_us_west_2": {
"IamFleetRole": "arn:aws:iam::357432474442:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-09T23:26:23.000Z",
"ValidUntil": "2024-02-09T23:26:23.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"SpotPrice": "0.1591",
"LaunchSpecifications": [
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "c5.4xlarge",
"SpotPrice": "0.105"
},
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "m5.4xlarge",
"SpotPrice": "0.096"
},
{
"ImageId": "ami-0143f8d6",
"KeyName": "key_pair",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-0269d441",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-025fbc72",
"DeleteOnTermination": true,
"Groups": [
"sg-051250fc"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "r5.4xlarge",
"SpotPrice": "0.096"
}
]
}
}
},
{
"region": "us-east-1",
"group_configs": {
"sep_group_1_us_east_1": {
"IamFleetRole": "arn:aws:iam::357432474442:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-10T00:18:59.000Z",
"ValidUntil": "2024-02-10T00:18:59.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"SpotPrice": "0.166",
"LaunchSpecifications": [
{
"ImageId": "ami-05bb48a63a3cd2b9d",
"KeyName": "SEP_Testing_US_EAST_1",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-02ca95518f592b256",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"TagSpecifications": [
{
"ResourceType": "instance",
"Tags": [
{
"Key": "DeadlineAWSTrackedResource",
"Value": "SpotEventPlugin"
}
]
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-071c65ca18cc3d951",
"DeleteOnTermination": true,
"Groups": [
"sg-0a226f1a98d5f8a6d"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "c5.4xlarge",
"SpotPrice": "0.105"
},
{
"ImageId": "ami-05bb48a63a3cd2b9d",
"KeyName": "SEP_Testing_US_EAST_1",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-02ca95518f592b256",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"TagSpecifications": [
{
"ResourceType": "instance",
"Tags": [
{
"Key": "DeadlineAWSTrackedResource",
"Value": "SpotEventPlugin"
}
]
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-071c65ca18cc3d951",
"DeleteOnTermination": true,
"Groups": [
"sg-0a226f1a98d5f8a6d"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "m5.4xlarge",
"SpotPrice": "0.085"
},
{
"ImageId": "ami-05bb48a63a3cd2b9d",
"KeyName": "SEP_Testing_US_EAST_1",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-02ca95518f592b256",
"VolumeSize": 64,
"VolumeType": "gp2",
"Encrypted": false
}
}
],
"TagSpecifications": [
{
"ResourceType": "instance",
"Tags": [
{
"Key": "DeadlineAWSTrackedResource",
"Value": "SpotEventPlugin"
}
]
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::357432474442:instance-profile/DeadlineSpotWorker"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-071c65ca18cc3d951",
"DeleteOnTermination": true,
"Groups": [
"sg-0a226f1a98d5f8a6d"
],
"AssociatePublicIpAddress": true
}
],
"InstanceType": "r5.4xlarge",
"SpotPrice": "0.096"
}
]
}
}
}
]
Example Multi-Region Configuration Using Launch Templates¶
Warning
If using Launch Templates you must add the following key/value pair as a Tag in the Launch Template.
DeadlineTrackedAWSResource : SpotEventPlugin
Failure to add this Tag will result in EC2 Instances being launched that won’t be managed/monitored by the Resource Tracker if it’s enabled.
It’s possible that if this Tag is not included, EC2 Instances that can’t connect to the repository may become stuck and won’t shut themselves down.
It’s crucial that this Tag is added to prevent large overspends in your AWS Account that could otherwise be prevented by the Resource Tracker and other mechanisms.
[
{
"region": "us-west-2",
"group_configs": {
"group_name_1": {
"IamFleetRole": "arn:aws:iam::357432474442:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-08T22:48:24.000Z",
"ValidUntil": "2024-02-08T22:48:24.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"OnDemandAllocationStrategy": "lowestPrice",
"LaunchSpecifications": [],
"LaunchTemplateConfigs": [
{
"LaunchTemplateSpecification": {
"LaunchTemplateId": "lt-0d4626a3",
"Version": "$Latest"
},
"Overrides": [
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "c5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "r5.4xlarge",
"WeightedCapacity": 1
}
]
}
]
},
"group_name_2": {
"IamFleetRole": "arn:aws:iam::357432474442:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-08T22:48:24.000Z",
"ValidUntil": "2024-02-08T22:48:24.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"OnDemandAllocationStrategy": "lowestPrice",
"LaunchSpecifications": [],
"LaunchTemplateConfigs": [
{
"LaunchTemplateSpecification": {
"LaunchTemplateId": "lt-0d4626a3",
"Version": "$Latest"
},
"Overrides": [
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "c5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "r5.4xlarge",
"WeightedCapacity": 1
}
]
}
]
}
}
},
{
"region": "us-east-1",
"group_configs": {
"group_name_3": {
"IamFleetRole": "arn:aws:iam::357432474442:role/aws-ec2-spot-fleet-tagging-role",
"AllocationStrategy": "capacityOptimized",
"TargetCapacity": 0,
"ValidFrom": "2023-02-08T22:47:02.000Z",
"ValidUntil": "2024-02-08T22:47:02.000Z",
"TerminateInstancesWithExpiration": true,
"Type": "maintain",
"OnDemandAllocationStrategy": "lowestPrice",
"LaunchSpecifications": [],
"LaunchTemplateConfigs": [
{
"LaunchTemplateSpecification": {
"LaunchTemplateId": "lt-074b5b3d",
"Version": "$Latest"
},
"Overrides": [
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "c5.4xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "r5.4xlarge",
"WeightedCapacity": 1
}
]
}
]
}
}
}
]
FAQ¶
Can all AWS Regions be used with Multi-Region mode?
No. At this time the AWS regions that can be used with Multi-Region mode are the same as in the dropdown for the Region setting. See here for how to access the Spot Event Plugin configuration where the Region setting is available.
Can Wildcards be used in Group Names with Multi-Region mode?
Yes. Wildcards can be used with Multi-Region mode the same as with Single-Region mode. See Wildcards In Group Names for more details.
Warning
Deadline Groups can’t be configured to be used for more than one AWS region in Multi-Region mode. Ensure that any Wildcards will not result in duplicate Deadline Groups being used across AWS regions.
Can Deadline Pools be used with Multi-Region mode?
Yes. Deadline Pools can be used in Multi-Region mode the same as in Single-Region mode. See Spot Fleet Request Group Pools for details.
