Spot Event Plugin¶
Overview¶
The Spot Event Plugin can scale cloud-based Amazon EC2 Spot instances dynamically based on the queued Jobs and Tasks in the Deadline Database.
It associates JSON dictionary Spot Fleet Requests with named Deadline Worker Groups, allowing multiple Spot Fleets with different hardware and software specifications to be launched for different types of Jobs based on their Group assignment.
What Is Amazon EC2 Spot?¶
Amazon EC2 Spot Instances let you take advantage of unused EC2 capacity in the Amazon Web Services cloud at steep discounts.
EC2 Spot instances are identical to On-Demand EC2 instances of the same type and size, except for the following details:
- The cost can be up to 90% lower than On-Demand instances.
The cost varies slightly between AWS Regions, Availability Zones (AZ), instance families and sizes, and time of day based on supply and demand.
The cost can never exceed the On-Demand price.
Take a look at the AWS Spot pricing page for current pricing.
- If capacity is insufficient to provide On-Demand instances to customers, AWS can terminate Spot instances.
If an instance is reclaimed by AWS within the first hour, you will not be charged for that hour.
After the first hour, you won’t be charged for a partial hour of a reclaimed Windows instance, while a Linux instance will be charged for the usage in seconds.
Another Spot Instance from a different capacity pool (different AZ, different instance size or instance type) specified in the Spot Fleet Request may replace the reclaimed instance, maintaining the requested target capacity.
See also
You can learn more about Spot Instances on the AWS Spot Instance Website.
Lifecycle¶
The Spot Event Plugin controls the lifecycle of multiple Spot Fleet Requests by scaling up and down the Target Capacity of the Spot Fleets.
In order to maintain the Spot Fleet Requests, the Spot Event Plugin triggers after House Cleaning.
When the Event triggers, the Spot Event Plugin determines the number of Workers that could be actively working at the same time for each Group.
Next, for each configured Group it will create or modify a Spot Fleet Request that will contain a number of instances equal to the available work up to the maximum capacity set by the user.
Once the Spot Fleet Requests have been fulfilled and the Workers have started, they will automatically assign themselves to the Group they were created for and optionally one or more Pools.
Finally, when the Worker is either no longer rendering, or the Spot instance is going to be interrupted, the Worker will
Mark itself as Offline,
Optionally remove itself from the Workers list,
Terminate itself.
Determining Targets¶
For every Group you have configured, the Spot Event Plugin will scale the target number of instances based on the state of your farm during House Cleaning. Here are some of the factors it takes into account:
State: Only Queued Tasks are considered eligible to work on.
Allow Lists: If a job has an allow list then no Spot instances will be started for it.
Limits: The number of available Limit stubs you have will constrain the number of instances to start.
Machine Limits: The machine limits for individual jobs are taken into account for determining the available work for that job.
Concurrent Tasks: If Concurrent Tasks are enabled on a Job and on its Plugin, then they will be taken into account.
Note
The Spot Event Plugin assumes that each AMI is configured to only start 1 Worker. Due to this, we treat Worker Level limits the same as Machine Level Limits.
Warning
If you select an instance type with fewer CPU’s than the number of concurrent tasks for your jobs, you may not receive the number of instances you are expecting. This is done because Deadline can cap the number of Concurrent Tasks to the number of CPUs on your machine.
Resource Tracker¶
The Resource Tracker is an optional, but highly recommended, service that monitors the health of instances started by Deadline in your AWS account.
It monitors a heartbeat reported by each render node and terminates any instances that are in an unhealthy state.
If enough instances are failing, the Resource Tracker will cancel the Spot Fleet.
Prerequisites¶
Before you can use the Spot Event Plugin, you will need the following:
An AWS account. Create one if necessary.
A Remote Connection Server (RCS) or Pulse set up as part of your render farm.
Warning
All SEP components: Workers, RCS, or Pulse instances MUST be configured with UTC time format.
Warning
Ensure “House Cleaning Interval” in Configure Repository Options -> “House Cleaning” is NOT set below 30 seconds (default=60 seconds) and the setting: “Allow Workers to Perform House Cleaning If Pulse is not Running” is disabled as shown below in Configure Repository Options. Consider running at least one other instance of Pulse on another machine for redundancy.
If you are using Multiple Regions you may need to set the OnHouseCleaning Interval longer than 60 seconds to ensure that there is enough time for the Spot Event Plugin to manage all configured AWS regions before the next iteration of OnHouseCleaning.
Setup¶
The following setup is required in order to use the Spot Event Plugin :
Event Plugin Configuration¶
After creating your IAM Policies and creating your Spot Fleet, you are now ready to configure the Spot Event Plugin.
To perform this step, enter Super User mode and select Tools > Configure Events from the Monitor’s menu. Then select the Spot entry from the list on the left.
As part of configuring the Spot Event Plugin, you will need to configure the following sections:
An additional Spot Event Configuration Utility can be used to edit existing Spot Fleet Request Configurations, or create Configurations from existing ones.
Security¶
Additonal Information¶
FAQ¶
Can the Spot Event Plugin be used in combination with the Deadline Balancer?
No. The Deadline Balancer and Spot Event plugin can technically be run at the same time but they do not complement each other. They will create Workers for the same tasks.
Does the AWS Portal work with this Spot Event Plugin?
No. Both components can technically be run at the same time but they do not complement each other.
Can the Spot Event Plugin be used in combination with the Amazon Cloud Plugin?
No. The Amazon Cloud Plugin starts EC2 On-Demand/Reserved instances whereas the Spot Event Plugin uses the heavily discounted EC2 Spot instances for compute.
Does the Spot Event Plugin take Deadline ‘Pools’ into account when calculating number of EC2 Spot Instances to request?
No. You should use Deadline Groups to configure which Deadline jobs are run on Spot instances in AWS EC2. However, you can configure an order based, comma-separated string of the Pool names you would like to be assigned to all the Deadline Workers which are started on EC2 Spot for the relevant Deadline Group that made the Spot request.
Can I use Limits to control the number of Spot instances provisioned by the Spot Event Plugin?
Yes. Deadline Resource and License Limits at the app plugin and job level are taken into account when calculating the number of Spot instances to be requested per Deadline Group configured in your Spot Fleet JSON configuration file.
Do I need software licenses (floating or Usage-based) to render with Spot Event Plugin?
Deadline does not require any licenses for it to run. You will still need licenses for your rendering software if they require one.
