If you’re new to AWS Portal we recommend starting here. If you’re new to Deadline we recommend starting here.
The AWS Portal Asset Transfer System (Advanced) - Storage¶
An asset file could be stored in more than one location:
The S3 bucket. The uploads of on-prem files to the S3 bucket occur when the S3-Backed Cache Client is requesting them or AWS Portal Asset Server tries to pre-cache them. The files that have been uploaded to the S3 bucket are persistent unless you explicitly remove them.
The EBS Volume of the S3-Backed Cache Client. The downloads of files onto the EBS volume occurs when the S3-Backed Cache Client tries to download a file from the S3 bucket. It could be that more than one S3-Backed Cache Client has a copy of the exact same file. (The main reason for this is allow multiple spot instances to render the same job in parallel.) The files that have been downloaded onto the EBS volume may or may not get removed after a render.
Note
An EBS Volume and the S3-Backed Cache Client reside on the Gateway instance. This is for backwards-compatibility with Windows AWS Portal AMIs prior to version 10.0.26.0. For such legacy AMIs, Windows instances mounted a Samba share hosted on the Gateway instance that handled the asset transfer to/from the S3 bucket.
Security Concerns¶
All the components use HTTPS to transfer the asset files between themselves and the S3 bucket. These components are only allowed to upload/download the files to/from the S3 bucket.
Note
To be clear, the components themselves don’t have any permissions assigned to them. The EC2 instances where these components deployed have these permissions.
The metadata of the files are stored in the databases maintained by the Worker Controller and by the S3 Backed Cache Central Controller.
Logging¶
Access Logs for the S3 bucket are stored in a corresponding logging bucket. Logs will persist there for 60 days by default. To change that value, follow these instructions on How to set a Lifecycle Policy.
Versioning¶
By default, versioning is not enabled on the S3-Backed Cache. If you’d like to enable versioning, or subsequently suspend versioning, follow these instructions.
