Try SpotterEfficiently upload multiple files to S3
| "added in version" 1.0.0 of community.aws"
Authors: Ted Timmons (@tedder)
Install with ansible-galaxy collection install community.aws:==5.1.0
collections:
- name: community.aws
version: 5.1.0The S3 module is great, but it is very slow for a large volume of files- even a dozen will be noticeable. In addition to speed, it handles globbing, inclusions/exclusions, mime types, expiration mapping, recursion, cache control and smart directory mapping.
- name: basic upload
community.aws.s3_sync:
bucket: tedder
file_root: roles/s3/files/
- name: basic upload using the glacier storage class
community.aws.s3_sync:
bucket: tedder
file_root: roles/s3/files/
storage_class: GLACIER
- name: basic individual file upload
community.aws.s3_sync:
bucket: tedder
file_root: roles/s3/files/file_name
- name: all the options
community.aws.s3_sync:
bucket: tedder
file_root: roles/s3/files
mime_map:
.yml: application/text
.json: application/text
key_prefix: config_files/web
file_change_strategy: force
permission: public-read
cache_control: "public, max-age=31536000"
storage_class: "GLACIER"
include: "*"
exclude: "*.txt,.*"
mode:
choices:
- push
default: push
description:
- sync direction.
type: str
bucket:
description:
- Bucket name.
required: true
type: str
delete:
default: false
description:
- Remove remote files that exist in bucket but are not present in the file root.
required: false
type: bool
region:
aliases:
- aws_region
- ec2_region
description:
- The AWS region to use.
- For global services such as IAM, Route53 and CloudFront, I(region) is ignored.
- The C(AWS_REGION) or C(EC2_REGION) environment variables may also be used.
- See the Amazon AWS documentation for more information U(http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region).
- The C(ec2_region) alias has been deprecated and will be removed in a release after
2024-12-01
- Support for the C(EC2_REGION) environment variable has been deprecated and will
be removed in a release after 2024-12-01.
type: str
exclude:
default: .*
description:
- Shell pattern-style file matching.
- Used after include to remove files (for instance, skip C("*.txt"))
- For multiple patterns, comma-separate them.
required: false
type: str
include:
default: '*'
description:
- Shell pattern-style file matching.
- Used before exclude to determine eligible files (for instance, only C("*.gif"))
- For multiple patterns, comma-separate them.
required: false
type: str
profile:
aliases:
- aws_profile
description:
- A named AWS profile to use for authentication.
- See the AWS documentation for more information about named profiles U(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- The C(AWS_PROFILE) environment variable may also be used.
- The I(profile) option is mutually exclusive with the I(aws_access_key), I(aws_secret_key)
and I(security_token) options.
type: str
mime_map:
description:
- 'Dict entry from extension to MIME type. This will override any default/sniffed
MIME type. For example C({".txt": "application/text", ".yml": "application/text"})
'
required: false
type: dict
file_root:
description:
- File/directory path for synchronization. This is a local path.
- This root path is scrubbed from the key name, so subdirectories will remain as keys.
required: true
type: path
access_key:
aliases:
- aws_access_key_id
- aws_access_key
- ec2_access_key
description:
- AWS access key ID.
- See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
- The C(AWS_ACCESS_KEY_ID), C(AWS_ACCESS_KEY) or C(EC2_ACCESS_KEY) environment variables
may also be used in decreasing order of preference.
- The I(aws_access_key) and I(profile) options are mutually exclusive.
- The I(aws_access_key_id) alias was added in release 5.1.0 for consistency with the
AWS botocore SDK.
- The I(ec2_access_key) alias has been deprecated and will be removed in a release
after 2024-12-01.
- Support for the C(EC2_ACCESS_KEY) environment variable has been deprecated and will
be removed in a release after 2024-12-01.
type: str
aws_config:
description:
- A dictionary to modify the botocore configuration.
- Parameters can be found in the AWS documentation U(https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore.config.Config).
type: dict
key_prefix:
default: ''
description:
- In addition to file path, prepend s3 path with this prefix. Module will add slash
at end of prefix if necessary.
required: false
type: str
permission:
choices:
- private
- public-read
- public-read-write
- authenticated-read
- aws-exec-read
- bucket-owner-read
- bucket-owner-full-control
description:
- Canned ACL to apply to synced files.
- Changing this ACL only changes newly synced files, it does not trigger a full reupload.
required: false
type: str
secret_key:
aliases:
- aws_secret_access_key
- aws_secret_key
- ec2_secret_key
description:
- AWS secret access key.
- See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
- The C(AWS_SECRET_ACCESS_KEY), C(AWS_SECRET_KEY), or C(EC2_SECRET_KEY) environment
variables may also be used in decreasing order of preference.
- The I(secret_key) and I(profile) options are mutually exclusive.
- The I(aws_secret_access_key) alias was added in release 5.1.0 for consistency with
the AWS botocore SDK.
- The I(ec2_secret_key) alias has been deprecated and will be removed in a release
after 2024-12-01.
- Support for the C(EC2_SECRET_KEY) environment variable has been deprecated and will
be removed in a release after 2024-12-01.
type: str
endpoint_url:
aliases:
- ec2_url
- aws_endpoint_url
- s3_url
description:
- URL to connect to instead of the default AWS endpoints. While this can be used
to connection to other AWS-compatible services the amazon.aws and community.aws
collections are only tested against AWS.
- The C(AWS_URL) or C(EC2_URL) environment variables may also be used, in decreasing
order of preference.
- The I(ec2_url) and I(s3_url) aliases have been deprecated and will be removed in
a release after 2024-12-01.
- Support for the C(EC2_URL) environment variable has been deprecated and will be
removed in a release after 2024-12-01.
type: str
aws_ca_bundle:
description:
- The location of a CA Bundle to use when validating SSL certificates.
- The C(AWS_CA_BUNDLE) environment variable may also be used.
type: path
cache_control:
default: ''
description:
- Cache-Control header set on uploaded objects.
- Directives are separated by commas.
required: false
type: str
session_token:
aliases:
- aws_session_token
- security_token
- aws_security_token
- access_token
description:
- AWS STS session token for use with temporary credentials.
- See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
- The C(AWS_SESSION_TOKEN), C(AWS_SECURITY_TOKEN) or C(EC2_SECURITY_TOKEN) environment
variables may also be used in decreasing order of preference.
- The I(security_token) and I(profile) options are mutually exclusive.
- Aliases I(aws_session_token) and I(session_token) were added in release 3.2.0, with
the parameter being renamed from I(security_token) to I(session_token) in release
6.0.0.
- The I(security_token), I(aws_security_token), and I(access_token) aliases have been
deprecated and will be removed in a release after 2024-12-01.
- Support for the C(EC2_SECRET_KEY) and C(AWS_SECURITY_TOKEN) environment variables
has been deprecated and will be removed in a release after 2024-12-01.
type: str
storage_class:
choices:
- STANDARD
- REDUCED_REDUNDANCY
- STANDARD_IA
- ONEZONE_IA
- INTELLIGENT_TIERING
- GLACIER
- DEEP_ARCHIVE
- OUTPOSTS
default: STANDARD
description:
- Storage class to be associated to each object added to the S3 bucket.
required: false
type: str
version_added: 1.5.0
version_added_collection: community.aws
validate_certs:
default: true
description:
- When set to C(false), SSL certificates will not be validated for communication with
the AWS APIs.
- Setting I(validate_certs=false) is strongly discouraged, as an alternative, consider
setting I(aws_ca_bundle) instead.
type: bool
file_change_strategy:
choices:
- force
- checksum
- date_size
default: date_size
description:
- Difference determination method to allow changes-only syncing. Unlike rsync, files
are not patched- they are fully skipped or fully uploaded.
- date_size will upload if file sizes don't match or if local file modified date is
newer than s3's version
- checksum will compare etag values based on s3's implementation of chunked md5s.
- force will always upload all files.
required: false
type: str
debug_botocore_endpoint_logs:
default: false
description:
- Use a C(botocore.endpoint) logger to parse the unique (rather than total) C("resource:action")
API calls made during a task, outputing the set to the resource_actions key in the
task results. Use the C(aws_resource_action) callback to output to total list made
during a playbook.
- The C(ANSIBLE_DEBUG_BOTOCORE_LOGS) environment variable may also be used.
type: bool
filelist_actionable:
description: file listing (dicts) of files that will be uploaded after the strategy
decision
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
mime_type: application/json
modified_epoch: 1477931256
s3_path: s3sync/policy.json
whysize: 151 / 151
whytime: 1477931256 / 1477929260
type: list
filelist_initial:
description: file listing (dicts) from initial globbing
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
modified_epoch: 1477416706
type: list
filelist_local_etag:
description: file listing (dicts) including calculated local etag
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
mime_type: application/json
modified_epoch: 1477416706
s3_path: s3sync/policy.json
type: list
filelist_s3:
description: file listing (dicts) including information about previously-uploaded
versions
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
mime_type: application/json
modified_epoch: 1477416706
s3_path: s3sync/policy.json
type: list
filelist_typed:
description: file listing (dicts) with calculated or overridden mime types
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
mime_type: application/json
modified_epoch: 1477416706
type: list
uploads:
description: file listing (dicts) of files that were actually uploaded
returned: always
sample:
- bytes: 151
chopped_path: policy.json
fullpath: roles/cf/files/policy.json
s3_path: s3sync/policy.json
whysize: 151 / 151
whytime: 1477931637 / 1477931489
type: list