community / community.aws / 5.1.0 / module / s3_sync Efficiently upload multiple files to S3 | "added in version" 1.0.0 of community.aws" Authors: Ted Timmons (@tedder)community.aws.s3_sync (5.1.0) — module
Install with ansible-galaxy collection install community.aws:==5.1.0
collections: - name: community.aws version: 5.1.0
The S3 module is great, but it is very slow for a large volume of files- even a dozen will be noticeable. In addition to speed, it handles globbing, inclusions/exclusions, mime types, expiration mapping, recursion, cache control and smart directory mapping.
- name: basic upload community.aws.s3_sync: bucket: tedder file_root: roles/s3/files/
- name: basic upload using the glacier storage class community.aws.s3_sync: bucket: tedder file_root: roles/s3/files/ storage_class: GLACIER
- name: basic individual file upload community.aws.s3_sync: bucket: tedder file_root: roles/s3/files/file_name
- name: all the options community.aws.s3_sync: bucket: tedder file_root: roles/s3/files mime_map: .yml: application/text .json: application/text key_prefix: config_files/web file_change_strategy: force permission: public-read cache_control: "public, max-age=31536000" storage_class: "GLACIER" include: "*" exclude: "*.txt,.*"
mode: choices: - push default: push description: - sync direction. type: str bucket: description: - Bucket name. required: true type: str delete: default: false description: - Remove remote files that exist in bucket but are not present in the file root. required: false type: bool region: aliases: - aws_region - ec2_region description: - The AWS region to use. - For global services such as IAM, Route53 and CloudFront, I(region) is ignored. - The C(AWS_REGION) or C(EC2_REGION) environment variables may also be used. - See the Amazon AWS documentation for more information U(http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region). - The C(ec2_region) alias has been deprecated and will be removed in a release after 2024-12-01 - Support for the C(EC2_REGION) environment variable has been deprecated and will be removed in a release after 2024-12-01. type: str exclude: default: .* description: - Shell pattern-style file matching. - Used after include to remove files (for instance, skip C("*.txt")) - For multiple patterns, comma-separate them. required: false type: str include: default: '*' description: - Shell pattern-style file matching. - Used before exclude to determine eligible files (for instance, only C("*.gif")) - For multiple patterns, comma-separate them. required: false type: str profile: aliases: - aws_profile description: - A named AWS profile to use for authentication. - See the AWS documentation for more information about named profiles U(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - The C(AWS_PROFILE) environment variable may also be used. - The I(profile) option is mutually exclusive with the I(aws_access_key), I(aws_secret_key) and I(security_token) options. type: str mime_map: description: - 'Dict entry from extension to MIME type. This will override any default/sniffed MIME type. For example C({".txt": "application/text", ".yml": "application/text"}) ' required: false type: dict file_root: description: - File/directory path for synchronization. This is a local path. - This root path is scrubbed from the key name, so subdirectories will remain as keys. required: true type: path access_key: aliases: - aws_access_key_id - aws_access_key - ec2_access_key description: - AWS access key ID. - See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). - The C(AWS_ACCESS_KEY_ID), C(AWS_ACCESS_KEY) or C(EC2_ACCESS_KEY) environment variables may also be used in decreasing order of preference. - The I(aws_access_key) and I(profile) options are mutually exclusive. - The I(aws_access_key_id) alias was added in release 5.1.0 for consistency with the AWS botocore SDK. - The I(ec2_access_key) alias has been deprecated and will be removed in a release after 2024-12-01. - Support for the C(EC2_ACCESS_KEY) environment variable has been deprecated and will be removed in a release after 2024-12-01. type: str aws_config: description: - A dictionary to modify the botocore configuration. - Parameters can be found in the AWS documentation U(https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore.config.Config). type: dict key_prefix: default: '' description: - In addition to file path, prepend s3 path with this prefix. Module will add slash at end of prefix if necessary. required: false type: str permission: choices: - private - public-read - public-read-write - authenticated-read - aws-exec-read - bucket-owner-read - bucket-owner-full-control description: - Canned ACL to apply to synced files. - Changing this ACL only changes newly synced files, it does not trigger a full reupload. required: false type: str secret_key: aliases: - aws_secret_access_key - aws_secret_key - ec2_secret_key description: - AWS secret access key. - See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). - The C(AWS_SECRET_ACCESS_KEY), C(AWS_SECRET_KEY), or C(EC2_SECRET_KEY) environment variables may also be used in decreasing order of preference. - The I(secret_key) and I(profile) options are mutually exclusive. - The I(aws_secret_access_key) alias was added in release 5.1.0 for consistency with the AWS botocore SDK. - The I(ec2_secret_key) alias has been deprecated and will be removed in a release after 2024-12-01. - Support for the C(EC2_SECRET_KEY) environment variable has been deprecated and will be removed in a release after 2024-12-01. type: str endpoint_url: aliases: - ec2_url - aws_endpoint_url - s3_url description: - URL to connect to instead of the default AWS endpoints. While this can be used to connection to other AWS-compatible services the amazon.aws and community.aws collections are only tested against AWS. - The C(AWS_URL) or C(EC2_URL) environment variables may also be used, in decreasing order of preference. - The I(ec2_url) and I(s3_url) aliases have been deprecated and will be removed in a release after 2024-12-01. - Support for the C(EC2_URL) environment variable has been deprecated and will be removed in a release after 2024-12-01. type: str aws_ca_bundle: description: - The location of a CA Bundle to use when validating SSL certificates. - The C(AWS_CA_BUNDLE) environment variable may also be used. type: path cache_control: default: '' description: - Cache-Control header set on uploaded objects. - Directives are separated by commas. required: false type: str session_token: aliases: - aws_session_token - security_token - aws_security_token - access_token description: - AWS STS session token for use with temporary credentials. - See the AWS documentation for more information about access tokens U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). - The C(AWS_SESSION_TOKEN), C(AWS_SECURITY_TOKEN) or C(EC2_SECURITY_TOKEN) environment variables may also be used in decreasing order of preference. - The I(security_token) and I(profile) options are mutually exclusive. - Aliases I(aws_session_token) and I(session_token) were added in release 3.2.0, with the parameter being renamed from I(security_token) to I(session_token) in release 6.0.0. - The I(security_token), I(aws_security_token), and I(access_token) aliases have been deprecated and will be removed in a release after 2024-12-01. - Support for the C(EC2_SECRET_KEY) and C(AWS_SECURITY_TOKEN) environment variables has been deprecated and will be removed in a release after 2024-12-01. type: str storage_class: choices: - STANDARD - REDUCED_REDUNDANCY - STANDARD_IA - ONEZONE_IA - INTELLIGENT_TIERING - GLACIER - DEEP_ARCHIVE - OUTPOSTS default: STANDARD description: - Storage class to be associated to each object added to the S3 bucket. required: false type: str version_added: 1.5.0 version_added_collection: community.aws validate_certs: default: true description: - When set to C(false), SSL certificates will not be validated for communication with the AWS APIs. - Setting I(validate_certs=false) is strongly discouraged, as an alternative, consider setting I(aws_ca_bundle) instead. type: bool file_change_strategy: choices: - force - checksum - date_size default: date_size description: - Difference determination method to allow changes-only syncing. Unlike rsync, files are not patched- they are fully skipped or fully uploaded. - date_size will upload if file sizes don't match or if local file modified date is newer than s3's version - checksum will compare etag values based on s3's implementation of chunked md5s. - force will always upload all files. required: false type: str debug_botocore_endpoint_logs: default: false description: - Use a C(botocore.endpoint) logger to parse the unique (rather than total) C("resource:action") API calls made during a task, outputing the set to the resource_actions key in the task results. Use the C(aws_resource_action) callback to output to total list made during a playbook. - The C(ANSIBLE_DEBUG_BOTOCORE_LOGS) environment variable may also be used. type: bool
filelist_actionable: description: file listing (dicts) of files that will be uploaded after the strategy decision returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json mime_type: application/json modified_epoch: 1477931256 s3_path: s3sync/policy.json whysize: 151 / 151 whytime: 1477931256 / 1477929260 type: list filelist_initial: description: file listing (dicts) from initial globbing returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json modified_epoch: 1477416706 type: list filelist_local_etag: description: file listing (dicts) including calculated local etag returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json mime_type: application/json modified_epoch: 1477416706 s3_path: s3sync/policy.json type: list filelist_s3: description: file listing (dicts) including information about previously-uploaded versions returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json mime_type: application/json modified_epoch: 1477416706 s3_path: s3sync/policy.json type: list filelist_typed: description: file listing (dicts) with calculated or overridden mime types returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json mime_type: application/json modified_epoch: 1477416706 type: list uploads: description: file listing (dicts) of files that were actually uploaded returned: always sample: - bytes: 151 chopped_path: policy.json fullpath: roles/cf/files/policy.json s3_path: s3sync/policy.json whysize: 151 / 151 whytime: 1477931637 / 1477931489 type: list