S3 Sync
Paul Campbell
8c89cc2489
* [domain] SizeTranslation includes decimals for larger sizes * [core] MD5HashGenerator rewrite for memory efficiency No longer attempt to create an Array the size of the file to be parsed. Now it creates a single small buffer and reads 8kb chunks in at a time. Only creating an additional smaller buffer to read the tail of the file. Remove methods to parsing only part of a file are they were no longer used, and remove the relevant tests. |
||
|---|---|---|
| .github | ||
| aws-api/src/main/scala/net/kemitix/s3thorp/aws/api | ||
| aws-lib/src | ||
| cli/src/main/scala/net/kemitix/s3thorp/cli | ||
| core/src | ||
| domain/src | ||
| project | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
s3thorp
Synchronisation of files with S3 using the hash of the file contents.
https://api.codacy.com/project/badge/Grade/14ea6ad0825249c994a27a82d3485180
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
s3thorp
Usage: s3thorp [options]
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-v, --verbose <value> Verbosity level (1-5)
The --include and --exclude parameters can be used more than once.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |