Paul Campbell
8d0c3e23c9
* [aws-api] TTFI UploadProgressLogging#logTransfer * [aws-api] TTFI UploadProgressLogging#logRequestCycle * [aws-api] TTFI UploadProgressLogging#logByteTransfer * [aws-api] TTFI UploadProgressListener * [aws-lib] TTFI UploaderLogging * [aws-api] UploadProgressListener refactoring * [aws-api] UploadProgressListener remove IO/Monad This class is a wrapper for a callback method used by the AWS SDK. Unfortunately you can't get the IO() created when that listener is called by the SDK, so unless we manually run unsafeRunSync, as we have done previously, it would never be executed. This removes the IO monad entirely and simply runs the code when the callback is triggered. * [aws-lib] S3ClientLogging remove unused method * [aws-lib] TTFI S3ClientLogging * [aws-lib] TTFI S3ClientCopier * [aws-lib] TTFI S3ClientObjectLister * [aws-lib] TTFI Uploader * [aws-lib] TTFI S3ClientDeleter * [aws-api] TTFI S3Client * [aws-lib] TTFI S3ClientBuilder and ThorpS3Client * [core] TTFI ActionSubmitter * [cli] TTFI Logger * [core] TTFI MD5HashGenerator * [core] TTFI LocalFileStream * [core] Sync refactoring * [core] TTFI Sync * [aws-lib] S3ObjectsByHashSuite truncate lastmodified to match Date * [aws-lib] ThorpS3ClientSuite truncate lastmodified to match Date * [core] MD5HashGeneratorTest switch to Id from IO * [sbt] restrict cats-effect to cli module, cats-core elsewhere * [core] MD5HashGenerator collapse lines |
||
|---|---|---|
| .github | ||
| aws-api/src/main/scala/net/kemitix/s3thorp/aws/api | ||
| aws-lib/src | ||
| bin | ||
| cli/src/main/scala/net/kemitix/s3thorp/cli | ||
| core/src | ||
| domain/src | ||
| project | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
s3thorp
Synchronisation of files with S3 using the hash of the file contents.
file:https://img.shields.io/codacy/grade/14ea6ad0825249c994a27a82d3485180.svg?style=for-the-badge
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
s3thorp
Usage: s3thorp [options]
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-v, --verbose <value> Verbosity level (1-5)
The --include and --exclude parameters can be used more than once.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Executable JAR
To build as an executable jar, perform `sbt assembly`
This will create the file `cli/target/scala-2.12/s3thorp-assembly-$VERSION.jar` (where $VERSION is substituted)
Copy and rename this file as `s3thorp.jar` into the same directory as the `bin/s3throp` shell script.