S3 Sync
Paul Campbell
aa7fb1eb24
* [sbt] add scalamock as a test dependency * [aws-lib]SyncSuite: minor layout changes * [aws-lib]SyncSuite: remove test * [core] move SyncSuite to same module as subject it tests * [aws-lib]ThorpS3Client: remove commented lines * [aws-lib] remove PutObject versions of Uploader * [aws-lib] rename to TransferManager to remove Multi-part from name * [aws-lib]TransferManager: change logging prefix * [aws-lib] convert logging classes to objects * [aws-lib] convert ObjectLister to use V1 SDK * [aws-lib] convert Copier to use V1 SDK * [aws-lib] extract S3ObjectsBy{Hash,Key} to objects * [aws-lib]S3ClientSuite: rewrite test using mocks * [aws-lib]TransferManager rewrite using for-comprehension * [aws-lib]Copier: remote bucket name from target remote key * [aws-lib]TransferManager: refactor logging to use IO themselves * [aws-lib] Remove test class MyAmazonS3 * [aws-lib]ObjectLister: optimise imports * [aws-lib] S3ClientSuite remove commented code * [aws-lib]ThropS3ClientSuite update to V1 api * [aws-lib]S3ClientSuite: make test as pending It works okay on its own, but when run as part of a suite it fails. Will look at this again once all V2 SDK is removed. * [aws-lib] convert Deleter to use V1 SDK * [aws-lib] Client Logging remove redundant braces * [aws-lib] stop injecting the V2 SDK * [sbt] remove v2 SDK dependencies * [aws-lib] remove redundant helpers for v2 SDK * [sbt] upgrade aws jackson dependencies The jackson libraries used by AWS have security flaws, but are Java 6 compatible, which AWS want to preserve. * [aws-lib] clean up TransferManager tests |
||
---|---|---|
.github | ||
aws-api/src/main/scala/net/kemitix/s3thorp/aws/api | ||
aws-lib/src | ||
cli/src/main | ||
core/src | ||
domain/src | ||
project | ||
.gitignore | ||
.travis.yml | ||
build.sbt | ||
CHANGELOG.org | ||
README.org |
s3thorp
Synchronisation of files with S3 using the hash of the file contents.
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ...
command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
s3thorp Usage: s3thorp [options] -s, --source <value> Source directory to sync to S3 -b, --bucket <value> S3 bucket name -p, --prefix <value> Prefix within the S3 Bucket -x, --exclude <value>[,<values>] Exclude matching paths -v, --verbose <value> Verbosity level (1-5)
Behaviour
When considering a local file, the following table governs what should happen:
# | local file | remote key | hash of same key | hash of other keys | action |
1 | exists | exists | matches | - | do nothing |
2 | exists | is missing | - | matches | copy from other key |
3 | exists | is missing | - | no matches | upload |
4 | exists | exists | no match | matches | copy from other key |
5 | exists | exists | no match | no matches | upload |
6 | is missing | exists | - | - | delete |
Creating Native Images
Note: the created image currently can't be run outside of the base of the project. See Issue #15
-
Download and install GraalVM
-
Install
native-image
using the graal updatergu install native-image
-
Create native image
native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \ -H:Name=s3thorp \ -H:Class=net.kemitix.s3thorp.Main \ --allow-incomplete-classpath \ --force-fallback
- Resulting file requires a JDK for execution