kemitix/thorp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
S3 Sync
145 commits 1 branch 18 tags 14 MiB
Java 100%
Find a file
Paul Campbell 44c66c042c
Include and Exclude behave more like the AWS CLI (#48) 
* [domain] rename Filter as Include

* [cli]ParseArgs allow exclude and include parameters to be repeated

* [core] don't include include/exclude details in logging

* [domain] combine Include and Exclude into Filter

Config now collect includes and Excludes into a single list and passed
each file to the Filter.isIncluded method, with the list of Filters,
to determine if a file should be included.
2019-06-08 20:31:20 +01:00
.github [github] Add stale configuration 2019-05-14 07:05:48 +01:00
aws-api/src/main/scala/net/kemitix/s3thorp/aws/api Split into subprojects (#36) 2019-06-06 19:24:15 +01:00
aws-lib/src [core] MD5HashGenerator uses IO to return where there is file IO (#47) 2019-06-08 18:19:15 +01:00
cli/src/main Include and Exclude behave more like the AWS CLI (#48) 2019-06-08 20:31:20 +01:00
core/src Include and Exclude behave more like the AWS CLI (#48) 2019-06-08 20:31:20 +01:00
domain/src Include and Exclude behave more like the AWS CLI (#48) 2019-06-08 20:31:20 +01:00
project [gitignote] update to allow some project files 2019-05-11 08:54:35 +01:00
.gitignore [gitignore] ignore zip files 2019-05-14 07:27:14 +01:00
.travis.yml [travis] define AWS_REGION environment variable 2019-05-16 19:28:50 +01:00
build.sbt Update aws-java-sdk-s3 to 1.11.568 (#42) 2019-06-08 08:21:07 +01:00
CHANGELOG.org Support multiple filters (#18) 2019-05-23 19:35:48 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
README.org [readme] add note about broken native images 2019-05-30 18:38:23 +01:00

README.org

s3thorp

Synchronisation of files with S3 using the hash of the file contents.

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage 

  s3thorp
  Usage: s3thorp [options]

    -s, --source <value>             Source directory to sync to S3
    -b, --bucket <value>             S3 bucket name
    -p, --prefix <value>             Prefix within the S3 Bucket
    -x, --exclude <value>[,<values>] Exclude matching paths
    -v, --verbose <value>            Verbosity level (1-5)

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Creating Native Images

Note: the created image currently can't be run outside of the base of the project. See Issue #15

  • Download and install GraalVM

  • Install native-image using the graal updater

      gu install native-image

  • Create native image

      native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \
               -H:Name=s3thorp \
               -H:Class=net.kemitix.s3thorp.Main \
               --allow-incomplete-classpath \
               --force-fallback
  • Resulting file requires a JDK for execution