S3 Sync
Find a file
Paul Campbell fe4bec0f12
Handle when a file is not found after initial scan (#49)
* [aws-lib] Uploader refactoring

* [aws-lib] Uploader remove redundant braces

* [aws-lib] Uploader start upload inside IO

The upload method starts the upload to S3, so should be within the IO.

* [core] log count of errors

* [aws-lib] Uploader handle errors and count them
2019-06-10 21:22:46 +01:00
.github [github] Add stale configuration 2019-05-14 07:05:48 +01:00
aws-api/src/main/scala/net/kemitix/s3thorp/aws/api Improve upload logging (#44) 2019-06-10 19:45:36 +01:00
aws-lib/src Handle when a file is not found after initial scan (#49) 2019-06-10 21:22:46 +01:00
cli/src/main/scala/net/kemitix/s3thorp/cli Improve upload logging (#44) 2019-06-10 19:45:36 +01:00
core/src Handle when a file is not found after initial scan (#49) 2019-06-10 21:22:46 +01:00
domain/src Improve upload logging (#44) 2019-06-10 19:45:36 +01:00
project [gitignote] update to allow some project files 2019-05-11 08:54:35 +01:00
.gitignore [gitignore] ignore zip files 2019-05-14 07:27:14 +01:00
.travis.yml [travis] define AWS_REGION environment variable 2019-05-16 19:28:50 +01:00
build.sbt Improve upload logging (#44) 2019-06-10 19:45:36 +01:00
CHANGELOG.org Support multiple filters (#18) 2019-05-23 19:35:48 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
README.org [readme] add note about broken native images 2019-05-30 18:38:23 +01:00

s3thorp

Synchronisation of files with S3 using the hash of the file contents.

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  s3thorp
  Usage: s3thorp [options]

    -s, --source <value>             Source directory to sync to S3
    -b, --bucket <value>             S3 bucket name
    -p, --prefix <value>             Prefix within the S3 Bucket
    -x, --exclude <value>[,<values>] Exclude matching paths
    -v, --verbose <value>            Verbosity level (1-5)

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Creating Native Images

Note: the created image currently can't be run outside of the base of the project. See Issue #15

  • Download and install GraalVM

  • Install native-image using the graal updater

      gu install native-image
    
  • Create native image

      native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \
                   -H:Name=s3thorp \
                   -H:Class=net.kemitix.s3thorp.Main \
                   --allow-incomplete-classpath \
                   --force-fallback
    
  • Resulting file requires a JDK for execution