S3 Sync
Find a file
Paul Campbell 21b8917395
Simplify logging (#59)
* [cli] Logger add debug method

* [cli] Logger: add alternate info method

* [cli] replace direct calls to legacy Logger.info

* [cli] Extract trait from Logger into core and rename

* [cli] Program use debug and new info

* [core] MD5HashGenerator uses Logger

* [core] Sync uses Logger

* [core] ActionSubmitter uses Logger

* [core] LocalFileStream uses Logger

* [core] SyncLogging uses Logger

* [domain] Move Logger into module

The allows it to be used in the aws-api module

* [aws-lib] S3ClientObjectLister uses Logger

* [aws-lib] Uploader uses Logger

* [aws-lib] S3ClientDeleter uses Logger

* [aws-lib] S3ClientCopier uses Logger

* [core] remove unused legacy logging

* [aws-lib] remove used logging method

* [cli] PrintLogger remove legacy info method
2019-06-14 17:18:53 +01:00
.github [github] Add stale configuration 2019-05-14 07:05:48 +01:00
aws-api/src/main/scala/net/kemitix/s3thorp/aws/api Simplify logging (#59) 2019-06-14 17:18:53 +01:00
aws-lib/src Simplify logging (#59) 2019-06-14 17:18:53 +01:00
bin Enable running outside of sbt (#55) 2019-06-11 23:36:08 +01:00
cli/src/main/scala/net/kemitix/s3thorp/cli Simplify logging (#59) 2019-06-14 17:18:53 +01:00
core/src Simplify logging (#59) 2019-06-14 17:18:53 +01:00
domain/src Simplify logging (#59) 2019-06-14 17:18:53 +01:00
project Enable running outside of sbt (#55) 2019-06-11 23:36:08 +01:00
.gitignore [gitignore] ignore zip files 2019-05-14 07:27:14 +01:00
.travis.yml [travis] define AWS_REGION environment variable 2019-05-16 19:28:50 +01:00
build.sbt Update aws-java-sdk-s3 to 1.11.570 (#57) 2019-06-14 08:52:05 +01:00
CHANGELOG.org [changelog] Updated for 0.4.0 2019-06-11 08:04:31 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
README.org Enable running outside of sbt (#55) 2019-06-11 23:36:08 +01:00

s3thorp

Synchronisation of files with S3 using the hash of the file contents.

file:https://img.shields.io/codacy/grade/14ea6ad0825249c994a27a82d3485180.svg?style=for-the-badge

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  s3thorp
  Usage: s3thorp [options]

    -s, --source <value>  Source directory to sync to S3
    -b, --bucket <value>  S3 bucket name
    -p, --prefix <value>  Prefix within the S3 Bucket
    -i, --include <value> Include matching paths
    -x, --exclude <value> Exclude matching paths
    -v, --verbose <value> Verbosity level (1-5)

The --include and --exclude parameters can be used more than once.

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Executable JAR

To build as an executable jar, perform `sbt assembly`

This will create the file `cli/target/scala-2.12/s3thorp-assembly-$VERSION.jar` (where $VERSION is substituted)

Copy and rename this file as `s3thorp.jar` into the same directory as the `bin/s3throp` shell script.