thorp

S3 Sync

Find a file

Paul Campbell 7ffa386b29 [core] MD5HashGenerator uses IO to return where there is file IO (#47 ) * [core] MD5HashGenerator uses IO to return where there is file IO This required that LocalFile in the domain module no longer be supplied with a function to convert a File into an MD5Hash. Because such a function requires reading the file it now must use IO, which we don't allow in the domain module. Unfortunate ripple effects out to users of MD5HashGenerator and LocalFile. * [aws-lib] Add own copy of test class MD5HashData		2019-06-08 18:19:15 +01:00
.github	[github] Add stale configuration	2019-05-14 07:05:48 +01:00
aws-api/src/main/scala/net/kemitix/s3thorp/aws/api	Split into subprojects (#36 )	2019-06-06 19:24:15 +01:00
aws-lib/src	[core] MD5HashGenerator uses IO to return where there is file IO (#47 )	2019-06-08 18:19:15 +01:00
cli/src/main	Split into subprojects (#36 )	2019-06-06 19:24:15 +01:00
core/src	[core] MD5HashGenerator uses IO to return where there is file IO (#47 )	2019-06-08 18:19:15 +01:00
domain/src	[core] MD5HashGenerator uses IO to return where there is file IO (#47 )	2019-06-08 18:19:15 +01:00
project	[gitignote] update to allow some project files	2019-05-11 08:54:35 +01:00
.gitignore	[gitignore] ignore zip files	2019-05-14 07:27:14 +01:00
.travis.yml	[travis] define AWS_REGION environment variable	2019-05-16 19:28:50 +01:00
build.sbt	Update aws-java-sdk-s3 to 1.11.568 (#42 )	2019-06-08 08:21:07 +01:00
CHANGELOG.org	Support multiple filters (#18 )	2019-05-23 19:35:48 +01:00
LICENSE	Create LICENSE	2019-06-07 21:25:23 +01:00
README.org	[readme] add note about broken native images	2019-05-30 18:38:23 +01:00

s3thorp

Synchronisation of files with S3 using the hash of the file contents.

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  s3thorp
  Usage: s3thorp [options]

    -s, --source <value>             Source directory to sync to S3
    -b, --bucket <value>             S3 bucket name
    -p, --prefix <value>             Prefix within the S3 Bucket
    -x, --exclude <value>[,<values>] Exclude matching paths
    -v, --verbose <value>            Verbosity level (1-5)

Behaviour

When considering a local file, the following table governs what should happen:

#	local file	remote key	hash of same key	hash of other keys	action
1	exists	exists	matches	-	do nothing
2	exists	is missing	-	matches	copy from other key
3	exists	is missing	-	no matches	upload
4	exists	exists	no match	matches	copy from other key
5	exists	exists	no match	no matches	upload
6	is missing	exists	-	-	delete

Creating Native Images

Note: the created image currently can't be run outside of the base of the project. See Issue #15

Download and install GraalVM
- https://github.com/oracle/graal/releases
Install native-image using the graal updater
```
  gu install native-image
```

Create native image

  native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \
               -H:Name=s3thorp \
               -H:Class=net.kemitix.s3thorp.Main \
               --allow-incomplete-classpath \
               --force-fallback

Resulting file requires a JDK for execution