thorp

S3 Sync

Find a file

Paul Campbell 574d4c5885 Display upload progress (#29 ) * [S3ClientMultiPartTransferManager] use request object * [ActionSubmitter] unwrap RemoteKey in log messages * [ActionSubmitter] rename variable * [Logging] include log level in info messages * [LocalFileStream] log when entering directory at level 2 * [UploadProgress{Listener,Logging}: add initial implementations * [S3Client] def upload not requires an UploadProgressListener as a parameter * [UploadProgressListener] rename method * [S3ClientPutObjectUploader] Log upload progress for file <5Mb Switched to using the AWS SDK V1 for PutObject as the V2 doesn't support progress callbacks. * Fix up tests * Adjust logging levels		2019-05-30 16:59:37 +01:00
.github	[github] Add stale configuration	2019-05-14 07:05:48 +01:00
project	[gitignote] update to allow some project files	2019-05-11 08:54:35 +01:00
src	Display upload progress (#29 )	2019-05-30 16:59:37 +01:00
.gitignore	[gitignore] ignore zip files	2019-05-14 07:27:14 +01:00
.travis.yml	[travis] define AWS_REGION environment variable	2019-05-16 19:28:50 +01:00
build.sbt	Update aws-java-sdk-s3 to 1.11.562 (#30 )	2019-05-30 15:12:34 +01:00
CHANGELOG.org	Support multiple filters (#18 )	2019-05-23 19:35:48 +01:00
README.org	Rename filter as exclude (#23 )	2019-05-27 23:23:13 +01:00

s3thorp

Synchronisation of files with S3 using the hash of the file contents.

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  s3thorp
  Usage: s3thorp [options]

    -s, --source <value>             Source directory to sync to S3
    -b, --bucket <value>             S3 bucket name
    -p, --prefix <value>             Prefix within the S3 Bucket
    -x, --exclude <value>[,<values>] Exclude matching paths
    -v, --verbose <value>            Verbosity level (1-5)

Behaviour

When considering a local file, the following table governs what should happen:

#	local file	remote key	hash of same key	hash of other keys	action
1	exists	exists	matches	-	do nothing
2	exists	is missing	-	matches	copy from other key
3	exists	is missing	-	no matches	upload
4	exists	exists	no match	matches	copy from other key
5	exists	exists	no match	no matches	upload
6	is missing	exists	-	-	delete

Creating Native Images

Download and install GraalVM
- https://github.com/oracle/graal/releases
Install native-image using the graal updater
```
  gu install native-image
```

Create native image

  native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \
               -H:Name=s3thorp \
               -H:Class=net.kemitix.s3thorp.Main \
               --allow-incomplete-classpath \
               --force-fallback

Resulting file requires a JDK for execution