S3 Sync
Paul Campbell
97efed76b4
* [aws-lib] fold S3ClientUploader trait into it's only implementation This trait was only implemented by S3ClientTransferManager. * [core] SyncLogging: more robust matching No longer cares about parameters to case classes, just their types. * [cli] Logger uses IO for log methods * [aws-lib] remove 'transfer-manager'prefix and only show tryCount > 1 * [sbt,cli] remove log4j and scala-logging dependencies * [domain] move QuoteStripper to Domain Use it directly in MD5Hash to strip quotes from any input. * [core] SyncLogging call info in proper context If the IO.unit returned by the info calls isn't part of the chain that is returned from the function, then the delayed IO action is never called. * [aws-lib] Display size in bytes of file being uploaded * [core] call info in correct context * [cli] call info in correct context * [aws-lib] raise summary fetch message to info 1 * [cli] include correct level in info messages * [aws-lib] S3ClientLogging adjust logging levels * [aws-lib] display file sizes in english * [aws-lib] ObjectLister use IO.bracket properly * [aws-lib] Copier use IO.bracket properly * [aws-lib] Deleter refactor * [aws-lib] TransferManagerLogging remove unused methods * [aws-lib] TransferManager refactor * [aws-lib] TransferManager refactor * [aws-lib] TransferManager displays log messages Use the UploadProgressListener that was being ignored, and use unsafeRunSync to execute the suspended effect within the IO[Unit]. Using unsafeRunSync is required to render the effects as the listener returns Unit, meaning the suspended effects would be discarded. * [domain] Extract SizeTranslation into module * [aws-api] report bytes transferred in progress * [core] fix calls to info info now returns an IO already, so don't need to wrap it in one. * [aws-lib] remove unused class * [aws-lib] UploadProgress displays progress bar while uploading * [aws-api] UploadProgressLogging optimise imports * [aws-api] UploadProgressLogging rename variables * [domain] add Terminal object * [aws-api] UploadProgressLogging use console width and two lines - Improved clearing of lines after progress bar - Use console width for progress bar size * [aws-lib] S3ClientLogging optimise imports * [aws-lib] TransferManager clear line before logging * [aws-lib] rename class as TransferManager * [aws-lib] rename TransferManger as Uploader to not clash We are using an AWS SDK class with the same name. |
||
|---|---|---|
| .github | ||
| aws-api/src/main/scala/net/kemitix/s3thorp/aws/api | ||
| aws-lib/src | ||
| cli/src/main/scala/net/kemitix/s3thorp/cli | ||
| core/src | ||
| domain/src | ||
| project | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
s3thorp
Synchronisation of files with S3 using the hash of the file contents.
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
s3thorp
Usage: s3thorp [options]
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-x, --exclude <value>[,<values>] Exclude matching paths
-v, --verbose <value> Verbosity level (1-5)
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Creating Native Images
Note: the created image currently can't be run outside of the base of the project. See Issue #15
-
Download and install GraalVM
-
Install
native-imageusing the graal updatergu install native-image
-
Create native image
native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \ -H:Name=s3thorp \ -H:Class=net.kemitix.s3thorp.Main \ --allow-incomplete-classpath \ --force-fallback - Resulting file requires a JDK for execution