51 lines
1.6 KiB
Org Mode
51 lines
1.6 KiB
Org Mode
* s3thorp
|
|
|
|
Synchronisation of files with S3 using the hash of the file contents.
|
|
|
|
Based on Alex Kudlick's JavaScript implementation [[https://github.com/akud/aws-s3-sync-by-hash][aws-s3-sync-by-hash]].
|
|
|
|
The normal ~aws s3 sync ...~ command only uses the time stamp of files
|
|
to decide what files need to be copied. This utility looks at the md5
|
|
hash of the file contents.
|
|
|
|
* Usage
|
|
|
|
#+begin_example
|
|
s3thorp
|
|
Usage: S3Thorp [options]
|
|
|
|
-s, --source <value> Source directory to sync to S3
|
|
-b, --bucket <value> S3 bucket name
|
|
-p, --prefix <value> Prefix within the S3 Bucket
|
|
#+end_example
|
|
|
|
* Creating Native Images
|
|
|
|
- Download and install GraalVM
|
|
- https://github.com/oracle/graal/releases
|
|
- Install ~native-image~ using the graal updater
|
|
#+begin_example bash
|
|
gu install native-image
|
|
#+end_example
|
|
- Create native image
|
|
#+begin_example bash
|
|
native-image -cp `sbt 'export runtime:fullClasspath'|tail -n 1` \
|
|
-H:Name=s3thorp \
|
|
-H:Class=net.kemitix.s3thorp.Main \
|
|
--allow-incomplete-classpath \
|
|
--force-fallback
|
|
#+end_example
|
|
- Resulting file requires a JDK for execution
|
|
|
|
* TO DO
|
|
|
|
- [X] Improve test coverage
|
|
- [X] Create os-native binaries
|
|
- [X] Replace println with real logging
|
|
- [ ] Add support for logging options
|
|
- [ ] Add support for exclusion filters
|
|
- [ ] Bulk fetching of Hash values from S3
|
|
- [ ] ? When lastModified matches local file, skip calculating local MD5 ?
|
|
- [ ] Add support for multi-part uploads for large files
|
|
- [ ] Add support for upload progress - may only be available with
|
|
multi-part uploads
|