Paul Campbell
910688ee32
* [core] rename the config supplied from CLI as such This distinguishes it as config supplied from the command line. * [core] add ConfigOption * [core] ConfigOption can update a Config * [core] new validator for config * [domain] Config doesn't validate source any more * [cli] PrintLogger default to not print debug messages * [cli] Use ConfigOptions and new ConfigValidator * [sbt] Use common settings for project root * [domain] RemoteKey can handle when prefix is empty * [cli] remove banner * [domain] Logger can create version of itself with debug flipped * [core] Build and validate Config within core module This means that the `thorp-lib` module can validate its input from a list of `ConfigOption`s. * [core] refactor ConfigurationBuilder * [core] refactor ConfigurationBuilder * [sbt] starting back from tagless-final by using IO where needed * [core] Add ParseConfigFile * [sbt] Make cats-effect available from the domain * Roll back from tagless-final to just use cat-effect's IO * [core] extract ParseConfigLines * [core] ConfigurationBuilder rename apply as buildConfig * [core] ParseConfig[Files,Lines] rename apply methods * [core] refactor ParseConfigFile and add tests * [core] Sync fix call to run * [core] SyncSuite update tests to use ConfigOptions |
||
|---|---|---|
| .github | ||
| aws-api/src/main/scala/net/kemitix/thorp/aws/api | ||
| aws-lib/src | ||
| bin | ||
| cli/src | ||
| core/src | ||
| domain/src | ||
| project | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
thorp
Synchronisation of files with S3 using the hash of the file contents.
file:https://img.shields.io/codacy/grade/14ea6ad0825249c994a27a82d3485180.svg?style=for-the-badge
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
thorp
Usage: thorp [options]
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-d, --debug Enable debug logging
The --include and --exclude parameters can be used more than once.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Executable JAR
To build as an executable jar, perform `sbt assembly`
This will create the file `cli/target/scala-2.12/thorp-assembly-$VERSION.jar` (where $VERSION is substituted)
Copy and rename this file as `thorp.jar` into the same directory as the `bin/throp` shell script.