Paul Campbell
b15350d959
* [changelog] updated
* [readme] updated
* [core] ConfigQuery added sources()
* [cli] ParseArgs allow specifying multiple sources
* [domain,core,cli] Source datatype changed to Path
* [domain] Sources added to hold multiple paths in order
* [domain] Config sources change datatype to Sources
* [core] Scan sources for .thorp.config and include any sources listed
This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.
* [core] ConfigurationBuilderTest add more pending tests
* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures
* [core] ConfigOptionTest use TemporaryFolder
* [core] ConfigOptionTest remove unused fields
* [cli] ParseArgsTest don't use get on an Option
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolder Move import to top of file
* [core] TemporaryFolder use Try over try-finally
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolders.withDirectory propogate errors
* [core] TemporaryFolders add writeFile and createFile
* [core] PlanBuilderTest create a plan with two sources with unique files in both
* [core] ActionGenerator only upload file by name in first source
create a plan
two sources
same filename in both
- only upload file in first source
* [domain] LastModified with no params is now()
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [domain] RemoteKey map to a file when prefix is empty
* [domain] S3ObjectData defaults to empty
* [core] KeyGenerator Avoid delimiter when empty prefix key
* [core] PlanBuilderTest when remote not in sources delete from remote
* [core] PlanBuilderTest extract helper md5Hash()
* [core] PlanBuilderTest one source a file no matching remote key
* [core] PlanBuildingTest file with matching key and hash do nothing
* [core] PlanBuilderTest file w/matching remote key and different hash
* [core] PlanBuilderTest a remote key with and without local file
* [core] DummyStorageService Use wildcards when selecting more than 6 elements
|
||
|---|---|---|
| .github | ||
| bin | ||
| cli/src | ||
| core/src | ||
| domain/src | ||
| project | ||
| storage-api/src/main/scala/net/kemitix/thorp/storage/api | ||
| storage-aws/src | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
thorp
Synchronisation of files with S3 using the hash of the file contents.
file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge file:https://img.shields.io/maven-central/v/net.kemitix.thorp/thorp_2.12.svg?style=for-the-badge
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
thorp
Usage: thorp [options]
-V, --version Display the version and quit
-B, --batch Enabled batch-mode
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-d, --debug Enable debug logging
--no-global Ignore global configuration
--no-user Ignore user configuration
If you don't provide a source the current diretory will be used.
The --include and --exclude parameters can be used more than once.
The --source parameter can be used more than once, in which case,
all files in all sources will be consolidated into the same
bucket/prefix.
Batch mode
Batch mode disable the ANSI console display and logs simple messages that can be written to a file.
Configuration
Configuration will be read from these files:
- Global:
/etc/thorp.conf - User: ~
/.config/thorp.conf - Source:
${source}/.thorp.conf
Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.
When there is more than one source, only the first ".thorp.conf" file found will be used.
Built-in config consists of using the current working directory as
the source.
Note, that include and exclude are cumulative across all
configuration files.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Executable JAR
To build as an executable jar, perform `sbt assembly`
This will create the file `cli/target/scala-2.12/thorp`
Copy this file to your `PATH`.