Paul Campbell
b15350d959
* [changelog] updated
* [readme] updated
* [core] ConfigQuery added sources()
* [cli] ParseArgs allow specifying multiple sources
* [domain,core,cli] Source datatype changed to Path
* [domain] Sources added to hold multiple paths in order
* [domain] Config sources change datatype to Sources
* [core] Scan sources for .thorp.config and include any sources listed
This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.
* [core] ConfigurationBuilderTest add more pending tests
* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures
* [core] ConfigOptionTest use TemporaryFolder
* [core] ConfigOptionTest remove unused fields
* [cli] ParseArgsTest don't use get on an Option
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolder Move import to top of file
* [core] TemporaryFolder use Try over try-finally
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolders.withDirectory propogate errors
* [core] TemporaryFolders add writeFile and createFile
* [core] PlanBuilderTest create a plan with two sources with unique files in both
* [core] ActionGenerator only upload file by name in first source
create a plan
two sources
same filename in both
- only upload file in first source
* [domain] LastModified with no params is now()
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [domain] RemoteKey map to a file when prefix is empty
* [domain] S3ObjectData defaults to empty
* [core] KeyGenerator Avoid delimiter when empty prefix key
* [core] PlanBuilderTest when remote not in sources delete from remote
* [core] PlanBuilderTest extract helper md5Hash()
* [core] PlanBuilderTest one source a file no matching remote key
* [core] PlanBuildingTest file with matching key and hash do nothing
* [core] PlanBuilderTest file w/matching remote key and different hash
* [core] PlanBuilderTest a remote key with and without local file
* [core] DummyStorageService Use wildcards when selecting more than 6 elements
87 lines
3.4 KiB
Org Mode
87 lines
3.4 KiB
Org Mode
* thorp
|
|
|
|
Synchronisation of files with S3 using the hash of the file contents.
|
|
|
|
[[https://www.codacy.com/app/kemitix/thorp][file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge]]
|
|
[[https://search.maven.org/search?q=net.kemitix.thorp][file:https://img.shields.io/maven-central/v/net.kemitix.thorp/thorp_2.12.svg?style=for-the-badge]]
|
|
|
|
Originally based on Alex Kudlick's [[https://github.com/akud/aws-s3-sync-by-hash][aws-s3-sync-by-hash]].
|
|
|
|
The normal ~aws s3 sync ...~ command only uses the time stamp of files
|
|
to decide what files need to be copied. This utility looks at the md5
|
|
hash of the file contents.
|
|
|
|
* Usage
|
|
|
|
#+begin_example
|
|
thorp
|
|
Usage: thorp [options]
|
|
|
|
-V, --version Display the version and quit
|
|
-B, --batch Enabled batch-mode
|
|
-s, --source <value> Source directory to sync to S3
|
|
-b, --bucket <value> S3 bucket name
|
|
-p, --prefix <value> Prefix within the S3 Bucket
|
|
-i, --include <value> Include matching paths
|
|
-x, --exclude <value> Exclude matching paths
|
|
-d, --debug Enable debug logging
|
|
--no-global Ignore global configuration
|
|
--no-user Ignore user configuration
|
|
#+end_example
|
|
|
|
If you don't provide a ~source~ the current diretory will be used.
|
|
|
|
The ~--include~ and ~--exclude~ parameters can be used more than once.
|
|
|
|
The ~--source~ parameter can be used more than once, in which case,
|
|
all files in all sources will be consolidated into the same
|
|
bucket/prefix.
|
|
|
|
** Batch mode
|
|
|
|
Batch mode disable the ANSI console display and logs simple messages
|
|
that can be written to a file.
|
|
|
|
* Configuration
|
|
|
|
Configuration will be read from these files:
|
|
|
|
- Global: ~/etc/thorp.conf~
|
|
- User: ~ ~/.config/thorp.conf~
|
|
- Source: ~${source}/.thorp.conf~
|
|
|
|
Command line arguments override those in Source, which override
|
|
those in User, which override those Global, which override any
|
|
built-in config.
|
|
|
|
When there is more than one source, only the first ".thorp.conf"
|
|
file found will be used.
|
|
|
|
Built-in config consists of using the current working directory as
|
|
the ~source~.
|
|
|
|
Note, that ~include~ and ~exclude~ are cumulative across all
|
|
configuration files.
|
|
|
|
* Behaviour
|
|
|
|
When considering a local file, the following table governs what should happen:
|
|
|
|
|---+------------+------------+------------------+--------------------+---------------------|
|
|
| # | local file | remote key | hash of same key | hash of other keys | action |
|
|
|---+------------+------------+------------------+--------------------+---------------------|
|
|
| 1 | exists | exists | matches | - | do nothing |
|
|
| 2 | exists | is missing | - | matches | copy from other key |
|
|
| 3 | exists | is missing | - | no matches | upload |
|
|
| 4 | exists | exists | no match | matches | copy from other key |
|
|
| 5 | exists | exists | no match | no matches | upload |
|
|
| 6 | is missing | exists | - | - | delete |
|
|
|---+------------+------------+------------------+--------------------+---------------------|
|
|
|
|
* Executable JAR
|
|
|
|
To build as an executable jar, perform `sbt assembly`
|
|
|
|
This will create the file `cli/target/scala-2.12/thorp`
|
|
|
|
Copy this file to your `PATH`.
|