Paul Campbell
00c04187e8
* [changelog] updated * [core] Wrap Stream[LocalFile] as LocalFiles * [core] LocalFiles counts files * [core] LocalFiles sums file lengths * [core] Restore logFileScan * [storage-aws] Lister logs when fetching object summaries * [storage-aws] Extract ListerLogger * [core] Synchronise use leftMap * [core] Syncronise extract assemblePlan * [core] Wrap Stream[Action] in SyncPlan * [core] Copy the file count and totalSizeBytes across to SyncPlan * [cli] Program rename actions as syncPlan * [cli] Program extract thorpArchive def * [cli] Program extract createPlan def * [cli] Program refactoring * [cli] Program remove println showing version * [cli] Program rename actions parameter as syncPlan * [core] ThorpArchive add an index to each action * [cli] Program make SyncPlan available to ThorpArchive * [core] Pass SyncTotals to Archive * [domain] Move SyncTotals into module * [domain] Pass index and SyncTotals to UploadEventListener * [domain] UploadEventLogger add file count a size progress bars * [domain] UploadEventLogger better display stability and add file index * [cli] Index files in correct order * [cli] Program extends Synchronise * [core] Rename Synchronise as PlanBuilder * [cli] Program add test to check actions don't get reordered from plan * [core] collect file size totals * [domain] UploadEventLogger include percentage * [cli] ProgramTest Use wildcards when selecting more than 6 elements |
||
|---|---|---|
| .github | ||
| bin | ||
| cli/src | ||
| core/src | ||
| domain/src | ||
| project | ||
| storage-api/src/main/scala/net/kemitix/thorp/storage/api | ||
| storage-aws/src | ||
| .gitignore | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| README.org | ||
thorp
Synchronisation of files with S3 using the hash of the file contents.
file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
thorp
Usage: thorp [options]
-V, --version Display the version and quit
-B, --batch Enabled batch-mode
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-d, --debug Enable debug logging
--no-global Ignore global configuration
--no-user Ignore user configuration
If you don't provide a source the current diretory will be used.
The --include and --exclude parameters can be used more than once.
Batch mode
Batch mode disable the ANSI console display and logs simple messages that can be written to a file.
Configuration
Configuration will be read from these files:
- Global:
/etc/thorp.conf - User: ~
/.config/thorp.conf - Source:
${source}/.thorp.conf
Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.
Built-in config consists of using the current working directory as the
source.
Note, that include and exclude are cumulative across all
configuration files.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Executable JAR
To build as an executable jar, perform `sbt assembly`
This will create the file `cli/target/scala-2.12/thorp`
Copy this file to your `PATH`.