S3 Sync
Find a file
Paul Campbell 1267b6e313
Add a batch mode that provides a simple log output (#85)
* [changelog] Updated

* [readme] Updated

* [domain] Config Add batch-mode flag

* [core] ConfigOption Add BatchMode option

* [core] ConfigQuery Add batchMode query

Also replaced verbose exists case clauses with a simple contains.

* [core] ConfigOptions added to replace Seq[ConfigOption]

* [core] Syncronise rename method to createPlan

* [cli] Program rename apply as run

* [storage-aws] S3StorageServiceBuilder stop using IO to create object

* [storage-aws] S3StorageServiceBuilder make default service lazy

* [storage-aws] Rename S3ClientCopier => Copier

* [storage-aws] Rename S3ClientDeleter => Deleter

* [storage-aws] Rename S3ClientObjectLister => Lister

* [storage-aws] Only attach upload listener when in batch mode

Only detects batch mode when selected as a command line option

* [core] Synchronise use leftMap rather than swap.map.swap

* [cli] ParseArgs add `-B` and `--batch` options to enable batch mode

* [core] ThorpArchive logs file uploaded when in batch mode
2019-07-02 08:43:52 +01:00
.github [github] Add stale configuration 2019-05-14 07:05:48 +01:00
bin Rename project to Thorp (#75) 2019-06-17 15:33:49 +01:00
cli/src Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
core/src Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
domain/src Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
project Add a version command-line option (#99) 2019-06-30 22:01:02 +01:00
storage-api/src/main/scala/net/kemitix/thorp/storage/api Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
storage-aws/src Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
.gitignore Rename project to Thorp (#75) 2019-06-17 15:33:49 +01:00
.travis.yml [sbt,travis] revert most of "publish fat-jar", keeping cli jar name (#97) 2019-06-30 14:38:50 +01:00
build.sbt Add a version command-line option (#99) 2019-06-30 22:01:02 +01:00
CHANGELOG.org Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
README.org Add a batch mode that provides a simple log output (#85) 2019-07-02 08:43:52 +01:00

thorp

Synchronisation of files with S3 using the hash of the file contents.

file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  thorp
  Usage: thorp [options]

    -V, --version         Display the version and quit
    -B, --batch           Enabled batch-mode
    -s, --source <value>  Source directory to sync to S3
    -b, --bucket <value>  S3 bucket name
    -p, --prefix <value>  Prefix within the S3 Bucket
    -i, --include <value> Include matching paths
    -x, --exclude <value> Exclude matching paths
    -d, --debug           Enable debug logging
    --no-global           Ignore global configuration
    --no-user             Ignore user configuration

If you don't provide a source the current diretory will be used.

The --include and --exclude parameters can be used more than once.

Batch mode

Batch mode disable the ANSI console display and logs simple messages that can be written to a file.

Configuration

Configuration will be read from these files:

  • Global: /etc/thorp.conf
  • User: ~ /.config/thorp.conf
  • Source: ${source}/.thorp.conf

Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.

Built-in config consists of using the current working directory as the source.

Note, that include and exclude are cumulative across all configuration files.

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Executable JAR

To build as an executable jar, perform `sbt assembly`

This will create the file `cli/target/scala-2.12/thorp`

Copy this file to your `PATH`.