S3 Sync
Find a file
Paul Campbell 708f3697ee
Release snapshots to sonatype from TravisCI (#92)
* [sbt,travis] Deploy releases and snapshots to sonatype

* [travis] remove oracle JDK builds

* [travis] remove 'compile' job

* [travis] Try adding scala 2.13

* [travis] test using openjdk8 rather than 10

* [travis] force use of scala version, overriding sbt version

* Revert "[travis] Try adding scala 2.13"

This reverts commit 6725199e009ae1f809247bbc0851b62d63121993.

* Revert "[travis] force use of scala version, overriding sbt version"

This reverts commit 21c94992f40cf2ef39c5126e67b66feaabe1b2c0.

* [sbt] Group modules under n.k.thorp organisation
2019-06-30 12:23:04 +01:00
.github [github] Add stale configuration 2019-05-14 07:05:48 +01:00
bin Rename project to Thorp (#75) 2019-06-17 15:33:49 +01:00
cli/src [cli] Program handle actions list in stack safe manner (#93) 2019-06-30 08:47:51 +01:00
core/src Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
domain/src Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
project Release snapshots to sonatype from TravisCI (#92) 2019-06-30 12:23:04 +01:00
storage-api/src/main/scala/net/kemitix/thorp/storage/api Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
storage-aws/src Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
.gitignore Rename project to Thorp (#75) 2019-06-17 15:33:49 +01:00
.travis.yml Release snapshots to sonatype from TravisCI (#92) 2019-06-30 12:23:04 +01:00
build.sbt Release snapshots to sonatype from TravisCI (#92) 2019-06-30 12:23:04 +01:00
CHANGELOG.org Restructure sync to use a State with foldLeft around actions (#74) 2019-06-25 08:27:38 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
README.org Is AWS SDK calculating MD5Hash again for a local file? (#50) 2019-06-21 19:20:35 +01:00

thorp

Synchronisation of files with S3 using the hash of the file contents.

file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

  thorp
  Usage: thorp [options]

    -s, --source <value>  Source directory to sync to S3
    -b, --bucket <value>  S3 bucket name
    -p, --prefix <value>  Prefix within the S3 Bucket
    -i, --include <value> Include matching paths
    -x, --exclude <value> Exclude matching paths
    -d, --debug           Enable debug logging
    --no-global           Ignore global configuration
    --no-user             Ignore user configuration

If you don't provide a source the current diretory will be used.

The --include and --exclude parameters can be used more than once.

Configuration

Configuration will be read from these files:

  • Global: /etc/thorp.conf
  • User: ~ /.config/thorp.conf
  • Source: ${source}/.thorp.conf

Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.

Built-in config consists of using the current working directory as the source.

Note, that include and exclude are cumulative across all configuration files.

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Executable JAR

To build as an executable jar, perform `sbt assembly`

This will create the file `cli/target/scala-2.12/thorp-assembly-$VERSION.jar` (where $VERSION is substituted)

Copy and rename this file as `thorp.jar` into the same directory as the `bin/throp` shell script.