S3 Sync
Find a file
dependabot[bot] 9e9844d35b Bump maven-assembly-plugin from 3.3.0 to 3.4.2
Bumps [maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) from 3.3.0 to 3.4.2.
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.4.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

commit-id:39648907
2022-12-03 11:48:22 +00:00
.github Create Dependabot config file (#524) 2020-09-15 22:00:38 +01:00
.run Convert lib module to Java (#487) 2020-06-27 18:00:22 +01:00
app Bump maven-assembly-plugin from 3.3.0 to 3.4.2 2022-12-03 11:48:22 +00:00
bin Rename project to Thorp (#75) 2019-06-17 15:33:49 +01:00
cli Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
config Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
console Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
docs/images Use kemitix-aws-java-sdk-s3-wrapper (#503) 2020-07-11 16:06:33 +01:00
domain Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
filesystem Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
lib Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
parent Bump assertj-core from 3.19.0 to 3.23.1 2022-12-03 11:45:48 +00:00
storage Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
storage-aws fix: storage-aws/pom.xml to reduce vulnerabilities (#626) 2021-12-29 11:38:52 +00:00
uishell Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
.gitignore Release 1.0.0 (#463) 2020-06-21 07:54:43 +01:00
CHANGELOG.org Release 1.0.0 (#463) 2020-06-21 07:54:43 +01:00
LICENSE Create LICENSE 2019-06-07 21:25:23 +01:00
pom.xml Version set to 2.0.1 (#546) 2020-11-06 18:49:20 +00:00
README.md readme: update badge to show github release (#480) 2020-06-25 06:58:18 +01:00

thorp

Synchronisation of files with S3 using the hash of the file contents.

GitHub release (latest SemVer)

Originally based on Alex Kudlick's aws-s3-sync-by-hash.

The normal aws s3 sync ... command only uses the time stamp of files to decide what files need to be copied. This utility looks at the md5 hash of the file contents.

Usage

$ thorp
Usage: thorp [options]

  -V, --version         Display the version and quit
  -B, --batch           Enabled batch-mode
  -s, --source <value>  Source directory to sync to S3
  -b, --bucket <value>  S3 bucket name
  -p, --prefix <value>  Prefix within the S3 Bucket
  -P, --parallel <value> Maximum parallel upload/copy operations
  -i, --include <value> Include matching paths
  -x, --exclude <value> Exclude matching paths
  -d, --debug           Enable debug logging
  --no-global           Ignore global configuration
  --no-user             Ignore user configuration

If you don't provide a source the current directory will be used.

The --include and --exclude parameters can be used more than once.

The --source parameter can be used more than once, in which case, all files in all sources will be consolidated into the same bucket/prefix.

Batch mode

Batch mode disable the ANSI console display and logs simple messages that can be written to a file.

Configuration

Configuration will be read from these files:

  • Global: /etc/thorp.conf
  • User: ~/.config/thorp.conf
  • Source: ${source}/.thorp.conf

Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.

When there is more than one source, only the first .thorp.conf file found will be used.

Built-in config consists of using the current working directory as the source.

Note, that include and exclude are cumulative across all configuration files.

Caching

The last modified time for files is used to decide whether to calculate the hash values for the file. If a file has not been updated, then the hash values stored in the .thorp.cache file located in the root of the source is used. Otherwise the file will be read to caculate the the new hashes.

Behaviour

When considering a local file, the following table governs what should happen:

# local file remote key hash of same key hash of other keys action
1 exists exists matches - do nothing
2 exists is missing - matches copy from other key
3 exists is missing - no matches upload
4 exists exists no match matches copy from other key
5 exists exists no match no matches upload
6 is missing exists - - delete

Executable JAR

To build as an executable jar, perform mvn package

This will create the file app/target/thorp-${version}-jar-with-dependencies.jar

Copy and rename this file into your PATH.

Structure/Dependencies

Dependency Graph