Paul Campbell
c5d7d4933c
* [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method |
||
|---|---|---|
| .github | ||
| app/src/main/scala/net/kemitix/thorp | ||
| bin | ||
| cli/src | ||
| config/src | ||
| console/src/main/scala/net/kemitix/thorp/console | ||
| domain/src | ||
| filesystem/src | ||
| lib/src | ||
| project | ||
| storage/src/main/scala/net/kemitix/thorp/storage | ||
| storage-aws/src | ||
| uishell/src/main/scala/net/kemitix/throp/uishell | ||
| .gitignore | ||
| .scalafmt.conf | ||
| .travis.yml | ||
| build.sbt | ||
| CHANGELOG.org | ||
| LICENSE | ||
| modules.dot | ||
| README.org | ||
thorp
Synchronisation of files with S3 using the hash of the file contents.
file:https://img.shields.io/codacy/grade/c1719d44f1f045a8b71e1665a6d3ce6c.svg?style=for-the-badge file:https://img.shields.io/maven-central/v/net.kemitix.thorp/thorp_2.12.svg?style=for-the-badge
Originally based on Alex Kudlick's aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
Usage
thorp
Usage: thorp [options]
-V, --version Display the version and quit
-B, --batch Enabled batch-mode
-s, --source <value> Source directory to sync to S3
-b, --bucket <value> S3 bucket name
-p, --prefix <value> Prefix within the S3 Bucket
-i, --include <value> Include matching paths
-x, --exclude <value> Exclude matching paths
-d, --debug Enable debug logging
--no-global Ignore global configuration
--no-user Ignore user configuration
If you don't provide a source the current diretory will be used.
The --include and --exclude parameters can be used more than once.
The --source parameter can be used more than once, in which case,
all files in all sources will be consolidated into the same
bucket/prefix.
Batch mode
Batch mode disable the ANSI console display and logs simple messages that can be written to a file.
Configuration
Configuration will be read from these files:
- Global:
/etc/thorp.conf - User: ~
/.config/thorp.conf - Source:
${source}/.thorp.conf
Command line arguments override those in Source, which override those in User, which override those Global, which override any built-in config.
When there is more than one source, only the first ".thorp.conf" file found will be used.
Built-in config consists of using the current working directory as
the source.
Note, that include and exclude are cumulative across all
configuration files.
Behaviour
When considering a local file, the following table governs what should happen:
| # | local file | remote key | hash of same key | hash of other keys | action |
| 1 | exists | exists | matches | - | do nothing |
| 2 | exists | is missing | - | matches | copy from other key |
| 3 | exists | is missing | - | no matches | upload |
| 4 | exists | exists | no match | matches | copy from other key |
| 5 | exists | exists | no match | no matches | upload |
| 6 | is missing | exists | - | - | delete |
Executable JAR
To build as an executable jar, perform `sbt assembly`
This will create the file `cli/target/scala-2.12/thorp`
Copy this file to your `PATH`.