Commit graph

75 commits

Author SHA1 Message Date
dependabot-preview[bot]
f3e6fba4e6
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.853 to 1.11.856 (#526) 2020-09-07 06:54:45 +00:00
dependabot-preview[bot]
3b567e1603
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.840 to 1.11.853 (#522) 2020-09-03 06:43:05 +00:00
dependabot-preview[bot]
020fbed6d7
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.837 to 1.11.840 (#512) 2020-08-14 08:19:31 +00:00
dependabot-preview[bot]
4de7a881db
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.834 to 1.11.837 (#511) 2020-08-12 08:29:58 +00:00
dependabot-preview[bot]
87898cc9b0
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.820 to 1.11.834 (#510) 2020-08-06 08:13:16 +00:00
dependabot-preview[bot]
58b54b170e
Bump kemitix-aws-java-sdk-s3-wrapper from 1.11.816 to 1.11.820 (#505) 2020-07-13 08:51:13 +00:00
f67e550aab
Use kemitix-aws-java-sdk-s3-wrapper (#503)
* Use kemitix-aws-java-sdk-s3-wrapper

* Update reactor graph
2020-07-11 16:06:33 +01:00
dependabot-preview[bot]
6c8b056ed9
Bump aws-java-sdk-s3 from 1.11.818 to 1.11.819 (#500) 2020-07-09 08:37:41 +00:00
dependabot-preview[bot]
d723e350eb
Bump aws-java-sdk-s3 from 1.11.817 to 1.11.818 (#499) 2020-07-08 08:26:42 +00:00
dependabot-preview[bot]
9246a98e41
Bump aws-java-sdk-s3 from 1.11.816 to 1.11.817 (#498) 2020-07-07 08:46:57 +00:00
dependabot-preview[bot]
7cb53cabb8
Bump aws-java-sdk-s3 from 1.11.815 to 1.11.816 (#496) 2020-07-03 08:22:26 +00:00
dependabot-preview[bot]
3f5be20fd1
Bump aws-java-sdk-s3 from 1.11.814 to 1.11.815 (#495) 2020-07-02 08:42:43 +00:00
dependabot-preview[bot]
a065caa36b
Bump aws-java-sdk-s3 from 1.11.813 to 1.11.814 (#494) 2020-07-01 08:44:09 +00:00
dependabot-preview[bot]
199f88fd33
Bump aws-java-sdk-s3 from 1.11.812 to 1.11.813 (#493) 2020-06-30 08:24:57 +00:00
dependabot-preview[bot]
8e218a5717
Bump aws-java-sdk-s3 from 1.11.811 to 1.11.812 (#492) 2020-06-29 08:44:04 +00:00
b960f2c8ab
Release 2.0.0 (#491)
* Version set to 2.0.0

* Remove lingering scala plugins
2020-06-27 20:09:45 +01:00
783402f745
Convert lib module to Java (#487)
* lib.FileScannerTest: convert to Java

* lib.Filters: convert to Java

* lib.FiltersTest: convert to Java

* lib.Archive: convert to Java

* lib.LocalFileSystem: convert to Java

* domain.Channel: better channel termination

* domain.Channel: improve assurance that threads die

* Ensure uisink and storage shutdown when exiting program

* domain.HashesTest: clean up

* domain.Channel.run() already shuts down channel when runner finishes

* lib.FileScannerTest: add listener before run

* domain.Channel: add lock around take and only interrupt if waiting

* .run: add intellij test run configurations

* app. update in-code version

* app: chain ui channel construction

* domain.Channel: add tracing option and simplify thread naming

* storage-aws: use default multipartUploadThreshold

* app: force exit once program is completed

* domain.channel: split up and move Channel to its own package

* lib.LocalFileSystemTest: convert to Java

* lib.SequencedAction: convert to Java

* domain.LocalFile: don’t box file length

* lib.FileScannerTest: include hashes

* lib.UnversionedMirrorArchive: convert to Java

* lib: remove scala dependencies
2020-06-27 18:00:22 +01:00
dependabot-preview[bot]
f30021873e
Bump aws-java-sdk-s3 from 1.11.810 to 1.11.811 (#486) 2020-06-26 08:29:26 +00:00
78b1bcf6ac
Convert storage-aws tests to Java (#484)
* storage-aws.ListerTest: convert to Java

* storage-aws: convert to Java
2020-06-26 08:01:29 +01:00
dependabot-preview[bot]
5661c9e7da
Bump aws-java-sdk-s3 from 1.11.809 to 1.11.810 (#482) 2020-06-25 08:41:57 +00:00
dependabot-preview[bot]
085f9f61ec
Bump jackson-dataformat-cbor from 2.11.0 to 2.11.1 (#483) 2020-06-25 08:37:26 +00:00
ebd1910cba
Java rewrite uishell module (#481)
* uishell.UploadEventListener: convert to Java

* uishell.UIShell: convert to Java

* app: shutdown storate once complete

* uishell: remove scala dependencies and plugin
2020-06-25 09:06:10 +01:00
a4bd24ebce
Remove ZIO (#479)
* [BROKEN]Remove ZIO and EIP-ZIO

* [BROKEN]uishell.UIShell.receiver: implement

* [BROKEN]domain.Channel: new implementation replacing MessageChannel

* [BROKEN] use domain.Channel

* Shutdown filescanner channel when finished

* start the uiChannel

* domain.Channel: channel completes once shutdown and queue empty

* uishell: down use eraseLineForward in batch mode

* lib: set file runner before adding listener

* uishell: don’t log do nothing events when not in batch mode

* domain.Channel: if exception in child thread then shutdown channel

* uishell: use correct line endings for showing chosen actions

* domain.Channel: don’t wait for shutdown if not running

* domain: remove legacy MessageChannel

* domain.Channel: don’t hold thread array

* lib.LocalFileSystem: restore delete scanner

* lib.LocalFileSystem: shutdown deletetion channel

* domain.Channel: improved shutdown logic

* clean up
2020-06-24 22:39:16 +01:00
dependabot-preview[bot]
ec4ed099a5
Bump aws-java-sdk-s3 from 1.11.808 to 1.11.809 (#478) 2020-06-24 09:02:10 +00:00
0ae523a1e7
Convert Storage environment to Java (#476)
* uishell.UploadProgressEvent: convert to Java

* uishell.UIEvent: convert to Java

* uishell.ProgressUI: convert to Java

* uishell.ProgressEvent: remove unused

* lib.MessageChannel: added as replacement for eip-zio version

* domain: move MessageChannel to module

* storage: convert to Java
2020-06-23 10:23:28 +01:00
dependabot-preview[bot]
97d0e0d190
Bump aws-java-sdk-s3 from 1.11.807 to 1.11.808 (#477) 2020-06-23 08:57:33 +00:00
dependabot-preview[bot]
9cd271da35
Bump aws-java-sdk-s3 from 1.11.806 to 1.11.807 (#475) 2020-06-22 09:11:22 +00:00
0104d9c08d
Post-release version set to 1.1.0-SNAPSHOT (#472) 2020-06-21 19:42:29 +01:00
53eaeeb75f
Convert console module to Java (#471)
* console.ConsoleOut: convert to Java

* console.Console: convert to Java

* console: remove scala dependencies and plugins

* console.ConsoleOut: remove lingering scala import
2020-06-21 17:46:14 +01:00
5cb929e7e4
Release 1.0.0 (#463)
Release 1.0.0
2020-06-21 07:54:43 +01:00
319c46f403
Convert to Java (domain, config, storage-aws and filesystem) (#446)
* Java rewrite domain (#438)

* domain.Bucket: convert to Java

* domain.LastModified: convert to Java

* domain.QuoteStripper: convert to Java

* domain.HexEncoder: convert to Java

* domain.MD5Hash: convert to Java

* remove unused import

* domain.RemoteKey: convert to Java

* domain.Action: convert to Java

* domain.Counters: convert to Java

* domain.HashType: convert to Java

* domain.Hashes: convert to Java

* domain.MD5HashData: convert to Java

* domain.Filter: convert to Java

* domain.LocalFile: convert to Java

* domain: make immutable field public

* domain.SizeTranslation: convert to Java

* domain.HashType: restrict access to contstructor

* domain.RemoteObjects: convert to Java

Introduce MapView and Tuple.

* domain.Sources: convert to Java

* domain.StorageEvent: convert to Java

* domain.Terminal: convert to Java

* domain => config: move SimpleLens to only module that uses it

* domain => filesystem: move TemporaryFolder

* domain.Implicits: removed

* parent: make junit, et al available

* domain: add testing dependencies

* domain.HexEncoder: convert test to Java and fix bugs

* domain.HexEncoderTest: replace with Java version

* domain.MD5HashTest: convert to Java

* domain.RemoteKeyTest: convert to Java

* domain.SizeTranslationTest: convert to Java

* domain.TerminalTest: convert to Java

* domain: remove unused dependencies

* parent: rollback zio-streams to match zio and pin them together

* storage-aws: resolve transitive dependency conflicts

* Java rewrite storage aws (#445)

* storage-aws.AmazonS3: convert to Java as AmazonS3Client

* storage-aws.S3Copier: convert to Java

* storage-aws.S3Uploader: convert to Java

* storage-aws.S3Deleter: convert to Java

* storage-aws.S3Lister: convert to Java

* filesystem: write cache data correctly (as supplied)

* domain,filesystem: fix MD5Hash generation

* filesystem: convert to Java (#450)

* remove legacy

* Rewrite config module in Java (#461)

* config.ParseConfigFile: convert to Java

* config.ParseConfigFile: convert to Java

* config.SourceConfigLoader: convert to Java

* WIP config.Configuration: convert to Java

* config.ConfigOption: convert to Java

* config.ConfigOptions: convert to Java

* config.ConfigValidation: convert to Java

* config.ConfigQuery: convert to Java

* config: move classes to correct location

* config.ConfigValidationException: convert to Java

* config.ConfigValidator: convert to Java

* config.ConfigurationBuilder: convert to Java

* config.SimpleLens: removed

* config.Config: remove environment

* config.ConfigOptionTest: convert to Java

* config.ConfigQueryTest: convert to Java

* config.ConfigurationBuilderTest: convert to Java

* config.ParseConfigFileTest: convert to Java

* config.ParseConfigLinesTest: convert to Java

* config: remove scala dependencies and plugin
2020-06-21 07:21:21 +01:00
dependabot-preview[bot]
bae324e15d
Bump aws-java-sdk-s3 from 1.11.805 to 1.11.806 (#457) 2020-06-19 08:29:46 +00:00
dependabot-preview[bot]
9e9babb72e
Bump aws-java-sdk-s3 from 1.11.804 to 1.11.805 (#454) 2020-06-18 08:32:15 +00:00
dependabot-preview[bot]
b3d0bff3fb
Bump aws-java-sdk-s3 from 1.11.803 to 1.11.804 (#452) 2020-06-17 08:38:21 +00:00
dependabot-preview[bot]
99aeb8581d
Bump aws-java-sdk-s3 from 1.11.802 to 1.11.803 (#449) 2020-06-16 08:45:19 +00:00
dependabot-preview[bot]
8b1b3a8011
Bump aws-java-sdk-s3 from 1.11.801 to 1.11.802 (#447) 2020-06-15 09:06:26 +00:00
dependabot-preview[bot]
fc00be8707
Bump scala-library from 2.13.0 to 2.13.2 (#439)
* Bump scala-library from 2.13.0 to 2.13.2

Bumps [scala-library](https://github.com/scala/scala) from 2.13.0 to 2.13.2.
- [Release notes](https://github.com/scala/scala/releases)
- [Commits](https://github.com/scala/scala/compare/v2.13.0...v2.13.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update scala syntax from 2.13.0 to 2.13.2 (#443)

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Paul Campbell <pcampbell@kemitix.net>
2020-06-12 17:00:02 +01:00
dependabot-preview[bot]
d229745305
Bump aws-java-sdk-s3 from 1.11.800 to 1.11.801 (#442) 2020-06-12 08:46:27 +00:00
dependabot-preview[bot]
3ccee2dcc7
Bump aws-java-sdk-s3 from 1.11.792 to 1.11.800 (#436) 2020-06-11 20:47:44 +00:00
dependabot-preview[bot]
b9f2c5685e
Bump jackson-dataformat-cbor from 2.10.4 to 2.11.0 (#432) 2020-06-11 20:40:32 +00:00
baea18f3f3
Java rewrite - step 1 - build with Maven (#431)
* Add root and parent pom.xml files

* parent: add scala-maven-plugin to pluginManagement

* fix packaging

* domain: add module

* parent,domain: add dependencies

* maven: add filesystem module

* parent: disable coverage and pitest

* domain: move classes from test to main as they are used in filesystem

* config: add module

* console: add module

* Add reactor-graph

* fix typo

* uishell: add module

* Restore explicit dependencies

Don’t depend upon transitive dependencies if we want to be able to restructure module dependencies.

* storage: add module

* lib: add module

Included adding a Resource case class in filesystem to replace one generated by a scala plugin

* storage-aws: add module

* cli: add module

* app: add module

* app: make jar runnable

* Updated build instructions in README

* Convert readme to markdown

* Add reactor graph to readme

* Switch Guthub Actions to using Maven to build
2020-06-11 21:33:28 +01:00
f35ea9795d
Create and use a cache of hashes for local files (#249)
* [domain] Define Hashes in domain package

* [filesystem] Load and parse any .thorp.cache files found

* [filesystem] Use cached file data when available and up-to-date

* [lib] FileScanner refactoring

* [filesystem] scan sub-dirs first to minimise time cache is on heap

* [filesystem] Write new cache data to temp file

* [lib] replace cache file when finished updating

* [filesystem] AppendLines to correct file with new lines

* [domain] decode HashType from String

* [filesystem] Store last modified time as epoch milliseconds

* [filesystem] parse lastmodified as a long

* [filesystem] use all hash values in cache

* [lib] FileScanner rearrange code

* [lib] Create and use a single cache file per source

* [storage-aws] Use ETag hash from cache when available

* [filesystem] Merge file data together correctly

* [filesystem] Handle exceptions thrown by Files.mode correctly

* [readme] Add section on caching

* [changelog] updated

* [changelog] add pending dependencies notes

* [lib] Filters should not name methods after their defining object

* [lib] Fix up test
2019-10-27 19:53:00 +00:00
5214bacc0b
Not wrapping exceptions thrown in waitForUploadResult (#162)
* [storage-aws] Uploader move implementation to companion

* [app] Program Refactoring

* [storage-aws] AmazonTransferManager refactoring

* [lib] UnversionedMirrorArchive refactoring

* [console] Add Console.putStr

* [uishell] UIShell show chosen actions

* [storage-aws] AmazonTransferManager try to handle errors

* [uishell] UIShell avoid line wrap with long file paths

* [storage] Log when fetching remote summaries

* Handle exceptions thrown in waitForUploadResult

* [uishell] log errors

* [console] Swap batch/non-batch error messages

* fix tests
2019-09-23 13:30:34 +01:00
a2c061d655
Upload progress via uievent (#196)
* [uishell] Fix package name

* [sbt] Update eip-zio from 0.3.1 to 0.3.2

* [sbt] add dependency on eip-zio to domain

* [uishell] move Upload*Event* to uishell

* UploadEventListener
* UploadEventLogger
* UploadProgressEvent

* [uishell] Log upload progress using UIShell

* [uishell] inline UploadEventLogger into UIShell

* [uishell] Remove println and use Console
2019-09-11 22:47:42 +01:00
3d4c238030
Add more tests (#192)
* [storage-aws] Refactoring

* [lib] Add test for FileScanner

* [lib] Add LocalFileSystemTest for scanCopyUpload

* [lib] Add LocalFileSystem tests for scanDelete

Also send a UIEvent for ActionChosen(ToDelete)

* [lib] LocalFileSystemTest can handle files in any order

* [lib] LocalFileSystemTest can handle files in any order #2

Don't include accountCounter or byteCounter are the order can change.

* [domain] Remove LocalFile.relativeToSource

* [sbt] Add missing modules to aggregate
2019-09-08 18:59:43 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
321773f04c
Upgrade to Scala 2.13.0 (#176) 2019-09-01 21:30:16 +01:00
242804294d [storage-aws] force upload progress updates to strick sequence
Prevents the situation where a second progress update arrives while
the first is still on the process of being processed. The new update
is blocked until the first is completed.
2019-08-14 10:10:23 +01:00
e0beef9d73 [storage-aws] don't create new event listener for every event 2019-08-14 09:23:34 +01:00