Commit graph

15 commits

Author SHA1 Message Date
ee5a831e81
Convert app module to Java (#488)
* app.Program: convert to Java

* app.Main: convert to Java

* app: remove scala dependencies

* parent: remove scala dependencies
2020-06-27 18:50:34 +01:00
783402f745
Convert lib module to Java (#487)
* lib.FileScannerTest: convert to Java

* lib.Filters: convert to Java

* lib.FiltersTest: convert to Java

* lib.Archive: convert to Java

* lib.LocalFileSystem: convert to Java

* domain.Channel: better channel termination

* domain.Channel: improve assurance that threads die

* Ensure uisink and storage shutdown when exiting program

* domain.HashesTest: clean up

* domain.Channel.run() already shuts down channel when runner finishes

* lib.FileScannerTest: add listener before run

* domain.Channel: add lock around take and only interrupt if waiting

* .run: add intellij test run configurations

* app. update in-code version

* app: chain ui channel construction

* domain.Channel: add tracing option and simplify thread naming

* storage-aws: use default multipartUploadThreshold

* app: force exit once program is completed

* domain.channel: split up and move Channel to its own package

* lib.LocalFileSystemTest: convert to Java

* lib.SequencedAction: convert to Java

* domain.LocalFile: don’t box file length

* lib.FileScannerTest: include hashes

* lib.UnversionedMirrorArchive: convert to Java

* lib: remove scala dependencies
2020-06-27 18:00:22 +01:00
ebd1910cba
Java rewrite uishell module (#481)
* uishell.UploadEventListener: convert to Java

* uishell.UIShell: convert to Java

* app: shutdown storate once complete

* uishell: remove scala dependencies and plugin
2020-06-25 09:06:10 +01:00
a4bd24ebce
Remove ZIO (#479)
* [BROKEN]Remove ZIO and EIP-ZIO

* [BROKEN]uishell.UIShell.receiver: implement

* [BROKEN]domain.Channel: new implementation replacing MessageChannel

* [BROKEN] use domain.Channel

* Shutdown filescanner channel when finished

* start the uiChannel

* domain.Channel: channel completes once shutdown and queue empty

* uishell: down use eraseLineForward in batch mode

* lib: set file runner before adding listener

* uishell: don’t log do nothing events when not in batch mode

* domain.Channel: if exception in child thread then shutdown channel

* uishell: use correct line endings for showing chosen actions

* domain.Channel: don’t wait for shutdown if not running

* domain: remove legacy MessageChannel

* domain.Channel: don’t hold thread array

* lib.LocalFileSystem: restore delete scanner

* lib.LocalFileSystem: shutdown deletetion channel

* domain.Channel: improved shutdown logic

* clean up
2020-06-24 22:39:16 +01:00
0ae523a1e7
Convert Storage environment to Java (#476)
* uishell.UploadProgressEvent: convert to Java

* uishell.UIEvent: convert to Java

* uishell.ProgressUI: convert to Java

* uishell.ProgressEvent: remove unused

* lib.MessageChannel: added as replacement for eip-zio version

* domain: move MessageChannel to module

* storage: convert to Java
2020-06-23 10:23:28 +01:00
0092d163f2
cli: module convert to Java (#473) 2020-06-22 06:36:44 +01:00
53eaeeb75f
Convert console module to Java (#471)
* console.ConsoleOut: convert to Java

* console.Console: convert to Java

* console: remove scala dependencies and plugins

* console.ConsoleOut: remove lingering scala import
2020-06-21 17:46:14 +01:00
319c46f403
Convert to Java (domain, config, storage-aws and filesystem) (#446)
* Java rewrite domain (#438)

* domain.Bucket: convert to Java

* domain.LastModified: convert to Java

* domain.QuoteStripper: convert to Java

* domain.HexEncoder: convert to Java

* domain.MD5Hash: convert to Java

* remove unused import

* domain.RemoteKey: convert to Java

* domain.Action: convert to Java

* domain.Counters: convert to Java

* domain.HashType: convert to Java

* domain.Hashes: convert to Java

* domain.MD5HashData: convert to Java

* domain.Filter: convert to Java

* domain.LocalFile: convert to Java

* domain: make immutable field public

* domain.SizeTranslation: convert to Java

* domain.HashType: restrict access to contstructor

* domain.RemoteObjects: convert to Java

Introduce MapView and Tuple.

* domain.Sources: convert to Java

* domain.StorageEvent: convert to Java

* domain.Terminal: convert to Java

* domain => config: move SimpleLens to only module that uses it

* domain => filesystem: move TemporaryFolder

* domain.Implicits: removed

* parent: make junit, et al available

* domain: add testing dependencies

* domain.HexEncoder: convert test to Java and fix bugs

* domain.HexEncoderTest: replace with Java version

* domain.MD5HashTest: convert to Java

* domain.RemoteKeyTest: convert to Java

* domain.SizeTranslationTest: convert to Java

* domain.TerminalTest: convert to Java

* domain: remove unused dependencies

* parent: rollback zio-streams to match zio and pin them together

* storage-aws: resolve transitive dependency conflicts

* Java rewrite storage aws (#445)

* storage-aws.AmazonS3: convert to Java as AmazonS3Client

* storage-aws.S3Copier: convert to Java

* storage-aws.S3Uploader: convert to Java

* storage-aws.S3Deleter: convert to Java

* storage-aws.S3Lister: convert to Java

* filesystem: write cache data correctly (as supplied)

* domain,filesystem: fix MD5Hash generation

* filesystem: convert to Java (#450)

* remove legacy

* Rewrite config module in Java (#461)

* config.ParseConfigFile: convert to Java

* config.ParseConfigFile: convert to Java

* config.SourceConfigLoader: convert to Java

* WIP config.Configuration: convert to Java

* config.ConfigOption: convert to Java

* config.ConfigOptions: convert to Java

* config.ConfigValidation: convert to Java

* config.ConfigQuery: convert to Java

* config: move classes to correct location

* config.ConfigValidationException: convert to Java

* config.ConfigValidator: convert to Java

* config.ConfigurationBuilder: convert to Java

* config.SimpleLens: removed

* config.Config: remove environment

* config.ConfigOptionTest: convert to Java

* config.ConfigQueryTest: convert to Java

* config.ConfigurationBuilderTest: convert to Java

* config.ParseConfigFileTest: convert to Java

* config.ParseConfigLinesTest: convert to Java

* config: remove scala dependencies and plugin
2020-06-21 07:21:21 +01:00
baea18f3f3
Java rewrite - step 1 - build with Maven (#431)
* Add root and parent pom.xml files

* parent: add scala-maven-plugin to pluginManagement

* fix packaging

* domain: add module

* parent,domain: add dependencies

* maven: add filesystem module

* parent: disable coverage and pitest

* domain: move classes from test to main as they are used in filesystem

* config: add module

* console: add module

* Add reactor-graph

* fix typo

* uishell: add module

* Restore explicit dependencies

Don’t depend upon transitive dependencies if we want to be able to restructure module dependencies.

* storage: add module

* lib: add module

Included adding a Resource case class in filesystem to replace one generated by a scala plugin

* storage-aws: add module

* cli: add module

* app: add module

* app: make jar runnable

* Updated build instructions in README

* Convert readme to markdown

* Add reactor graph to readme

* Switch Guthub Actions to using Maven to build
2020-06-11 21:33:28 +01:00
Scala Steward
4934f9ff81 Update zio, zio-streams to 1.0.0-RC15 (#241)
* Update zio, zio-streams to 1.0.0-RC15

* [app] Update to use ZEnv
2019-10-27 20:27:02 +00:00
06dd4f8fed
Support multiple parallel uploads (#188)
* [filesystem] listFiles no longer returns errors

* [cli,config] Take parallel parameter

* [config] Config add .parallel

* [lib] Perform copy and upload in parallel

* [uishell] Extract UIRequestCycle

* [uishell] Display all pending uploads progress

* [app] Always display version

* [app] Highlight version

* [uishell] UIRequestCycle refactoring

* [uishell] UIShell Don’t hide chosen actions in batch mode

* [uishell] UIShell fix typo

* [console] ConsoleOut fix typo
2019-09-27 16:08:16 +01:00
5214bacc0b
Not wrapping exceptions thrown in waitForUploadResult (#162)
* [storage-aws] Uploader move implementation to companion

* [app] Program Refactoring

* [storage-aws] AmazonTransferManager refactoring

* [lib] UnversionedMirrorArchive refactoring

* [console] Add Console.putStr

* [uishell] UIShell show chosen actions

* [storage-aws] AmazonTransferManager try to handle errors

* [uishell] UIShell avoid line wrap with long file paths

* [storage] Log when fetching remote summaries

* Handle exceptions thrown in waitForUploadResult

* [uishell] log errors

* [console] Swap batch/non-batch error messages

* fix tests
2019-09-23 13:30:34 +01:00
a2c061d655
Upload progress via uievent (#196)
* [uishell] Fix package name

* [sbt] Update eip-zio from 0.3.1 to 0.3.2

* [sbt] add dependency on eip-zio to domain

* [uishell] move Upload*Event* to uishell

* UploadEventListener
* UploadEventLogger
* UploadProgressEvent

* [uishell] Log upload progress using UIShell

* [uishell] inline UploadEventLogger into UIShell

* [uishell] Remove println and use Console
2019-09-11 22:47:42 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00