Commit graph

10 commits

Author SHA1 Message Date
06dd4f8fed
Support multiple parallel uploads (#188)
* [filesystem] listFiles no longer returns errors

* [cli,config] Take parallel parameter

* [config] Config add .parallel

* [lib] Perform copy and upload in parallel

* [uishell] Extract UIRequestCycle

* [uishell] Display all pending uploads progress

* [app] Always display version

* [app] Highlight version

* [uishell] UIRequestCycle refactoring

* [uishell] UIShell Don’t hide chosen actions in batch mode

* [uishell] UIShell fix typo

* [console] ConsoleOut fix typo
2019-09-27 16:08:16 +01:00
5214bacc0b
Not wrapping exceptions thrown in waitForUploadResult (#162)
* [storage-aws] Uploader move implementation to companion

* [app] Program Refactoring

* [storage-aws] AmazonTransferManager refactoring

* [lib] UnversionedMirrorArchive refactoring

* [console] Add Console.putStr

* [uishell] UIShell show chosen actions

* [storage-aws] AmazonTransferManager try to handle errors

* [uishell] UIShell avoid line wrap with long file paths

* [storage] Log when fetching remote summaries

* Handle exceptions thrown in waitForUploadResult

* [uishell] log errors

* [console] Swap batch/non-batch error messages

* fix tests
2019-09-23 13:30:34 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
af7733952c
Install WartRemover pluging (#150)
* [sbt] Install WartRemover

* remove warts

* remote warts

* fix tests

* [domain] UploadEventListener fix progress bar (again)

* [domain] Remove LastModified - wasn't being used for anything
2019-08-06 18:19:05 +01:00
e3b0260b6d
Refactor ThorpArchive.logEvent(StorageQueueEvent) (#141)
* [core] ThorpArchive Refactoring

* [core] ThorpArchive Refactoring

* [core] ThorpArchive refactored

* [core] ThorpArchive convert output to ConsoleOut.WithBatchMode messages

* [console] Console rename putMessageLnB to overload putMessageLn
2019-08-01 22:39:52 +01:00
a93781007d
Create FileSystem as full ZIO effect module (#135)
* [changelog] updated

* [sbt] Add filesystem module

* [filesystem] stub effect module

* [filesystem] Add fileExists

* [filesystem] Add openFile

* [filesystem] Add fileLines

* [config] Use FileSystem.fileExists

* [filesystem] FileSystem.Test implement test methods

* [filesystem] Replace package object with FS object

* [storage] merge storage package object into Storage object

* [console] merge console package with Console object
2019-07-30 08:07:26 +01:00
ccefd286f9
Convert Config to full ZIO effect module (#134)
* [config] new module

* [config] stub module

* [domain] Rename domain.Config as domain.LegacyConfig

* [config] Move LegacyConfig to config module

* [config] Move config parsing and validation into module

* [config] Complete migration to module for Config

* [config] Config You should not name methods after their defining object

* [config] Rename LegacyConfig as Configuration

Also remove redundant uses

* [core] LocalFileStream Refactoring

* [changelog] update
2019-07-28 21:47:01 +01:00
96a83e6c3e
Convert Storage to full ZIO effect module (#133)
* [console] Rename MyConsole as Console

* [console] break infinite loop

* [console] fix typo

* [console] clean up helpers

* [cli] Main use ZIO#provide to run program

* [cli] Main define Program type alias

* [cli] Program handle cli args in Program

* [cli] Program doesn't extend PlanBuilder

* [cli] refactoring

* [cli] rename ParseArgs as CliArgs

* [cli] CliArgs#apply renamed a parse

* [storage-aws] S3StorageService renamed as S3Storage

* [storage-api] Rename StorageService as Storage.Service

* [storage-api] make Storage.copy effectTotal

* [storage-api] make Storage.delete effectTotal

* [storage-api] make Storage.shutdown effectTotal

* [storage-api] make Storage.upload effectTotal

* [storage-aws] Lister refactoring

* [storage-aws] make Lister into a trait

* [storage-aws] make Copier into a trait

* [storage-aws] make Deleter into a trait

* [storate-aws] make Uploader into a trait

* [storage-aws] AmazonS3 move error handling out of client wrapper

* [storage-aws] DeleterTest added

* [storage-aws] ListerTest added

* [storage-aws] Uploader refactoring

* [storage-aws] CopierTest test Copier directly

* [storage-aws] DeleterTest test Deleter directly

* [storate-aws] ListerTest test Lister directly

* [storage-aws] UploaderTest added

* [storage-aws] S3Storage.Live replaces S3StorageServiceBuilder

* Complete migration to Module for Storage

* [cli] Main define LiveThorpApp object

* [core] Add CoreTypes

* [cli] Program Refactoring

* [core] PlanBuilding Refactoring

* [changelog] updated

* [console] Console.Live Usage of get on optional type

* [storage-aws] AmazonS3ClientTestFixture Use wildcards when selecting more than 6 elements
2019-07-28 20:11:03 +01:00
f277b5e789
Improve logging consistency (#119)
* [console] create new effect MyConsole

* [core] SyncLogging use ConsoleOut.ValidConfig

* [storage-aws] Extract AmazonS3 trait to help testing

* [console] MyConsole use UIO

* [storage-aws] S3StorageServiceSuite convert to FreeSpec

* [storage-aws] S3StorageServiceSuite extract S3ClientTest trait

* [storage-aws] remove incomplete test UploaderSuite

* [storage-aws] extract and rename AmazonS3ClientTestFixture

* [storage-aws] Copier handle and log errors

* [core] ThropArchive log completed uploads here

* [core] ThorpArchive log copies and deletes

* Improve consistency of logging

* [storage-aws] CopierTest extract from S3StorageServiceSuite

* [storage-aws] Copier handle hash match errors properly

* [core] ThropArchive display erros in red

* [core] SequencePlan extracted

* [core] Clear line after deletion log

* [changelog] updated

* [cli] Program tidy up imports

* [storage-aws] Copier replace null with Option

* [storage-aws] AmazonS3 copyObject returns Some[CopyObjectResult]
2019-07-23 23:13:09 +01:00