Commit graph

393 commits

Author SHA1 Message Date
Scala Steward
4934f9ff81 Update zio, zio-streams to 1.0.0-RC15 (#241)
* Update zio, zio-streams to 1.0.0-RC15

* [app] Update to use ZEnv
2019-10-27 20:27:02 +00:00
f35ea9795d
Create and use a cache of hashes for local files (#249)
* [domain] Define Hashes in domain package

* [filesystem] Load and parse any .thorp.cache files found

* [filesystem] Use cached file data when available and up-to-date

* [lib] FileScanner refactoring

* [filesystem] scan sub-dirs first to minimise time cache is on heap

* [filesystem] Write new cache data to temp file

* [lib] replace cache file when finished updating

* [filesystem] AppendLines to correct file with new lines

* [domain] decode HashType from String

* [filesystem] Store last modified time as epoch milliseconds

* [filesystem] parse lastmodified as a long

* [filesystem] use all hash values in cache

* [lib] FileScanner rearrange code

* [lib] Create and use a single cache file per source

* [storage-aws] Use ETag hash from cache when available

* [filesystem] Merge file data together correctly

* [filesystem] Handle exceptions thrown by Files.mode correctly

* [readme] Add section on caching

* [changelog] updated

* [changelog] add pending dependencies notes

* [lib] Filters should not name methods after their defining object

* [lib] Fix up test
2019-10-27 19:53:00 +00:00
Scala Steward
ed1f0ec7ee Update sbt-bloop to 1.3.5 (#247) 2019-10-27 17:36:53 +00:00
Scala Steward
5ec9627296 Update sbt to 1.3.3 (#238) 2019-10-17 11:38:56 +01:00
82d509361a Revert "[sbt] Rollback sbt-ci-release to 1.3.2 (#231)"
This reverts commit a41a99a88a.
2019-10-09 07:30:11 +01:00
1bca90c815 [readme] minor update 2019-10-08 19:35:24 +01:00
c974f05ade [changelog] Updated 2019-10-08 19:34:32 +01:00
a41a99a88a
[sbt] Rollback sbt-ci-release to 1.3.2 (#231) 2019-10-08 18:45:11 +01:00
1ef912ceb3
[config] Accept ‘parallel’ in config files (#230) 2019-10-08 13:58:40 +01:00
Scala Steward
aee0c50733 Update sbt-bloop to 1.3.4 (#228) 2019-10-08 13:32:38 +01:00
Scala Steward
1cbf3b4aaf Update zio, zio-streams to 1.0.0-RC14 (#226) 2019-10-08 13:31:55 +01:00
Scala Steward
a2fec47d94 Update sbt-bloop to 1.3.3 (#224) 2019-10-03 12:04:59 +01:00
86a22215cd
[lib] Perform delete operations in parallel (#223) 2019-10-03 12:01:05 +01:00
61231b8e97
[uishell] ProgressUI uses only 2 line per file (#221) 2019-09-29 20:01:46 +01:00
d4d6d7ffbe [changelog] updated 2019-09-28 19:07:35 +01:00
22d2cf4a57
Progress UI should be updated when upload fails (#218)
* [uishell] Rename UIRequestCycle as ProgressUI

* [uishell] Rename ProgressUI.handle as requestCycle

* [uishell] Remove upload from ProgressUI upon error

* [uishell] Remove upload from ProgressUI upon completion
2019-09-28 17:42:45 +01:00
Scala Steward
7b567ab28c Update aws-java-sdk-s3 to 1.11.641 (#217) 2019-09-28 17:21:25 +01:00
fde500cf53 [changelog] Updated 2019-09-27 16:12:36 +01:00
844d44596b [readme] updated 2019-09-27 16:12:28 +01:00
Scala Steward
ce0a826e6d Update jackson-dataformat-cbor to 2.10.0 (#216) 2019-09-27 16:08:37 +01:00
06dd4f8fed
Support multiple parallel uploads (#188)
* [filesystem] listFiles no longer returns errors

* [cli,config] Take parallel parameter

* [config] Config add .parallel

* [lib] Perform copy and upload in parallel

* [uishell] Extract UIRequestCycle

* [uishell] Display all pending uploads progress

* [app] Always display version

* [app] Highlight version

* [uishell] UIRequestCycle refactoring

* [uishell] UIShell Don’t hide chosen actions in batch mode

* [uishell] UIShell fix typo

* [console] ConsoleOut fix typo
2019-09-27 16:08:16 +01:00
Scala Steward
f51501b13d Update jackson-databind to 2.10.0 (#214) 2019-09-27 15:47:44 +01:00
Scala Steward
cbd59254b1 Update sbt-ci-release to 1.4.31 (#211) 2019-09-24 18:50:16 +01:00
2a52668849 [changelog] updated 2019-09-23 15:30:11 +01:00
5214bacc0b
Not wrapping exceptions thrown in waitForUploadResult (#162)
* [storage-aws] Uploader move implementation to companion

* [app] Program Refactoring

* [storage-aws] AmazonTransferManager refactoring

* [lib] UnversionedMirrorArchive refactoring

* [console] Add Console.putStr

* [uishell] UIShell show chosen actions

* [storage-aws] AmazonTransferManager try to handle errors

* [uishell] UIShell avoid line wrap with long file paths

* [storage] Log when fetching remote summaries

* Handle exceptions thrown in waitForUploadResult

* [uishell] log errors

* [console] Swap batch/non-batch error messages

* fix tests
2019-09-23 13:30:34 +01:00
Scala Steward
6adffd8da7 Update jackson-dataformat-cbor to 2.9.10 (#208)
* Update jackson-dataformat-cbor to 2.9.10

* Update jackson-dataformat-cbor to 2.9.10
2019-09-23 12:02:40 +01:00
Scala Steward
75d23b25f0 Update jackson-databind to 2.9.10 (#206) 2019-09-22 23:13:28 +01:00
Scala Steward
517cba061d Update sbt to 1.3.2 (#207) 2019-09-22 23:12:49 +01:00
Scala Steward
cfc284185b Update zio, zio-streams to 1.0.0-RC13 (#209) 2019-09-22 23:08:11 +01:00
Scala Steward
ef0d867b50 Update aws-java-sdk-s3 to 1.11.635 (#203) 2019-09-20 11:06:18 +01:00
Scala Steward
db123a591e Update sbt to 1.3.1 (#204) 2019-09-20 11:04:27 +01:00
Scala Steward
bc8c6b21fd Update sbt-wartremover to 2.4.3 (#202) 2019-09-20 11:03:34 +01:00
Scala Steward
df2ef41d7a Update sbt-ci-release to 1.3.2 (#200) 2019-09-20 11:02:53 +01:00
1d1a20de67 [changelog] update 2019-09-11 23:02:01 +01:00
Scala Steward
96401858b9 Update eip-zio to 0.3.2 (#193) 2019-09-11 22:49:31 +01:00
a2c061d655
Upload progress via uievent (#196)
* [uishell] Fix package name

* [sbt] Update eip-zio from 0.3.1 to 0.3.2

* [sbt] add dependency on eip-zio to domain

* [uishell] move Upload*Event* to uishell

* UploadEventListener
* UploadEventLogger
* UploadProgressEvent

* [uishell] Log upload progress using UIShell

* [uishell] inline UploadEventLogger into UIShell

* [uishell] Remove println and use Console
2019-09-11 22:47:42 +01:00
3d4c238030
Add more tests (#192)
* [storage-aws] Refactoring

* [lib] Add test for FileScanner

* [lib] Add LocalFileSystemTest for scanCopyUpload

* [lib] Add LocalFileSystem tests for scanDelete

Also send a UIEvent for ActionChosen(ToDelete)

* [lib] LocalFileSystemTest can handle files in any order

* [lib] LocalFileSystemTest can handle files in any order #2

Don't include accountCounter or byteCounter are the order can change.

* [domain] Remove LocalFile.relativeToSource

* [sbt] Add missing modules to aggregate
2019-09-08 18:59:43 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
777bc970d7
Enable Coverage reporting to Codecov (#189)
* [sbt] Enable Jacoco XML reports

* [travis] Add coverage step with codacy

* [sbt] Replace jacoco with scoverage

Jacoco doesn't appear to support scala code properly, only counting
some lines of a method.
2019-09-07 19:32:43 +01:00
9ebca7e6a6 [changelog] updated 2019-09-07 10:29:06 +01:00
Scala Steward
00f45cd4dc Update sbt-ci-release to 1.3.1 (#186) 2019-09-07 09:48:43 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
3d4dc956d0
[sbt] Add sbt-jacoco plugin for code coverage (#184) 2019-09-06 20:31:24 +01:00
Scala Steward
911d8bc9e4 Update sbt to 1.3.0 (#180) 2019-09-06 20:24:51 +01:00
Scala Steward
5d7e106e83 Update zio to 1.0.0-RC12-1 (#178) 2019-09-03 18:27:59 +01:00
321773f04c
Upgrade to Scala 2.13.0 (#176) 2019-09-01 21:30:16 +01:00
64a01dea0c
Add GitHub Action 2019-08-28 07:16:42 +01:00
551988d6b8
[core] PlanExecutor handles building large action list better (#163)
The original code had been able to handle this, but the extraction the
a separate module introduced the stackoverflow.
2019-08-17 18:41:34 +01:00
Scala Steward
2be82eafa2 Update scalamock to 4.4.0 (#165) 2019-08-17 18:27:22 +01:00
656f3fec30 [core] Extract PlanExecutor 2019-08-14 22:51:06 +01:00