Commit graph

65 commits

Author SHA1 Message Date
dependabot-preview[bot]
7cb53cabb8
Bump aws-java-sdk-s3 from 1.11.815 to 1.11.816 (#496) 2020-07-03 08:22:26 +00:00
dependabot-preview[bot]
3f5be20fd1
Bump aws-java-sdk-s3 from 1.11.814 to 1.11.815 (#495) 2020-07-02 08:42:43 +00:00
dependabot-preview[bot]
a065caa36b
Bump aws-java-sdk-s3 from 1.11.813 to 1.11.814 (#494) 2020-07-01 08:44:09 +00:00
dependabot-preview[bot]
199f88fd33
Bump aws-java-sdk-s3 from 1.11.812 to 1.11.813 (#493) 2020-06-30 08:24:57 +00:00
dependabot-preview[bot]
8e218a5717
Bump aws-java-sdk-s3 from 1.11.811 to 1.11.812 (#492) 2020-06-29 08:44:04 +00:00
b960f2c8ab
Release 2.0.0 (#491)
* Version set to 2.0.0

* Remove lingering scala plugins
2020-06-27 20:09:45 +01:00
783402f745
Convert lib module to Java (#487)
* lib.FileScannerTest: convert to Java

* lib.Filters: convert to Java

* lib.FiltersTest: convert to Java

* lib.Archive: convert to Java

* lib.LocalFileSystem: convert to Java

* domain.Channel: better channel termination

* domain.Channel: improve assurance that threads die

* Ensure uisink and storage shutdown when exiting program

* domain.HashesTest: clean up

* domain.Channel.run() already shuts down channel when runner finishes

* lib.FileScannerTest: add listener before run

* domain.Channel: add lock around take and only interrupt if waiting

* .run: add intellij test run configurations

* app. update in-code version

* app: chain ui channel construction

* domain.Channel: add tracing option and simplify thread naming

* storage-aws: use default multipartUploadThreshold

* app: force exit once program is completed

* domain.channel: split up and move Channel to its own package

* lib.LocalFileSystemTest: convert to Java

* lib.SequencedAction: convert to Java

* domain.LocalFile: don’t box file length

* lib.FileScannerTest: include hashes

* lib.UnversionedMirrorArchive: convert to Java

* lib: remove scala dependencies
2020-06-27 18:00:22 +01:00
dependabot-preview[bot]
f30021873e
Bump aws-java-sdk-s3 from 1.11.810 to 1.11.811 (#486) 2020-06-26 08:29:26 +00:00
78b1bcf6ac
Convert storage-aws tests to Java (#484)
* storage-aws.ListerTest: convert to Java

* storage-aws: convert to Java
2020-06-26 08:01:29 +01:00
dependabot-preview[bot]
5661c9e7da
Bump aws-java-sdk-s3 from 1.11.809 to 1.11.810 (#482) 2020-06-25 08:41:57 +00:00
dependabot-preview[bot]
085f9f61ec
Bump jackson-dataformat-cbor from 2.11.0 to 2.11.1 (#483) 2020-06-25 08:37:26 +00:00
ebd1910cba
Java rewrite uishell module (#481)
* uishell.UploadEventListener: convert to Java

* uishell.UIShell: convert to Java

* app: shutdown storate once complete

* uishell: remove scala dependencies and plugin
2020-06-25 09:06:10 +01:00
a4bd24ebce
Remove ZIO (#479)
* [BROKEN]Remove ZIO and EIP-ZIO

* [BROKEN]uishell.UIShell.receiver: implement

* [BROKEN]domain.Channel: new implementation replacing MessageChannel

* [BROKEN] use domain.Channel

* Shutdown filescanner channel when finished

* start the uiChannel

* domain.Channel: channel completes once shutdown and queue empty

* uishell: down use eraseLineForward in batch mode

* lib: set file runner before adding listener

* uishell: don’t log do nothing events when not in batch mode

* domain.Channel: if exception in child thread then shutdown channel

* uishell: use correct line endings for showing chosen actions

* domain.Channel: don’t wait for shutdown if not running

* domain: remove legacy MessageChannel

* domain.Channel: don’t hold thread array

* lib.LocalFileSystem: restore delete scanner

* lib.LocalFileSystem: shutdown deletetion channel

* domain.Channel: improved shutdown logic

* clean up
2020-06-24 22:39:16 +01:00
dependabot-preview[bot]
ec4ed099a5
Bump aws-java-sdk-s3 from 1.11.808 to 1.11.809 (#478) 2020-06-24 09:02:10 +00:00
0ae523a1e7
Convert Storage environment to Java (#476)
* uishell.UploadProgressEvent: convert to Java

* uishell.UIEvent: convert to Java

* uishell.ProgressUI: convert to Java

* uishell.ProgressEvent: remove unused

* lib.MessageChannel: added as replacement for eip-zio version

* domain: move MessageChannel to module

* storage: convert to Java
2020-06-23 10:23:28 +01:00
dependabot-preview[bot]
97d0e0d190
Bump aws-java-sdk-s3 from 1.11.807 to 1.11.808 (#477) 2020-06-23 08:57:33 +00:00
dependabot-preview[bot]
9cd271da35
Bump aws-java-sdk-s3 from 1.11.806 to 1.11.807 (#475) 2020-06-22 09:11:22 +00:00
0104d9c08d
Post-release version set to 1.1.0-SNAPSHOT (#472) 2020-06-21 19:42:29 +01:00
53eaeeb75f
Convert console module to Java (#471)
* console.ConsoleOut: convert to Java

* console.Console: convert to Java

* console: remove scala dependencies and plugins

* console.ConsoleOut: remove lingering scala import
2020-06-21 17:46:14 +01:00
5cb929e7e4
Release 1.0.0 (#463)
Release 1.0.0
2020-06-21 07:54:43 +01:00
319c46f403
Convert to Java (domain, config, storage-aws and filesystem) (#446)
* Java rewrite domain (#438)

* domain.Bucket: convert to Java

* domain.LastModified: convert to Java

* domain.QuoteStripper: convert to Java

* domain.HexEncoder: convert to Java

* domain.MD5Hash: convert to Java

* remove unused import

* domain.RemoteKey: convert to Java

* domain.Action: convert to Java

* domain.Counters: convert to Java

* domain.HashType: convert to Java

* domain.Hashes: convert to Java

* domain.MD5HashData: convert to Java

* domain.Filter: convert to Java

* domain.LocalFile: convert to Java

* domain: make immutable field public

* domain.SizeTranslation: convert to Java

* domain.HashType: restrict access to contstructor

* domain.RemoteObjects: convert to Java

Introduce MapView and Tuple.

* domain.Sources: convert to Java

* domain.StorageEvent: convert to Java

* domain.Terminal: convert to Java

* domain => config: move SimpleLens to only module that uses it

* domain => filesystem: move TemporaryFolder

* domain.Implicits: removed

* parent: make junit, et al available

* domain: add testing dependencies

* domain.HexEncoder: convert test to Java and fix bugs

* domain.HexEncoderTest: replace with Java version

* domain.MD5HashTest: convert to Java

* domain.RemoteKeyTest: convert to Java

* domain.SizeTranslationTest: convert to Java

* domain.TerminalTest: convert to Java

* domain: remove unused dependencies

* parent: rollback zio-streams to match zio and pin them together

* storage-aws: resolve transitive dependency conflicts

* Java rewrite storage aws (#445)

* storage-aws.AmazonS3: convert to Java as AmazonS3Client

* storage-aws.S3Copier: convert to Java

* storage-aws.S3Uploader: convert to Java

* storage-aws.S3Deleter: convert to Java

* storage-aws.S3Lister: convert to Java

* filesystem: write cache data correctly (as supplied)

* domain,filesystem: fix MD5Hash generation

* filesystem: convert to Java (#450)

* remove legacy

* Rewrite config module in Java (#461)

* config.ParseConfigFile: convert to Java

* config.ParseConfigFile: convert to Java

* config.SourceConfigLoader: convert to Java

* WIP config.Configuration: convert to Java

* config.ConfigOption: convert to Java

* config.ConfigOptions: convert to Java

* config.ConfigValidation: convert to Java

* config.ConfigQuery: convert to Java

* config: move classes to correct location

* config.ConfigValidationException: convert to Java

* config.ConfigValidator: convert to Java

* config.ConfigurationBuilder: convert to Java

* config.SimpleLens: removed

* config.Config: remove environment

* config.ConfigOptionTest: convert to Java

* config.ConfigQueryTest: convert to Java

* config.ConfigurationBuilderTest: convert to Java

* config.ParseConfigFileTest: convert to Java

* config.ParseConfigLinesTest: convert to Java

* config: remove scala dependencies and plugin
2020-06-21 07:21:21 +01:00
dependabot-preview[bot]
bae324e15d
Bump aws-java-sdk-s3 from 1.11.805 to 1.11.806 (#457) 2020-06-19 08:29:46 +00:00
dependabot-preview[bot]
9e9babb72e
Bump aws-java-sdk-s3 from 1.11.804 to 1.11.805 (#454) 2020-06-18 08:32:15 +00:00
dependabot-preview[bot]
b3d0bff3fb
Bump aws-java-sdk-s3 from 1.11.803 to 1.11.804 (#452) 2020-06-17 08:38:21 +00:00
dependabot-preview[bot]
99aeb8581d
Bump aws-java-sdk-s3 from 1.11.802 to 1.11.803 (#449) 2020-06-16 08:45:19 +00:00
dependabot-preview[bot]
8b1b3a8011
Bump aws-java-sdk-s3 from 1.11.801 to 1.11.802 (#447) 2020-06-15 09:06:26 +00:00
dependabot-preview[bot]
fc00be8707
Bump scala-library from 2.13.0 to 2.13.2 (#439)
* Bump scala-library from 2.13.0 to 2.13.2

Bumps [scala-library](https://github.com/scala/scala) from 2.13.0 to 2.13.2.
- [Release notes](https://github.com/scala/scala/releases)
- [Commits](https://github.com/scala/scala/compare/v2.13.0...v2.13.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update scala syntax from 2.13.0 to 2.13.2 (#443)

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Paul Campbell <pcampbell@kemitix.net>
2020-06-12 17:00:02 +01:00
dependabot-preview[bot]
d229745305
Bump aws-java-sdk-s3 from 1.11.800 to 1.11.801 (#442) 2020-06-12 08:46:27 +00:00
dependabot-preview[bot]
3ccee2dcc7
Bump aws-java-sdk-s3 from 1.11.792 to 1.11.800 (#436) 2020-06-11 20:47:44 +00:00
dependabot-preview[bot]
b9f2c5685e
Bump jackson-dataformat-cbor from 2.10.4 to 2.11.0 (#432) 2020-06-11 20:40:32 +00:00
baea18f3f3
Java rewrite - step 1 - build with Maven (#431)
* Add root and parent pom.xml files

* parent: add scala-maven-plugin to pluginManagement

* fix packaging

* domain: add module

* parent,domain: add dependencies

* maven: add filesystem module

* parent: disable coverage and pitest

* domain: move classes from test to main as they are used in filesystem

* config: add module

* console: add module

* Add reactor-graph

* fix typo

* uishell: add module

* Restore explicit dependencies

Don’t depend upon transitive dependencies if we want to be able to restructure module dependencies.

* storage: add module

* lib: add module

Included adding a Resource case class in filesystem to replace one generated by a scala plugin

* storage-aws: add module

* cli: add module

* app: add module

* app: make jar runnable

* Updated build instructions in README

* Convert readme to markdown

* Add reactor graph to readme

* Switch Guthub Actions to using Maven to build
2020-06-11 21:33:28 +01:00
f35ea9795d
Create and use a cache of hashes for local files (#249)
* [domain] Define Hashes in domain package

* [filesystem] Load and parse any .thorp.cache files found

* [filesystem] Use cached file data when available and up-to-date

* [lib] FileScanner refactoring

* [filesystem] scan sub-dirs first to minimise time cache is on heap

* [filesystem] Write new cache data to temp file

* [lib] replace cache file when finished updating

* [filesystem] AppendLines to correct file with new lines

* [domain] decode HashType from String

* [filesystem] Store last modified time as epoch milliseconds

* [filesystem] parse lastmodified as a long

* [filesystem] use all hash values in cache

* [lib] FileScanner rearrange code

* [lib] Create and use a single cache file per source

* [storage-aws] Use ETag hash from cache when available

* [filesystem] Merge file data together correctly

* [filesystem] Handle exceptions thrown by Files.mode correctly

* [readme] Add section on caching

* [changelog] updated

* [changelog] add pending dependencies notes

* [lib] Filters should not name methods after their defining object

* [lib] Fix up test
2019-10-27 19:53:00 +00:00
5214bacc0b
Not wrapping exceptions thrown in waitForUploadResult (#162)
* [storage-aws] Uploader move implementation to companion

* [app] Program Refactoring

* [storage-aws] AmazonTransferManager refactoring

* [lib] UnversionedMirrorArchive refactoring

* [console] Add Console.putStr

* [uishell] UIShell show chosen actions

* [storage-aws] AmazonTransferManager try to handle errors

* [uishell] UIShell avoid line wrap with long file paths

* [storage] Log when fetching remote summaries

* Handle exceptions thrown in waitForUploadResult

* [uishell] log errors

* [console] Swap batch/non-batch error messages

* fix tests
2019-09-23 13:30:34 +01:00
a2c061d655
Upload progress via uievent (#196)
* [uishell] Fix package name

* [sbt] Update eip-zio from 0.3.1 to 0.3.2

* [sbt] add dependency on eip-zio to domain

* [uishell] move Upload*Event* to uishell

* UploadEventListener
* UploadEventLogger
* UploadProgressEvent

* [uishell] Log upload progress using UIShell

* [uishell] inline UploadEventLogger into UIShell

* [uishell] Remove println and use Console
2019-09-11 22:47:42 +01:00
3d4c238030
Add more tests (#192)
* [storage-aws] Refactoring

* [lib] Add test for FileScanner

* [lib] Add LocalFileSystemTest for scanCopyUpload

* [lib] Add LocalFileSystem tests for scanDelete

Also send a UIEvent for ActionChosen(ToDelete)

* [lib] LocalFileSystemTest can handle files in any order

* [lib] LocalFileSystemTest can handle files in any order #2

Don't include accountCounter or byteCounter are the order can change.

* [domain] Remove LocalFile.relativeToSource

* [sbt] Add missing modules to aggregate
2019-09-08 18:59:43 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
321773f04c
Upgrade to Scala 2.13.0 (#176) 2019-09-01 21:30:16 +01:00
242804294d [storage-aws] force upload progress updates to strick sequence
Prevents the situation where a second progress update arrives while
the first is still on the process of being processed. The new update
is blocked until the first is completed.
2019-08-14 10:10:23 +01:00
e0beef9d73 [storage-aws] don't create new event listener for every event 2019-08-14 09:23:34 +01:00
af7733952c
Install WartRemover pluging (#150)
* [sbt] Install WartRemover

* remove warts

* remote warts

* fix tests

* [domain] UploadEventListener fix progress bar (again)

* [domain] Remove LastModified - wasn't being used for anything
2019-08-06 18:19:05 +01:00
07ca6b962f
Refactor ActionGenerator.genAction(S3MetaData,Stream,Bucket) (#143)
* [domain] rename S3MetaData as MatchedMetadata

* [domain] rename S3ObjectsData as RemoteObjects

* [core] ActionGenerator refactoring

* [core] ActionGenerator.createAction renamed and no longer a stream

* [core] ActionGenerator refactor

* [core] ActionGenerator Usage of head on collections
2019-08-05 12:07:06 +01:00
Scala Steward
9752d96ab4 Update zio to 1.0.0-RC11 (#148)
* Update zio to 1.0.0-RC11

* Update to be compatible with ZIO-1.0.0-RC11
2019-08-05 09:04:46 +01:00
8fad680a96
case classes shouldn't be OO objects (#147)
* [core] Extract Filters from domain.Filter

* [core] extract LocalFileValidator

* [domain] LocalFile remove unused isDirectory

* [domain] LocalFile move/rename relative as relativeToSource on companion

* [domain] LocalFile move and rename matches as matchesHash on companion

* [domain] LocalFile move md5base64 to companion

* [domain] Logger remove

* [domain] MD5Hash move hash to companion

* [domain] MD5Hash move digest to companion

* [domain] MD5Hash move hash64 to companion

* [domain] RemoteKey move class methods to companion

* [domain] Sources move forPath to companion

Led to being able to cleanup LocalFileStream.localFile and adding
LocalFiles.one

* [domain] UploadEventLogger rename method as apply
2019-08-05 08:38:44 +01:00
ec221d9f05
UploadEventListener - reset bytesTransfered for each file (#145) 2019-08-04 20:12:16 +01:00
adbf8af2f3
Refactor UploadEventLogger.statusWithBar(String,Long=>String,Long,Long,Long) (#128)
* [domain] UploadEventListener refactoring

* [domain] UploadEventListener rewrite
2019-08-01 13:31:48 +01:00
4a54038933
Refactor Uploader.upload(LocalFile,Bucket,Boolean,UploadEventListener,Int) (#129)
* [storate-api] Remove unused tryCount

* [storage-aws] Uploader Reduce the number of parameters
2019-08-01 08:59:29 +01:00
899e0724c9
Refactor Copier.copyObject(AmazonS3.Client,Bucket,RemoteKey,MD5Hash,RemoteKey) (#137)
* [core] Convert HashService into an effect Hasher

* [core] LocalFileStreamSuite fix prep of test env

* [core] LocalFileStreamSuite Refactor

* [core] Add Hasher.hashObjectChunk

* [core] make MD5HashGenerator private

Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.

Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.

* [domain] Add Monoid

* [domain] Sources use Monoid

* [core] ActionGenerator Refactor

* [core] Use >>= and *> operators

* [config] Refactoring

* [storage-aws] Refactoring

* [domain] Refactoring

* [core] Refactoring

* [config] refactoring

* Refactoring

* [core] PlanBuilder Refactoring

* [core] Make PlanBuilder into an object

* [storate-aws] Copier Reduce the number of parameters
2019-08-01 08:34:58 +01:00
e5de73e705
Merge config package object with Config object (#138)
* [config] merge config package object with Config object

* [config] Only extract batchMode value where used

Don't pull it out early then pass a boolean around
2019-07-30 10:33:06 +01:00
a93781007d
Create FileSystem as full ZIO effect module (#135)
* [changelog] updated

* [sbt] Add filesystem module

* [filesystem] stub effect module

* [filesystem] Add fileExists

* [filesystem] Add openFile

* [filesystem] Add fileLines

* [config] Use FileSystem.fileExists

* [filesystem] FileSystem.Test implement test methods

* [filesystem] Replace package object with FS object

* [storage] merge storage package object into Storage object

* [console] merge console package with Console object
2019-07-30 08:07:26 +01:00