Commit graph

94 commits

Author SHA1 Message Date
Scala Steward
dfc36490cb
Update jackson-databind to 2.10.4 (#388)
Co-authored-by: Paul Campbell <pcampbell@kemitix.net>
2020-06-08 06:55:03 +01:00
a127907073
Update aws-java-sdk-s3 to 1.11.797 (#425)
Co-authored-by: Scala Steward <me@scala-steward.org>
2020-06-08 02:17:42 +01:00
Scala Steward
efc2e10e4b Update aws-java-sdk-s3 to 1.11.661 (#250) 2019-10-29 15:06:23 +00:00
Scala Steward
56a05a60ba Update zio, zio-streams to 1.0.0-RC16 (#251) 2019-10-29 15:04:46 +00:00
Scala Steward
4934f9ff81 Update zio, zio-streams to 1.0.0-RC15 (#241)
* Update zio, zio-streams to 1.0.0-RC15

* [app] Update to use ZEnv
2019-10-27 20:27:02 +00:00
Scala Steward
1cbf3b4aaf Update zio, zio-streams to 1.0.0-RC14 (#226) 2019-10-08 13:31:55 +01:00
Scala Steward
7b567ab28c Update aws-java-sdk-s3 to 1.11.641 (#217) 2019-09-28 17:21:25 +01:00
Scala Steward
ce0a826e6d Update jackson-dataformat-cbor to 2.10.0 (#216) 2019-09-27 16:08:37 +01:00
Scala Steward
f51501b13d Update jackson-databind to 2.10.0 (#214) 2019-09-27 15:47:44 +01:00
Scala Steward
6adffd8da7 Update jackson-dataformat-cbor to 2.9.10 (#208)
* Update jackson-dataformat-cbor to 2.9.10

* Update jackson-dataformat-cbor to 2.9.10
2019-09-23 12:02:40 +01:00
Scala Steward
75d23b25f0 Update jackson-databind to 2.9.10 (#206) 2019-09-22 23:13:28 +01:00
Scala Steward
cfc284185b Update zio, zio-streams to 1.0.0-RC13 (#209) 2019-09-22 23:08:11 +01:00
Scala Steward
ef0d867b50 Update aws-java-sdk-s3 to 1.11.635 (#203) 2019-09-20 11:06:18 +01:00
a2c061d655
Upload progress via uievent (#196)
* [uishell] Fix package name

* [sbt] Update eip-zio from 0.3.1 to 0.3.2

* [sbt] add dependency on eip-zio to domain

* [uishell] move Upload*Event* to uishell

* UploadEventListener
* UploadEventLogger
* UploadProgressEvent

* [uishell] Log upload progress using UIShell

* [uishell] inline UploadEventLogger into UIShell

* [uishell] Remove println and use Console
2019-09-11 22:47:42 +01:00
3d4c238030
Add more tests (#192)
* [storage-aws] Refactoring

* [lib] Add test for FileScanner

* [lib] Add LocalFileSystemTest for scanCopyUpload

* [lib] Add LocalFileSystem tests for scanDelete

Also send a UIEvent for ActionChosen(ToDelete)

* [lib] LocalFileSystemTest can handle files in any order

* [lib] LocalFileSystemTest can handle files in any order #2

Don't include accountCounter or byteCounter are the order can change.

* [domain] Remove LocalFile.relativeToSource

* [sbt] Add missing modules to aggregate
2019-09-08 18:59:43 +01:00
becd297858
Remove dead code (#190)
* [lib] Remove PlanBuilder and PlanExecutor

* [lib] Remove ActionGenerator and LocalFileStream

* [lib] Remove S3MetaDataEnricher

* [lib] Remove Remote

* [filesystem] fix paths to test resources

* [lib] Remove LocalFileValidator

* [lib] Remove SyncPlan

* [lib] Remove SequencePlan

* [lib] Remove KeyGenerator

* [lib] Remove DummyStorageService

* [lib] Remove EventQueue

* [lib] Remove SyncLogging

* [lib] Remove LocalFiles

* [lib] inline CoreTypes into Program

* [lib] Remote EIPTest

* [lib] LocalFileSystem remove unneccary parens

* [domain] Remove Monoid

* [domain] Remove MatchedMetedata

* [domain] Remove NonUnit

* [domain] Remove RemoteMetaData

* [domain] Rename StorageQueueEvent as StorageEvent

* [domain] Remove SyncTotals

* [domain] Rename UploadEvent as UploadProgressEvent

* [sbt] fix assembly merge strategy to work with zio and zio-streams
2019-09-08 07:29:23 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
Scala Steward
5d7e106e83 Update zio to 1.0.0-RC12-1 (#178) 2019-09-03 18:27:59 +01:00
321773f04c
Upgrade to Scala 2.13.0 (#176) 2019-09-01 21:30:16 +01:00
Scala Steward
2be82eafa2 Update scalamock to 4.4.0 (#165) 2019-08-17 18:27:22 +01:00
Scala Steward
c646ae82fa Update aws-java-sdk-s3 to 1.11.610 (#158) 2019-08-14 07:00:52 +01:00
af7733952c
Install WartRemover pluging (#150)
* [sbt] Install WartRemover

* remove warts

* remote warts

* fix tests

* [domain] UploadEventListener fix progress bar (again)

* [domain] Remove LastModified - wasn't being used for anything
2019-08-06 18:19:05 +01:00
Scala Steward
9a6208025c Update jackson-databind to 2.9.9.3 (#152) 2019-08-06 17:34:55 +01:00
Scala Steward
e41e29127f Update zio to 1.0.0-RC11-1 (#149) 2019-08-05 11:52:13 +01:00
Scala Steward
9752d96ab4 Update zio to 1.0.0-RC11 (#148)
* Update zio to 1.0.0-RC11

* Update to be compatible with ZIO-1.0.0-RC11
2019-08-05 09:04:46 +01:00
8fad680a96
case classes shouldn't be OO objects (#147)
* [core] Extract Filters from domain.Filter

* [core] extract LocalFileValidator

* [domain] LocalFile remove unused isDirectory

* [domain] LocalFile move/rename relative as relativeToSource on companion

* [domain] LocalFile move and rename matches as matchesHash on companion

* [domain] LocalFile move md5base64 to companion

* [domain] Logger remove

* [domain] MD5Hash move hash to companion

* [domain] MD5Hash move digest to companion

* [domain] MD5Hash move hash64 to companion

* [domain] RemoteKey move class methods to companion

* [domain] Sources move forPath to companion

Led to being able to cleanup LocalFileStream.localFile and adding
LocalFiles.one

* [domain] UploadEventLogger rename method as apply
2019-08-05 08:38:44 +01:00
96ecedbe61
[sbt] Include JABX for JDK9+ (#146) 2019-08-04 20:13:04 +01:00
Scala Steward
f40fe46d84 Update aws-java-sdk-s3 to 1.11.603 (#144) 2019-08-04 20:07:42 +01:00
e3b0260b6d
Refactor ThorpArchive.logEvent(StorageQueueEvent) (#141)
* [core] ThorpArchive Refactoring

* [core] ThorpArchive Refactoring

* [core] ThorpArchive refactored

* [core] ThorpArchive convert output to ConsoleOut.WithBatchMode messages

* [console] Console rename putMessageLnB to overload putMessageLn
2019-08-01 22:39:52 +01:00
899e0724c9
Refactor Copier.copyObject(AmazonS3.Client,Bucket,RemoteKey,MD5Hash,RemoteKey) (#137)
* [core] Convert HashService into an effect Hasher

* [core] LocalFileStreamSuite fix prep of test env

* [core] LocalFileStreamSuite Refactor

* [core] Add Hasher.hashObjectChunk

* [core] make MD5HashGenerator private

Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.

Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.

* [domain] Add Monoid

* [domain] Sources use Monoid

* [core] ActionGenerator Refactor

* [core] Use >>= and *> operators

* [config] Refactoring

* [storage-aws] Refactoring

* [domain] Refactoring

* [core] Refactoring

* [config] refactoring

* Refactoring

* [core] PlanBuilder Refactoring

* [core] Make PlanBuilder into an object

* [storate-aws] Copier Reduce the number of parameters
2019-08-01 08:34:58 +01:00
e5de73e705
Merge config package object with Config object (#138)
* [config] merge config package object with Config object

* [config] Only extract batchMode value where used

Don't pull it out early then pass a boolean around
2019-07-30 10:33:06 +01:00
a93781007d
Create FileSystem as full ZIO effect module (#135)
* [changelog] updated

* [sbt] Add filesystem module

* [filesystem] stub effect module

* [filesystem] Add fileExists

* [filesystem] Add openFile

* [filesystem] Add fileLines

* [config] Use FileSystem.fileExists

* [filesystem] FileSystem.Test implement test methods

* [filesystem] Replace package object with FS object

* [storage] merge storage package object into Storage object

* [console] merge console package with Console object
2019-07-30 08:07:26 +01:00
Scala Steward
f6ce262f2b Update jackson-databind to 2.9.9.2 (#132) 2019-07-28 21:48:24 +01:00
ccefd286f9
Convert Config to full ZIO effect module (#134)
* [config] new module

* [config] stub module

* [domain] Rename domain.Config as domain.LegacyConfig

* [config] Move LegacyConfig to config module

* [config] Move config parsing and validation into module

* [config] Complete migration to module for Config

* [config] Config You should not name methods after their defining object

* [config] Rename LegacyConfig as Configuration

Also remove redundant uses

* [core] LocalFileStream Refactoring

* [changelog] update
2019-07-28 21:47:01 +01:00
cad152379e
Remove Monocle dependency (#121)
* Replace Monocle with local SimpleLens implementation

* [domain] SimpleLensTest avoid short variables and remove field imports
2019-07-24 09:40:56 +01:00
Scala Steward
763c813a1f Update aws-java-sdk-s3 to 1.11.596 (#118) 2019-07-23 23:14:23 +01:00
f277b5e789
Improve logging consistency (#119)
* [console] create new effect MyConsole

* [core] SyncLogging use ConsoleOut.ValidConfig

* [storage-aws] Extract AmazonS3 trait to help testing

* [console] MyConsole use UIO

* [storage-aws] S3StorageServiceSuite convert to FreeSpec

* [storage-aws] S3StorageServiceSuite extract S3ClientTest trait

* [storage-aws] remove incomplete test UploaderSuite

* [storage-aws] extract and rename AmazonS3ClientTestFixture

* [storage-aws] Copier handle and log errors

* [core] ThropArchive log completed uploads here

* [core] ThorpArchive log copies and deletes

* Improve consistency of logging

* [storage-aws] CopierTest extract from S3StorageServiceSuite

* [storage-aws] Copier handle hash match errors properly

* [core] ThropArchive display erros in red

* [core] SequencePlan extracted

* [core] Clear line after deletion log

* [changelog] updated

* [cli] Program tidy up imports

* [storage-aws] Copier replace null with Option

* [storage-aws] AmazonS3 copyObject returns Some[CopyObjectResult]
2019-07-23 23:13:09 +01:00
8cca46340c
Replace cats-effect with zio (#117)
* [sbt] Add ZIO dependency to storage-api

* Convert to use ZIO

* [sbt] remove cats-effect dependency

* [changelog] updated

* [cli] Program restore actions to correct order (copy, upload, delete)

* [cli] Program You should not name methods after their defining object

* [core] ConfigValidationException Redundant braces after class definition

* [core] LocalFileStreamSuite Usage of get on optional type.

* [core] PlanBuilderTest Usage of get on optional type.
2019-07-21 21:02:04 +01:00
32ef58ff11
NullPointerException creating scaladoc (#115)
* [domain] Drop use of Lenses macro and only create used lenses

* [core] Drop use of Lenses macro and only create used lenses

* [sbt] drop paradise plugin

* [changelog] updated
2019-07-19 23:30:20 +01:00
515b896993
Use Lenses (#113)
* [sbt] Add monocle for support for lenses

* [domain] UploadEventListener make a case class

* [domain] Add @Lenses to all case classes

* [core] Add @Lenses to case classes

* [core] ActionGenerator use lense

* [core] ConfigOption use Lenses

* [core] ConfigurationBuilder remove unused fields

* [core] ConfigurationBuilder refactoring

* [core] SyncLogging use lenses

* [core] syncLogging refactoring
2019-07-18 08:57:14 +01:00
afc55354e7
Apply scalafmt (#108)
* [scalafmt] Add .scalafmt

* [sbt] Add unused import warning and make them errors

* [core] ConfigurationBuilder remove unused import

* [core] ConfigurationBuilder apply scalafmt

* [domain] reformat

* [storage-api] reformat

* [core] reformat and some refactoring

* [storage-aws] reformat

* [cli] reformat

* [core] post rebase fix up

* [storage-aws] UploaderSuite tidy up

* [domain] MD5Hash tweak

Without doing this we have a file that we don't create a valid md5
hash for. See #103

* [storage-aws] UploaderSuite remove unused import

* [storage-aws] StorageServiceSuite reformatted

* [sbt] consistent settings for all modules

Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.

* [storage-aws] Add AmazonTransferManager

This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.

* [storage-aws] Add AmazonUpload

Prevents deprecation errors due to deprecated parameters on Transfer.

* [sbt] mode import to top of file
2019-07-16 07:56:54 +01:00
Scala Steward
2b284a9649 Update aws-java-sdk-s3 to 1.11.587 (#104) 2019-07-06 13:46:14 +01:00
26b4f6f794 Merge branch 'v0.6.x'
* v0.6.x:
  Release to sonatype using correct profile name (#102)
2019-07-03 18:42:40 +01:00
Scala Steward
6cda33a94a Update jackson-databind to 2.9.9.1 (#101) 2019-07-03 14:18:32 +01:00
8585023c3e
Release to sonatype using correct profile name (#102)
* [changelog] updated

* [sbt] Specify sonatypeProfileName in commonSettings

This prevents the value being overridded by the `organization` value
from `inThisBuild`.
2019-07-03 13:50:28 +01:00
5df752047f
Add a version command-line option (#99)
* [sbt] make name and version available within Scala

Available as `thorp.BuildInfo._`

* [core] ConfigOption Add Version option

* [cli] ParseArgs add command-line options `-V` and `--version`

* [core] ConfigQuery add showVersion query

* [cli] Program shows version then exit

* [cli] Main don't show "Done"
2019-06-30 22:01:02 +01:00
8120fca0ae [sbt] Only specify organization once 2019-06-30 19:00:10 +01:00
88dd2f37a4 [sbt] specify sonatype profile name 2019-06-30 17:42:16 +01:00
619424a416
Embed launch script in fat-jar (#98)
* [sbt] enable executable jar for shell

* [sbt] rename assembled jar to not include file suffix
2019-06-30 15:04:23 +01:00
124c269322
[sbt,travis] revert most of "publish fat-jar", keeping cli jar name (#97) 2019-06-30 14:38:50 +01:00