Commit graph

43 commits

Author SHA1 Message Date
dependabot-preview[bot]
446682745b
Bump picocli from 4.3.2 to 4.4.0 (#497) 2020-07-06 08:51:56 +00:00
b960f2c8ab
Release 2.0.0 (#491)
* Version set to 2.0.0

* Remove lingering scala plugins
2020-06-27 20:09:45 +01:00
0092d163f2
cli: module convert to Java (#473) 2020-06-22 06:36:44 +01:00
0104d9c08d
Post-release version set to 1.1.0-SNAPSHOT (#472) 2020-06-21 19:42:29 +01:00
5cb929e7e4
Release 1.0.0 (#463)
Release 1.0.0
2020-06-21 07:54:43 +01:00
319c46f403
Convert to Java (domain, config, storage-aws and filesystem) (#446)
* Java rewrite domain (#438)

* domain.Bucket: convert to Java

* domain.LastModified: convert to Java

* domain.QuoteStripper: convert to Java

* domain.HexEncoder: convert to Java

* domain.MD5Hash: convert to Java

* remove unused import

* domain.RemoteKey: convert to Java

* domain.Action: convert to Java

* domain.Counters: convert to Java

* domain.HashType: convert to Java

* domain.Hashes: convert to Java

* domain.MD5HashData: convert to Java

* domain.Filter: convert to Java

* domain.LocalFile: convert to Java

* domain: make immutable field public

* domain.SizeTranslation: convert to Java

* domain.HashType: restrict access to contstructor

* domain.RemoteObjects: convert to Java

Introduce MapView and Tuple.

* domain.Sources: convert to Java

* domain.StorageEvent: convert to Java

* domain.Terminal: convert to Java

* domain => config: move SimpleLens to only module that uses it

* domain => filesystem: move TemporaryFolder

* domain.Implicits: removed

* parent: make junit, et al available

* domain: add testing dependencies

* domain.HexEncoder: convert test to Java and fix bugs

* domain.HexEncoderTest: replace with Java version

* domain.MD5HashTest: convert to Java

* domain.RemoteKeyTest: convert to Java

* domain.SizeTranslationTest: convert to Java

* domain.TerminalTest: convert to Java

* domain: remove unused dependencies

* parent: rollback zio-streams to match zio and pin them together

* storage-aws: resolve transitive dependency conflicts

* Java rewrite storage aws (#445)

* storage-aws.AmazonS3: convert to Java as AmazonS3Client

* storage-aws.S3Copier: convert to Java

* storage-aws.S3Uploader: convert to Java

* storage-aws.S3Deleter: convert to Java

* storage-aws.S3Lister: convert to Java

* filesystem: write cache data correctly (as supplied)

* domain,filesystem: fix MD5Hash generation

* filesystem: convert to Java (#450)

* remove legacy

* Rewrite config module in Java (#461)

* config.ParseConfigFile: convert to Java

* config.ParseConfigFile: convert to Java

* config.SourceConfigLoader: convert to Java

* WIP config.Configuration: convert to Java

* config.ConfigOption: convert to Java

* config.ConfigOptions: convert to Java

* config.ConfigValidation: convert to Java

* config.ConfigQuery: convert to Java

* config: move classes to correct location

* config.ConfigValidationException: convert to Java

* config.ConfigValidator: convert to Java

* config.ConfigurationBuilder: convert to Java

* config.SimpleLens: removed

* config.Config: remove environment

* config.ConfigOptionTest: convert to Java

* config.ConfigQueryTest: convert to Java

* config.ConfigurationBuilderTest: convert to Java

* config.ParseConfigFileTest: convert to Java

* config.ParseConfigLinesTest: convert to Java

* config: remove scala dependencies and plugin
2020-06-21 07:21:21 +01:00
baea18f3f3
Java rewrite - step 1 - build with Maven (#431)
* Add root and parent pom.xml files

* parent: add scala-maven-plugin to pluginManagement

* fix packaging

* domain: add module

* parent,domain: add dependencies

* maven: add filesystem module

* parent: disable coverage and pitest

* domain: move classes from test to main as they are used in filesystem

* config: add module

* console: add module

* Add reactor-graph

* fix typo

* uishell: add module

* Restore explicit dependencies

Don’t depend upon transitive dependencies if we want to be able to restructure module dependencies.

* storage: add module

* lib: add module

Included adding a Resource case class in filesystem to replace one generated by a scala plugin

* storage-aws: add module

* cli: add module

* app: add module

* app: make jar runnable

* Updated build instructions in README

* Convert readme to markdown

* Add reactor graph to readme

* Switch Guthub Actions to using Maven to build
2020-06-11 21:33:28 +01:00
06dd4f8fed
Support multiple parallel uploads (#188)
* [filesystem] listFiles no longer returns errors

* [cli,config] Take parallel parameter

* [config] Config add .parallel

* [lib] Perform copy and upload in parallel

* [uishell] Extract UIRequestCycle

* [uishell] Display all pending uploads progress

* [app] Always display version

* [app] Highlight version

* [uishell] UIRequestCycle refactoring

* [uishell] UIShell Don’t hide chosen actions in batch mode

* [uishell] UIShell fix typo

* [console] ConsoleOut fix typo
2019-09-27 16:08:16 +01:00
c5d7d4933c
Restructure using EIP-ZIO channels (#183)
* [sbt] Rename storage-api as storage

* [storage] remove dependency upon console

* [storage] remove dependency upon config

* [console] remove dependency upon config

* [sbt] Add app module

Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.

* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib

* [sbt] make module dependencies more explicit

* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests

* [sbt] merge thorp-lib with core as lib

* [sbt] add zio streams

* [lib] Add EIPTest

* [sbt] Allow NonUnitStatements

* [lib] EIPTest Message Channel rewritten using ZIO Stream

* [sbt] Add eip-zip 0.2.0 as dependency in lib

* Remove file counter and total upload size progress

Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.

* Fetch Remote Data before preparing any plans

* [domain] RemoteObjects only holds a single RemoteKey per Hash

Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.

* [lib] Add a State trait

* [lib] Add FileScanner

* Add FileSystem.length(File)

* Add Clock to the Environment

* [domain] Sources update format

* [domain] Asking for a path that isn't in any Sources is fatal

There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.

* [lib] Add test on use of zio.Ref

* [uishell] Add stub module

* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0

* [uishell] Add UIEvent stub

* [uishell] Add UIShell stub

* [sbt] Add eip-zio dependencies to app module

* [app] Wrap existing execution in simple point to point channel

* [uishell] Add UIEvent.ShowValidConfig

* [app] Remember to end the channel to allow prog to exit

* [app] purify environment for showValidConfig

* [app] Create type alias for pure effect free channel ref

* [app] Program refactoring

* [uishell] Add UIEvent.RemoteDataFetched

* [domain] Move Counters from lib

* [uishell] Add UIEvent.ShowSummary

* [lib] Add stub for PushLocalChanges

* [lib] Clean up FileScanner Environment types

* [lib] End channel after scanning files

* [lib] PushLocalChanges uses FileScanner

Scans files and sends them to a dummy receiver.

* [uishell] Add UIEvent.FileFound

* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload

* [lib] FileScanner return LocalFile

* [domain] add length to LocalFile

* [domain] Add interogation queries to RemoteObjects

* [domain] Remove RemoteObject.keyForHashes

* [domain] RemoteObjects.remoteHasHash return the key and the hash

* [lib] LocalFileSystem.scanCopyUpload create Actions

* [domain] Move Action from lib

* [uishell] Log actions

* [lib] FileScanner respects Filters

* [lib] Create remoteKey for files correctly

* [lib] LocalFileSystem refactoring

* [lib] ThorpArchive.update doesn't need Console

* [uishell] Don't log choosen Action

* [uishell] Add UIEvent.ActionFinished

* [lib] LocalFileSystem refactoring

* [lib] Switch to using LocalFileSystem to do Copy/Upload

Todo or Broken:

- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros

* [lib] LocalFileStream display summary counters correctly

* [app] Restore ability to delete remote files

* [lib] LocalFileSystem deletes remote when local does NOT exist

* [filesystem] move hasLocalFile to FileSystem

* [filesystem] fix detection of local files from a RemoteKey

The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.

* [filesystem] fix broken FileSystem test

* [domain] fix RemoteKey test

* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility

* [app] Program refactorting

* [lib] Remove unused class

* [lib] Remove test

* [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
656f3fec30 [core] Extract PlanExecutor 2019-08-14 22:51:06 +01:00
b643b7ea92
Refactor Program.updateArchive(ThorpArchive,Action,Int,Stream,Long) (#142)
* [cli] Program Refactor

* [cli] Program Refactor
2019-08-02 20:49:14 +01:00
899e0724c9
Refactor Copier.copyObject(AmazonS3.Client,Bucket,RemoteKey,MD5Hash,RemoteKey) (#137)
* [core] Convert HashService into an effect Hasher

* [core] LocalFileStreamSuite fix prep of test env

* [core] LocalFileStreamSuite Refactor

* [core] Add Hasher.hashObjectChunk

* [core] make MD5HashGenerator private

Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.

Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.

* [domain] Add Monoid

* [domain] Sources use Monoid

* [core] ActionGenerator Refactor

* [core] Use >>= and *> operators

* [config] Refactoring

* [storage-aws] Refactoring

* [domain] Refactoring

* [core] Refactoring

* [config] refactoring

* Refactoring

* [core] PlanBuilder Refactoring

* [core] Make PlanBuilder into an object

* [storate-aws] Copier Reduce the number of parameters
2019-08-01 08:34:58 +01:00
e5de73e705
Merge config package object with Config object (#138)
* [config] merge config package object with Config object

* [config] Only extract batchMode value where used

Don't pull it out early then pass a boolean around
2019-07-30 10:33:06 +01:00
a93781007d
Create FileSystem as full ZIO effect module (#135)
* [changelog] updated

* [sbt] Add filesystem module

* [filesystem] stub effect module

* [filesystem] Add fileExists

* [filesystem] Add openFile

* [filesystem] Add fileLines

* [config] Use FileSystem.fileExists

* [filesystem] FileSystem.Test implement test methods

* [filesystem] Replace package object with FS object

* [storage] merge storage package object into Storage object

* [console] merge console package with Console object
2019-07-30 08:07:26 +01:00
ccefd286f9
Convert Config to full ZIO effect module (#134)
* [config] new module

* [config] stub module

* [domain] Rename domain.Config as domain.LegacyConfig

* [config] Move LegacyConfig to config module

* [config] Move config parsing and validation into module

* [config] Complete migration to module for Config

* [config] Config You should not name methods after their defining object

* [config] Rename LegacyConfig as Configuration

Also remove redundant uses

* [core] LocalFileStream Refactoring

* [changelog] update
2019-07-28 21:47:01 +01:00
96a83e6c3e
Convert Storage to full ZIO effect module (#133)
* [console] Rename MyConsole as Console

* [console] break infinite loop

* [console] fix typo

* [console] clean up helpers

* [cli] Main use ZIO#provide to run program

* [cli] Main define Program type alias

* [cli] Program handle cli args in Program

* [cli] Program doesn't extend PlanBuilder

* [cli] refactoring

* [cli] rename ParseArgs as CliArgs

* [cli] CliArgs#apply renamed a parse

* [storage-aws] S3StorageService renamed as S3Storage

* [storage-api] Rename StorageService as Storage.Service

* [storage-api] make Storage.copy effectTotal

* [storage-api] make Storage.delete effectTotal

* [storage-api] make Storage.shutdown effectTotal

* [storage-api] make Storage.upload effectTotal

* [storage-aws] Lister refactoring

* [storage-aws] make Lister into a trait

* [storage-aws] make Copier into a trait

* [storage-aws] make Deleter into a trait

* [storate-aws] make Uploader into a trait

* [storage-aws] AmazonS3 move error handling out of client wrapper

* [storage-aws] DeleterTest added

* [storage-aws] ListerTest added

* [storage-aws] Uploader refactoring

* [storage-aws] CopierTest test Copier directly

* [storage-aws] DeleterTest test Deleter directly

* [storate-aws] ListerTest test Lister directly

* [storage-aws] UploaderTest added

* [storage-aws] S3Storage.Live replaces S3StorageServiceBuilder

* Complete migration to Module for Storage

* [cli] Main define LiveThorpApp object

* [core] Add CoreTypes

* [cli] Program Refactoring

* [core] PlanBuilding Refactoring

* [changelog] updated

* [console] Console.Live Usage of get on optional type

* [storage-aws] AmazonS3ClientTestFixture Use wildcards when selecting more than 6 elements
2019-07-28 20:11:03 +01:00
f277b5e789
Improve logging consistency (#119)
* [console] create new effect MyConsole

* [core] SyncLogging use ConsoleOut.ValidConfig

* [storage-aws] Extract AmazonS3 trait to help testing

* [console] MyConsole use UIO

* [storage-aws] S3StorageServiceSuite convert to FreeSpec

* [storage-aws] S3StorageServiceSuite extract S3ClientTest trait

* [storage-aws] remove incomplete test UploaderSuite

* [storage-aws] extract and rename AmazonS3ClientTestFixture

* [storage-aws] Copier handle and log errors

* [core] ThropArchive log completed uploads here

* [core] ThorpArchive log copies and deletes

* Improve consistency of logging

* [storage-aws] CopierTest extract from S3StorageServiceSuite

* [storage-aws] Copier handle hash match errors properly

* [core] ThropArchive display erros in red

* [core] SequencePlan extracted

* [core] Clear line after deletion log

* [changelog] updated

* [cli] Program tidy up imports

* [storage-aws] Copier replace null with Option

* [storage-aws] AmazonS3 copyObject returns Some[CopyObjectResult]
2019-07-23 23:13:09 +01:00
8cca46340c
Replace cats-effect with zio (#117)
* [sbt] Add ZIO dependency to storage-api

* Convert to use ZIO

* [sbt] remove cats-effect dependency

* [changelog] updated

* [cli] Program restore actions to correct order (copy, upload, delete)

* [cli] Program You should not name methods after their defining object

* [core] ConfigValidationException Redundant braces after class definition

* [core] LocalFileStreamSuite Usage of get on optional type.

* [core] PlanBuilderTest Usage of get on optional type.
2019-07-21 21:02:04 +01:00
515b896993
Use Lenses (#113)
* [sbt] Add monocle for support for lenses

* [domain] UploadEventListener make a case class

* [domain] Add @Lenses to all case classes

* [core] Add @Lenses to case classes

* [core] ActionGenerator use lense

* [core] ConfigOption use Lenses

* [core] ConfigurationBuilder remove unused fields

* [core] ConfigurationBuilder refactoring

* [core] SyncLogging use lenses

* [core] syncLogging refactoring
2019-07-18 08:57:14 +01:00
afc55354e7
Apply scalafmt (#108)
* [scalafmt] Add .scalafmt

* [sbt] Add unused import warning and make them errors

* [core] ConfigurationBuilder remove unused import

* [core] ConfigurationBuilder apply scalafmt

* [domain] reformat

* [storage-api] reformat

* [core] reformat and some refactoring

* [storage-aws] reformat

* [cli] reformat

* [core] post rebase fix up

* [storage-aws] UploaderSuite tidy up

* [domain] MD5Hash tweak

Without doing this we have a file that we don't create a valid md5
hash for. See #103

* [storage-aws] UploaderSuite remove unused import

* [storage-aws] StorageServiceSuite reformatted

* [sbt] consistent settings for all modules

Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.

* [storage-aws] Add AmazonTransferManager

This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.

* [storage-aws] Add AmazonUpload

Prevents deprecation errors due to deprecated parameters on Transfer.

* [sbt] mode import to top of file
2019-07-16 07:56:54 +01:00
b15350d959
Sync more than one source directory into a single bucket/prefix (#25)
* [changelog] updated

* [readme] updated

* [core] ConfigQuery added sources()

* [cli] ParseArgs allow specifying multiple sources

* [domain,core,cli] Source datatype changed to Path

* [domain] Sources added to hold multiple paths in order

* [domain] Config sources change datatype to Sources

* [core] Scan sources for .thorp.config and include any sources listed

This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.

* [core] ConfigurationBuilderTest add more pending tests

* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures

* [core] ConfigOptionTest use TemporaryFolder

* [core] ConfigOptionTest remove unused fields

* [cli] ParseArgsTest don't use get on an Option

* [core] ConfigurationBuilderTest don't use get on Either

* [core] TemporaryFolder Move import to top of file

* [core] TemporaryFolder use Try over try-finally

* [core] ConfigurationBuilderTest don't use get on Either

* [core] TemporaryFolders.withDirectory propogate errors

* [core] TemporaryFolders add writeFile and createFile

* [core] PlanBuilderTest create a plan with two sources with unique files in both

* [core] ActionGenerator only upload file by name in first source

create a plan
  two sources
    same filename in both
    - only upload file in first source

* [domain] LastModified with no params is now()

* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing

* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing

* [domain] RemoteKey map to a file when prefix is empty

* [domain] S3ObjectData defaults to empty

* [core] KeyGenerator Avoid delimiter when empty prefix key

* [core] PlanBuilderTest when remote not in sources delete from remote

* [core] PlanBuilderTest extract helper md5Hash()

* [core] PlanBuilderTest one source a file no matching remote key

* [core] PlanBuildingTest file with matching key and hash do nothing

* [core] PlanBuilderTest file w/matching remote key and different hash

* [core] PlanBuilderTest a remote key with and without local file

* [core] DummyStorageService Use wildcards when selecting more than 6 elements
2019-07-12 07:42:42 +01:00
00c04187e8
Display total size and progress for entire run (#94)
* [changelog] updated

* [core] Wrap Stream[LocalFile] as LocalFiles

* [core] LocalFiles counts files

* [core] LocalFiles sums file lengths

* [core] Restore logFileScan

* [storage-aws] Lister logs when fetching object summaries

* [storage-aws] Extract ListerLogger

* [core] Synchronise use leftMap

* [core] Syncronise extract assemblePlan

* [core] Wrap Stream[Action] in SyncPlan

* [core] Copy the file count and totalSizeBytes across to SyncPlan

* [cli] Program rename actions as syncPlan

* [cli] Program extract thorpArchive def

* [cli] Program extract createPlan def

* [cli] Program refactoring

* [cli] Program remove println showing version

* [cli] Program rename actions parameter as syncPlan

* [core] ThorpArchive add an index to each action

* [cli] Program make SyncPlan available to ThorpArchive

* [core] Pass SyncTotals to Archive

* [domain] Move SyncTotals into module

* [domain] Pass index and SyncTotals to UploadEventListener

* [domain] UploadEventLogger add file count a size progress bars

* [domain] UploadEventLogger better display stability and add file index

* [cli] Index files in correct order

* [cli] Program extends Synchronise

* [core] Rename Synchronise as PlanBuilder

* [cli] Program add test to check actions don't get reordered from plan

* [core] collect file size totals

* [domain] UploadEventLogger include percentage

* [cli] ProgramTest Use wildcards when selecting more than 6 elements
2019-07-04 18:58:31 +01:00
1267b6e313
Add a batch mode that provides a simple log output (#85)
* [changelog] Updated

* [readme] Updated

* [domain] Config Add batch-mode flag

* [core] ConfigOption Add BatchMode option

* [core] ConfigQuery Add batchMode query

Also replaced verbose exists case clauses with a simple contains.

* [core] ConfigOptions added to replace Seq[ConfigOption]

* [core] Syncronise rename method to createPlan

* [cli] Program rename apply as run

* [storage-aws] S3StorageServiceBuilder stop using IO to create object

* [storage-aws] S3StorageServiceBuilder make default service lazy

* [storage-aws] Rename S3ClientCopier => Copier

* [storage-aws] Rename S3ClientDeleter => Deleter

* [storage-aws] Rename S3ClientObjectLister => Lister

* [storage-aws] Only attach upload listener when in batch mode

Only detects batch mode when selected as a command line option

* [core] Synchronise use leftMap rather than swap.map.swap

* [cli] ParseArgs add `-B` and `--batch` options to enable batch mode

* [core] ThorpArchive logs file uploaded when in batch mode
2019-07-02 08:43:52 +01:00
5df752047f
Add a version command-line option (#99)
* [sbt] make name and version available within Scala

Available as `thorp.BuildInfo._`

* [core] ConfigOption Add Version option

* [cli] ParseArgs add command-line options `-V` and `--version`

* [core] ConfigQuery add showVersion query

* [cli] Program shows version then exit

* [cli] Main don't show "Done"
2019-06-30 22:01:02 +01:00
0fa8382fa3
[cli] Program handle actions list in stack safe manner (#93) 2019-06-30 08:47:51 +01:00
c23376a037
Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
ac9a52f93f
Use correct hash locally for comparing multi-part uploaded files (#82)
* [storage-aws] ETagGenerator add stub

* [core] MD5HashGenerator add hex and digest helpers

* [domain] MD5Hash can always provide base64 and also digest

Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.

MD5Hash is now cleaner now that it no longer has Option parameters.

* [core] MD5HashGenerator add stubs to allow reading file chunks

* [domain] MD5HashData add sub-objects

* [domain] MD5HashData move back into test where it belongs

* [sbt] add sbt-bloop plugin

* [domain] MD5HashData Add hash of big-file

* [domain] MD5HashData Add hash of big-file

* [core] MD5HashGenerator find end of chunk correctly

* [core] MD5HashGenerator offset is a Long

* [core] MD5HashGenerator don't read past the end of the file

* [storage-aws] ETagGenerator can reproduce ETags

* [storage-aws] ETagGeneratorTest added

* [storate-aws] ETagGenerator refactoring

* [storage-aws] ETageGenerator refactoring

* [core] SyncSuite remove redundant braces

* [storage-api] HashService added

* [storage-aws] S3HashService added

* [core] LocalFileStream refactoring

* [core] integrate HashService and ETagGenerator

* Optimise imports

* [domain] HexEncoder added to replace java 8 only DataTypeConverter

* [core] MD5HashGenerator refactoring

* [core] S3MetaDataEnricher refactoring

* [core] S3MetaDataEnricherSuite refactoring

* [storage-aws] ETagGeneratorTest refactoring

* [storage-aws] StorageServiceSuite refactoring

* [core] S3MetaDataEnricher refactoring

* [core] refactoring

* [storage-aws] refactoring
2019-06-29 19:07:51 +01:00
9418744136
Display simple error message when bucket is invalid (#81)
* [core,storage-{api,aws}] Allow Lister errors to be reported to user

* [cli,core,storage-*] Display simple error message when bucket is invalid

* [core] optimise imports

* [storate-aws] optimise imports

* [core] SyncSuite don't use get on an optional type
2019-06-27 07:34:15 +01:00
0f8708e19f
Restructure sync to use a State with foldLeft around actions (#74)
* [changelog] updated

* [cli] Program rename parameter

* [core] Add AppState

* [core] Synchronise rought draft replacement for Sync

Uses the AppState

* [core] Synchronise as sequential for-comprehensions

* [core] Synchronise as nested for-comprehensions

* [sbt] thorp(root) depends on cli moduke

* [core] Synchronise extract methods

* [core] Synchronise rewritten

* [core] Synchronise generates actions

* [core] Remove AppState

* [core] ActionSubmitter remove unused implicit config parameter

* [cli] Program rewritten to use Synchronise

* [core] Synchronise useValidConfig accepts Logger implicitly

* [core] Synchronise reorder methods

* [core] Synchronise refactor errorMessages

* [core] SyncLogging logRunStart accepts explicit parameters

* [core] remove old Sync

* [core] Synchronise restore logRunStart

* [domain] Terminal add types to public methods and values

* [domain] UploadEventLogger force flush to terminal

Also make part of the progress message in green.

Not flushing, by using println, cause odd behaviour. Works on normal
terminal, but not great in an emacs terminal. Oh well.

* [core] SyncLogging.logRunFinished remove unused parameters

* [cli] Program restore final summary

* [storage-aws] remove logging from module

* [core] ThorpArchive replaces ActionSubmitter

ActionSubmitter implementation becomes UnversionedMirrorArchive

* [domain] cleaner upload progress messages

* [cli] Program remove unused Logger

* [cli] Program rename parameter

* [core] SyncSuite use Synchronise

* [sbt] Allow storage-aws to share core test classes

* [domain] LocalFile stop storing a lambda

The lambda breaks the equality test between LocalFile instances.

* [core] MD4HashData add missing base64 digest for leafFile

* [core] Synchronise drop DoNothing actions

* [core] SyncSuite update tests

* [sbt] aggregate modules from root module
2019-06-25 08:27:38 +01:00
9d2271fdcf
Introduce backend abstraction to hide S3 (#76)
Add backend abstraction to hide S3
2019-06-22 07:20:59 +01:00
761c1c9784
Is AWS SDK calculating MD5Hash again for a local file? (#50)
* [aws-lib] Uploader provide request with the already calculated md5 hash

* [aws-lib] remove unused accepts method

* [aws-lib] Uploader refactoring

* [domain] Config remove unused threshold and max retries items

* [core] Show upload errors in summary

* [domain] LocalFile add helper to explicitly compare by hash value

Looking to add an optional field to MD5Hash but we want to do our
checks here only on the hash value, not whether a digest is available
or not.

* [core] Sync refactoring

* [core] SyncSuite invoke subject inside it method and after declaring expectations

* [core] SyncSuite use the localfile hash rather than something arbitrary

* [cli] Add `--no-global` and `--no-user` options

* [core] LocalFileStream refactoring

* [core] SyncSuite: ignore user and global configuration files

* [domain] MD5Hash now can optionally store the base64 encoded hash

* [core] MD5HashGenerator pass the digest to MD5Hash

* [aws-lib] Uploader use the base64 encoded hash

* [changelog] updated
2019-06-21 19:20:35 +01:00
910688ee32
Add support for per directory configuration files (#71)
* [core] rename the config supplied from CLI as such

This distinguishes it as config supplied from the command line.

* [core] add ConfigOption

* [core] ConfigOption can update a Config

* [core] new validator for config

* [domain] Config doesn't validate source any more

* [cli] PrintLogger default to not print debug messages

* [cli] Use ConfigOptions and new ConfigValidator

* [sbt] Use common settings for project root

* [domain] RemoteKey can handle when prefix is empty

* [cli] remove banner

* [domain] Logger can create version of itself with debug flipped

* [core] Build and validate Config within core module

This means that the `thorp-lib` module can validate its input from a
list of `ConfigOption`s.

* [core] refactor ConfigurationBuilder

* [core] refactor ConfigurationBuilder

* [sbt] starting back from tagless-final by using IO where needed

* [core] Add ParseConfigFile

* [sbt] Make cats-effect available from the domain

* Roll back from tagless-final to just use cat-effect's IO

* [core] extract ParseConfigLines

* [core] ConfigurationBuilder rename apply as buildConfig

* [core] ParseConfig[Files,Lines] rename apply methods

* [core] refactor ParseConfigFile and add tests

* [core] Sync fix call to run

* [core] SyncSuite update tests to use ConfigOptions
2019-06-20 17:18:46 +01:00
9196dd623f
Rename project to Thorp (#75)
* [sbt] change application name

* [cli] rename package

* [cli] Change displayed application name and description

* [domain] rename package

* [core] fix bad package directory structure

* [core] rename package

* [aws-lib] rename package

* [aws-api] rename package

* [cli] rename programe for usage message

* [bin] rename and update script

* [gitignore] update

* [readme] update
2019-06-17 15:33:49 +01:00
90770eaafb
[cli] Remove verbosity flag (#63) 2019-06-14 20:21:58 +01:00
e3675b5394
Add a debug flag and make debug message hidden by default (#60)
* [cli] add a debug flag to control logging

* [core] show entering a directory as a debug message
2019-06-14 20:00:22 +01:00
a780c5fdfe
[cli] Display "Source is not a directory" if so (#51) 2019-06-14 19:01:46 +01:00
dc0c142762
[core] Sync simplify the call to run (#61) 2019-06-14 17:40:36 +01:00
21b8917395
Simplify logging (#59)
* [cli] Logger add debug method

* [cli] Logger: add alternate info method

* [cli] replace direct calls to legacy Logger.info

* [cli] Extract trait from Logger into core and rename

* [cli] Program use debug and new info

* [core] MD5HashGenerator uses Logger

* [core] Sync uses Logger

* [core] ActionSubmitter uses Logger

* [core] LocalFileStream uses Logger

* [core] SyncLogging uses Logger

* [domain] Move Logger into module

The allows it to be used in the aws-api module

* [aws-lib] S3ClientObjectLister uses Logger

* [aws-lib] Uploader uses Logger

* [aws-lib] S3ClientDeleter uses Logger

* [aws-lib] S3ClientCopier uses Logger

* [core] remove unused legacy logging

* [aws-lib] remove used logging method

* [cli] PrintLogger remove legacy info method
2019-06-14 17:18:53 +01:00
69c09d0091
Tidy Main by extracting Program (#58)
* [cli] ParseArgs now return an Option

* [cli] Program extracted from Main
2019-06-14 13:02:38 +01:00
8d0c3e23c9
Improve purity by moving all IO out to the edge (#52)
* [aws-api] TTFI UploadProgressLogging#logTransfer

* [aws-api] TTFI UploadProgressLogging#logRequestCycle

* [aws-api] TTFI UploadProgressLogging#logByteTransfer

* [aws-api] TTFI UploadProgressListener

* [aws-lib] TTFI UploaderLogging

* [aws-api] UploadProgressListener refactoring

* [aws-api] UploadProgressListener remove IO/Monad

This class is a wrapper for a callback method used by the AWS SDK.
Unfortunately you can't get the IO() created when that listener is
called by the SDK, so unless we manually run unsafeRunSync, as we have
done previously, it would never be executed.

This removes the IO monad entirely and simply runs the code when the
callback is triggered.

* [aws-lib] S3ClientLogging remove unused method

* [aws-lib] TTFI S3ClientLogging

* [aws-lib] TTFI S3ClientCopier

* [aws-lib] TTFI S3ClientObjectLister

* [aws-lib] TTFI Uploader

* [aws-lib] TTFI S3ClientDeleter

* [aws-api] TTFI S3Client

* [aws-lib] TTFI S3ClientBuilder and ThorpS3Client

* [core] TTFI ActionSubmitter

* [cli] TTFI Logger

* [core] TTFI MD5HashGenerator

* [core] TTFI LocalFileStream

* [core] Sync refactoring

* [core] TTFI Sync

* [aws-lib] S3ObjectsByHashSuite truncate lastmodified to match Date

* [aws-lib] ThorpS3ClientSuite truncate lastmodified to match Date

* [core] MD5HashGeneratorTest switch to Id from IO

* [sbt] restrict cats-effect to cli module, cats-core elsewhere

* [core] MD5HashGenerator collapse lines
2019-06-14 08:51:05 +01:00
97efed76b4
Improve upload logging (#44)
* [aws-lib] fold S3ClientUploader trait into it's only implementation

This trait was only implemented by S3ClientTransferManager.

* [core] SyncLogging: more robust matching

No longer cares about parameters to case classes, just their types.

* [cli] Logger uses IO for log methods

* [aws-lib] remove 'transfer-manager'prefix and only show tryCount > 1

* [sbt,cli] remove log4j and scala-logging dependencies

* [domain] move QuoteStripper to Domain

Use it directly in MD5Hash to strip quotes from any input.

* [core] SyncLogging call info in proper context

If the IO.unit returned by the info calls isn't part of the chain that
is returned from the function, then the delayed IO action is never
called.

* [aws-lib] Display size in bytes of file being uploaded

* [core] call info in correct context

* [cli] call info in correct context

* [aws-lib] raise summary fetch message to info 1

* [cli] include correct level in info messages

* [aws-lib] S3ClientLogging adjust logging levels

* [aws-lib] display file sizes in english

* [aws-lib] ObjectLister use IO.bracket properly

* [aws-lib] Copier use IO.bracket properly

* [aws-lib] Deleter refactor

* [aws-lib] TransferManagerLogging remove unused methods

* [aws-lib] TransferManager refactor

* [aws-lib] TransferManager refactor

* [aws-lib] TransferManager displays log messages

Use the UploadProgressListener that was being ignored, and use
unsafeRunSync to execute the suspended effect within the IO[Unit].
Using unsafeRunSync is required to render the effects as the listener
returns Unit, meaning the suspended effects would be discarded.

* [domain] Extract SizeTranslation into module

* [aws-api] report bytes transferred in progress

* [core] fix calls to info

info now returns an IO already, so don't need to wrap it in one.

* [aws-lib] remove unused class

* [aws-lib] UploadProgress displays progress bar while uploading

* [aws-api] UploadProgressLogging optimise imports

* [aws-api] UploadProgressLogging rename variables

* [domain] add Terminal object

* [aws-api] UploadProgressLogging use console width and two lines

- Improved clearing of lines after progress bar
- Use console width for progress bar size

* [aws-lib] S3ClientLogging optimise imports

* [aws-lib] TransferManager clear line before logging

* [aws-lib] rename class as TransferManager

* [aws-lib] rename TransferManger as Uploader to not clash

We are using an AWS SDK class with the same name.
2019-06-10 19:45:36 +01:00
44c66c042c
Include and Exclude behave more like the AWS CLI (#48)
* [domain] rename Filter as Include

* [cli]ParseArgs allow exclude and include parameters to be repeated

* [core] don't include include/exclude details in logging

* [domain] combine Include and Exclude into Filter

Config now collect includes and Excludes into a single list and passed
each file to the Filter.isIncluded method, with the list of Filters,
to determine if a file should be included.
2019-06-08 20:31:20 +01:00
f54c50aaf3
Split into subprojects (#36)
* [sbt] define existing single module project as legacyRoot

* [sbt] add empty cli module depending on legacyRoot

* [cli] move Main to cli module

* [cli] move ParseArgs to cli module

* [sbt] limit scope of scopt dependency to cli module

* [cli] moved logging config to cli module

* [cli] rename module directory

* [aws-api] added empty module

* [sbt] aggregate builds from cli

* [aws-lib] add empty module

* [core] add empty module

* [sbt] add comment graphing module dependencies

* [sbt] adjust module dependencies to reflect plan

Include legacyRoot at the base until it can be redistributed

* [legacy] make some awssdk classes non-private

during this transition, these classes being private would cause problems

* [aws-lib] create S3ClientBuilder

This is copied from the legacy S3Client companion object

* [domain] add empty module

* [domain] move Bucket into module

* [legacy] RemoteKey no longer has dependency on Config

* [domain] move RemoteKey into module

* [domain] move MD5Hash into module

* [legacy] LocalFile no longer had dependency on MD5HashGenerator

* [domain] move LocalFile into module

* [domain] mode LastModified into module

* [domain] move RemoteMetaData into module

* [domain] move S3MetaData into module

* [domain] move Exclude into module

* [domain] move Filter into module

* [domain] move KeyModified into module

* [domain] move HashModified into module

* [domain] RemoteKey.resolve added

* [domain] add dependency on scalatest

* [domain] LocalFile.resolve added

* [legacy] Remove UnitTest

* [legacy] optimise imports

* [domain] move S3ObjectsData moved into module

* [legacy] wrapper for using GeneralProgressListener

* [domain] move Config into module

* [sbt] move aws-api below legacyRoot in dependencies

This will allow use to move S3Client into the aws-api module

* [legacy] rename S3Client companion as S3ClientBuilder

Preparation to move this into its own file.

* Inject Logger via CLI (#34)

* [S3Client] refactor defaultClient()

* [S3Client] transfermanager explicitly uses the same s3client

* [S3ClientPutObjectUploader] refactor putObjectRequest creation

* [cli] copy in Logging trait as Logger class

* [cli] Main uses Logger

* [cli] simplify Logger and pass to Sync.run

* [legacy] SyncLogging converted to companion

* [cli] Logger info can more easily use levels again

* [legacy] LocalFileStream uses injected info

* [legacy] S3MetaDataEnricher remove unused Logging

* [legacy] ActionGenerator remove unused Logging

* [legacy] convert ActionGenerator to an object

* [legacy] import log methods from SyncLogging

* [legacy] move getS3Status from S3Client to S3MetaDataEnricher

* [legact] convert ActionsSubmitter to an object

* [legacy] convert LocalFileStream to an object

* [legacy] move Action case classes inside companion

* [legacy] move UploadEvent case classes inside companion and rename

* [legacy] move S3Action case classes into companion

* [legacy] convert Sync to an object

* [cli] Logger takes verbosity level at construction

No longer needs to be passed the whole Config implicitly for each info
call.

* [legacy] stop passing implicit Config for logging purposes

Pass a more specific implicit info: Int => String => Unit instead

* [legacy] remove DummyS3Client

* [legacy] remove Logging

* [legacy] convert MD5HashGenerator to an object

* [aws-api] move S3Client into module

* [legacy] convert KeyGenerator to an object

* [legacy] don't use IO.unsafeRunSync directly

* [legacy] refactor/rewrite Sync.run

* [legacy] Rewrite sort using a for-comprehension

* [legacy] Sync inline sorting

* [legacy] SyncLogging rename method

* [legacy] repair tests

* [sbt] move core module to a dependency of legacyRoot

* [sbt] add test dependencies to core module

* [core] move classes into module

* [aws-lib] move classes into module

* [sbt] remove legacy root
2019-06-06 19:24:15 +01:00