* [sbt] Rename storage-api as storage
* [storage] remove dependency upon console
* [storage] remove dependency upon config
* [console] remove dependency upon config
* [sbt] Add app module
Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.
* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib
* [sbt] make module dependencies more explicit
* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests
* [sbt] merge thorp-lib with core as lib
* [sbt] add zio streams
* [lib] Add EIPTest
* [sbt] Allow NonUnitStatements
* [lib] EIPTest Message Channel rewritten using ZIO Stream
* [sbt] Add eip-zip 0.2.0 as dependency in lib
* Remove file counter and total upload size progress
Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.
* Fetch Remote Data before preparing any plans
* [domain] RemoteObjects only holds a single RemoteKey per Hash
Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.
* [lib] Add a State trait
* [lib] Add FileScanner
* Add FileSystem.length(File)
* Add Clock to the Environment
* [domain] Sources update format
* [domain] Asking for a path that isn't in any Sources is fatal
There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.
* [lib] Add test on use of zio.Ref
* [uishell] Add stub module
* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0
* [uishell] Add UIEvent stub
* [uishell] Add UIShell stub
* [sbt] Add eip-zio dependencies to app module
* [app] Wrap existing execution in simple point to point channel
* [uishell] Add UIEvent.ShowValidConfig
* [app] Remember to end the channel to allow prog to exit
* [app] purify environment for showValidConfig
* [app] Create type alias for pure effect free channel ref
* [app] Program refactoring
* [uishell] Add UIEvent.RemoteDataFetched
* [domain] Move Counters from lib
* [uishell] Add UIEvent.ShowSummary
* [lib] Add stub for PushLocalChanges
* [lib] Clean up FileScanner Environment types
* [lib] End channel after scanning files
* [lib] PushLocalChanges uses FileScanner
Scans files and sends them to a dummy receiver.
* [uishell] Add UIEvent.FileFound
* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload
* [lib] FileScanner return LocalFile
* [domain] add length to LocalFile
* [domain] Add interogation queries to RemoteObjects
* [domain] Remove RemoteObject.keyForHashes
* [domain] RemoteObjects.remoteHasHash return the key and the hash
* [lib] LocalFileSystem.scanCopyUpload create Actions
* [domain] Move Action from lib
* [uishell] Log actions
* [lib] FileScanner respects Filters
* [lib] Create remoteKey for files correctly
* [lib] LocalFileSystem refactoring
* [lib] ThorpArchive.update doesn't need Console
* [uishell] Don't log choosen Action
* [uishell] Add UIEvent.ActionFinished
* [lib] LocalFileSystem refactoring
* [lib] Switch to using LocalFileSystem to do Copy/Upload
Todo or Broken:
- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros
* [lib] LocalFileStream display summary counters correctly
* [app] Restore ability to delete remote files
* [lib] LocalFileSystem deletes remote when local does NOT exist
* [filesystem] move hasLocalFile to FileSystem
* [filesystem] fix detection of local files from a RemoteKey
The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.
* [filesystem] fix broken FileSystem test
* [domain] fix RemoteKey test
* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility
* [app] Program refactorting
* [lib] Remove unused class
* [lib] Remove test
* [uishell] Refactor large method
* [core] Extract Filters from domain.Filter
* [core] extract LocalFileValidator
* [domain] LocalFile remove unused isDirectory
* [domain] LocalFile move/rename relative as relativeToSource on companion
* [domain] LocalFile move and rename matches as matchesHash on companion
* [domain] LocalFile move md5base64 to companion
* [domain] Logger remove
* [domain] MD5Hash move hash to companion
* [domain] MD5Hash move digest to companion
* [domain] MD5Hash move hash64 to companion
* [domain] RemoteKey move class methods to companion
* [domain] Sources move forPath to companion
Led to being able to cleanup LocalFileStream.localFile and adding
LocalFiles.one
* [domain] UploadEventLogger rename method as apply
* [core] Convert HashService into an effect Hasher
* [core] LocalFileStreamSuite fix prep of test env
* [core] LocalFileStreamSuite Refactor
* [core] Add Hasher.hashObjectChunk
* [core] make MD5HashGenerator private
Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.
Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.
* [domain] Add Monoid
* [domain] Sources use Monoid
* [core] ActionGenerator Refactor
* [core] Use >>= and *> operators
* [config] Refactoring
* [storage-aws] Refactoring
* [domain] Refactoring
* [core] Refactoring
* [config] refactoring
* Refactoring
* [core] PlanBuilder Refactoring
* [core] Make PlanBuilder into an object
* [storate-aws] Copier Reduce the number of parameters
* [config] merge config package object with Config object
* [config] Only extract batchMode value where used
Don't pull it out early then pass a boolean around
* [config] new module
* [config] stub module
* [domain] Rename domain.Config as domain.LegacyConfig
* [config] Move LegacyConfig to config module
* [config] Move config parsing and validation into module
* [config] Complete migration to module for Config
* [config] Config You should not name methods after their defining object
* [config] Rename LegacyConfig as Configuration
Also remove redundant uses
* [core] LocalFileStream Refactoring
* [changelog] update
* [sbt] Add ZIO dependency to storage-api
* Convert to use ZIO
* [sbt] remove cats-effect dependency
* [changelog] updated
* [cli] Program restore actions to correct order (copy, upload, delete)
* [cli] Program You should not name methods after their defining object
* [core] ConfigValidationException Redundant braces after class definition
* [core] LocalFileStreamSuite Usage of get on optional type.
* [core] PlanBuilderTest Usage of get on optional type.
* [domain] Drop use of Lenses macro and only create used lenses
* [core] Drop use of Lenses macro and only create used lenses
* [sbt] drop paradise plugin
* [changelog] updated
* [sbt] Add monocle for support for lenses
* [domain] UploadEventListener make a case class
* [domain] Add @Lenses to all case classes
* [core] Add @Lenses to case classes
* [core] ActionGenerator use lense
* [core] ConfigOption use Lenses
* [core] ConfigurationBuilder remove unused fields
* [core] ConfigurationBuilder refactoring
* [core] SyncLogging use lenses
* [core] syncLogging refactoring
* [scalafmt] Add .scalafmt
* [sbt] Add unused import warning and make them errors
* [core] ConfigurationBuilder remove unused import
* [core] ConfigurationBuilder apply scalafmt
* [domain] reformat
* [storage-api] reformat
* [core] reformat and some refactoring
* [storage-aws] reformat
* [cli] reformat
* [core] post rebase fix up
* [storage-aws] UploaderSuite tidy up
* [domain] MD5Hash tweak
Without doing this we have a file that we don't create a valid md5
hash for. See #103
* [storage-aws] UploaderSuite remove unused import
* [storage-aws] StorageServiceSuite reformatted
* [sbt] consistent settings for all modules
Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.
* [storage-aws] Add AmazonTransferManager
This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.
* [storage-aws] Add AmazonUpload
Prevents deprecation errors due to deprecated parameters on Transfer.
* [sbt] mode import to top of file
* [changelog] updated
* [sbt] Specify sonatypeProfileName in commonSettings
This prevents the value being overridded by the `organization` value
from `inThisBuild`.
* [sbt] make name and version available within Scala
Available as `thorp.BuildInfo._`
* [core] ConfigOption Add Version option
* [cli] ParseArgs add command-line options `-V` and `--version`
* [core] ConfigQuery add showVersion query
* [cli] Program shows version then exit
* [cli] Main don't show "Done"
* [sbt] set name for assembly jar
* [travis] run assembly prior to release
* [sbt] disable publishing non-cli or thorp-lib modules
* [travis] alternate format for multiple commands in script
* [sbt,travis] Deploy releases and snapshots to sonatype
* [travis] remove oracle JDK builds
* [travis] remove 'compile' job
* [travis] Try adding scala 2.13
* [travis] test using openjdk8 rather than 10
* [travis] force use of scala version, overriding sbt version
* Revert "[travis] Try adding scala 2.13"
This reverts commit 6725199e009ae1f809247bbc0851b62d63121993.
* Revert "[travis] force use of scala version, overriding sbt version"
This reverts commit 21c94992f40cf2ef39c5126e67b66feaabe1b2c0.
* [sbt] Group modules under n.k.thorp organisation
* [storage-aws] ETagGenerator add stub
* [core] MD5HashGenerator add hex and digest helpers
* [domain] MD5Hash can always provide base64 and also digest
Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.
MD5Hash is now cleaner now that it no longer has Option parameters.
* [core] MD5HashGenerator add stubs to allow reading file chunks
* [domain] MD5HashData add sub-objects
* [domain] MD5HashData move back into test where it belongs
* [sbt] add sbt-bloop plugin
* [domain] MD5HashData Add hash of big-file
* [domain] MD5HashData Add hash of big-file
* [core] MD5HashGenerator find end of chunk correctly
* [core] MD5HashGenerator offset is a Long
* [core] MD5HashGenerator don't read past the end of the file
* [storage-aws] ETagGenerator can reproduce ETags
* [storage-aws] ETagGeneratorTest added
* [storate-aws] ETagGenerator refactoring
* [storage-aws] ETageGenerator refactoring
* [core] SyncSuite remove redundant braces
* [storage-api] HashService added
* [storage-aws] S3HashService added
* [core] LocalFileStream refactoring
* [core] integrate HashService and ETagGenerator
* Optimise imports
* [domain] HexEncoder added to replace java 8 only DataTypeConverter
* [core] MD5HashGenerator refactoring
* [core] S3MetaDataEnricher refactoring
* [core] S3MetaDataEnricherSuite refactoring
* [storage-aws] ETagGeneratorTest refactoring
* [storage-aws] StorageServiceSuite refactoring
* [core] S3MetaDataEnricher refactoring
* [core] refactoring
* [storage-aws] refactoring
* [core] rename the config supplied from CLI as such
This distinguishes it as config supplied from the command line.
* [core] add ConfigOption
* [core] ConfigOption can update a Config
* [core] new validator for config
* [domain] Config doesn't validate source any more
* [cli] PrintLogger default to not print debug messages
* [cli] Use ConfigOptions and new ConfigValidator
* [sbt] Use common settings for project root
* [domain] RemoteKey can handle when prefix is empty
* [cli] remove banner
* [domain] Logger can create version of itself with debug flipped
* [core] Build and validate Config within core module
This means that the `thorp-lib` module can validate its input from a
list of `ConfigOption`s.
* [core] refactor ConfigurationBuilder
* [core] refactor ConfigurationBuilder
* [sbt] starting back from tagless-final by using IO where needed
* [core] Add ParseConfigFile
* [sbt] Make cats-effect available from the domain
* Roll back from tagless-final to just use cat-effect's IO
* [core] extract ParseConfigLines
* [core] ConfigurationBuilder rename apply as buildConfig
* [core] ParseConfig[Files,Lines] rename apply methods
* [core] refactor ParseConfigFile and add tests
* [core] Sync fix call to run
* [core] SyncSuite update tests to use ConfigOptions
* [aws-api] TTFI UploadProgressLogging#logTransfer
* [aws-api] TTFI UploadProgressLogging#logRequestCycle
* [aws-api] TTFI UploadProgressLogging#logByteTransfer
* [aws-api] TTFI UploadProgressListener
* [aws-lib] TTFI UploaderLogging
* [aws-api] UploadProgressListener refactoring
* [aws-api] UploadProgressListener remove IO/Monad
This class is a wrapper for a callback method used by the AWS SDK.
Unfortunately you can't get the IO() created when that listener is
called by the SDK, so unless we manually run unsafeRunSync, as we have
done previously, it would never be executed.
This removes the IO monad entirely and simply runs the code when the
callback is triggered.
* [aws-lib] S3ClientLogging remove unused method
* [aws-lib] TTFI S3ClientLogging
* [aws-lib] TTFI S3ClientCopier
* [aws-lib] TTFI S3ClientObjectLister
* [aws-lib] TTFI Uploader
* [aws-lib] TTFI S3ClientDeleter
* [aws-api] TTFI S3Client
* [aws-lib] TTFI S3ClientBuilder and ThorpS3Client
* [core] TTFI ActionSubmitter
* [cli] TTFI Logger
* [core] TTFI MD5HashGenerator
* [core] TTFI LocalFileStream
* [core] Sync refactoring
* [core] TTFI Sync
* [aws-lib] S3ObjectsByHashSuite truncate lastmodified to match Date
* [aws-lib] ThorpS3ClientSuite truncate lastmodified to match Date
* [core] MD5HashGeneratorTest switch to Id from IO
* [sbt] restrict cats-effect to cli module, cats-core elsewhere
* [core] MD5HashGenerator collapse lines
