thorp/build.sbt

173 lines
5 KiB
Text
Raw Normal View History

import sbtassembly.AssemblyPlugin.defaultShellScript
inThisBuild(List(
2019-06-30 19:00:10 +01:00
organization := "net.kemitix.thorp",
homepage := Some(url("https://github.com/kemitix/thorp")),
licenses := List("mit" -> url("https://opensource.org/licenses/MIT")),
developers := List(
Developer(
"kemitix",
"Paul Campbell",
"pcampbell@kemitix.net",
url("https://github.kemitix.net")
)
)
))
val commonSettings = Seq(
sonatypeProfileName := "net.kemitix",
2019-09-01 21:30:16 +01:00
scalaVersion := "2.13.0",
scalacOptions ++= Seq(
2019-09-01 21:30:16 +01:00
"-Ywarn-unused:imports",
"-Xfatal-warnings",
"-feature",
"-deprecation",
"-unchecked",
"-language:postfixOps",
2019-09-01 21:30:16 +01:00
"-language:higherKinds"),
wartremoverErrors ++= Warts.unsafe.filterNot(wart => List(
Wart.Any,
Wart.Nothing,
Wart.Serializable,
Wart.NonUnitStatements,
Wart.StringPlusAny
).contains(wart)),
test in assembly := {},
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
)
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
val applicationSettings = Seq(
name := "thorp",
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
)
val testDependencies = Seq(
libraryDependencies ++= Seq(
2019-06-11 07:36:58 +01:00
"org.scalatest" %% "scalatest" % "3.0.8" % Test,
2019-08-17 18:27:22 +01:00
"org.scalamock" %% "scalamock" % "4.4.0" % Test
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
)
)
val commandLineParsing = Seq(
libraryDependencies ++= Seq(
"com.github.scopt" %% "scopt" % "4.0.0-RC2"
)
)
val awsSdkDependencies = Seq(
libraryDependencies ++= Seq(
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.641",
Drop AWS SDK V2 client (#41) * [sbt] add scalamock as a test dependency * [aws-lib]SyncSuite: minor layout changes * [aws-lib]SyncSuite: remove test * [core] move SyncSuite to same module as subject it tests * [aws-lib]ThorpS3Client: remove commented lines * [aws-lib] remove PutObject versions of Uploader * [aws-lib] rename to TransferManager to remove Multi-part from name * [aws-lib]TransferManager: change logging prefix * [aws-lib] convert logging classes to objects * [aws-lib] convert ObjectLister to use V1 SDK * [aws-lib] convert Copier to use V1 SDK * [aws-lib] extract S3ObjectsBy{Hash,Key} to objects * [aws-lib]S3ClientSuite: rewrite test using mocks * [aws-lib]TransferManager rewrite using for-comprehension * [aws-lib]Copier: remote bucket name from target remote key * [aws-lib]TransferManager: refactor logging to use IO themselves * [aws-lib] Remove test class MyAmazonS3 * [aws-lib]ObjectLister: optimise imports * [aws-lib] S3ClientSuite remove commented code * [aws-lib]ThropS3ClientSuite update to V1 api * [aws-lib]S3ClientSuite: make test as pending It works okay on its own, but when run as part of a suite it fails. Will look at this again once all V2 SDK is removed. * [aws-lib] convert Deleter to use V1 SDK * [aws-lib] Client Logging remove redundant braces * [aws-lib] stop injecting the V2 SDK * [sbt] remove v2 SDK dependencies * [aws-lib] remove redundant helpers for v2 SDK * [sbt] upgrade aws jackson dependencies The jackson libraries used by AWS have security flaws, but are Java 6 compatible, which AWS want to preserve. * [aws-lib] clean up TransferManager tests
2019-06-07 21:17:14 +01:00
// override the versions AWS uses, which is they do to preserve Java 6 compatibility
"com.fasterxml.jackson.core" % "jackson-databind" % "2.10.0",
"com.fasterxml.jackson.dataformat" % "jackson-dataformat-cbor" % "2.10.0",
2019-08-04 20:13:04 +01:00
"javax.xml.bind" % "jaxb-api" % "2.3.1"
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
)
)
val zioDependencies = Seq(
libraryDependencies ++= Seq (
"dev.zio" %% "zio" % "1.0.0-RC13",
"dev.zio" %% "zio-streams" % "1.0.0-RC13"
)
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
)
2019-04-29 20:05:41 +01:00
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
val eipDependencies = Seq(
libraryDependencies ++= Seq(
"net.kemitix" %% "eip-zio" % "0.3.2"
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
)
)
2019-04-29 20:05:41 +01:00
lazy val thorp = (project in file("."))
.settings(commonSettings)
.aggregate(app, cli, config, console, domain, filesystem, lib, storage, `storage-aws`, uishell)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
lazy val app = (project in file("app"))
.settings(commonSettings)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.settings(mainClass in assembly := Some("net.kemitix.thorp.Main"))
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
.settings(applicationSettings)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.settings(eipDependencies)
.settings(Seq(
assemblyOption in assembly := (
assemblyOption in assembly).value
.copy(prependShellScript =
Some(defaultShellScript)),
assemblyJarName in assembly := "thorp"
))
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(cli)
.dependsOn(lib)
.dependsOn(`storage-aws`)
2019-06-17 16:47:01 +01:00
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
lazy val cli = (project in file("cli"))
2019-06-17 16:47:01 +01:00
.settings(commonSettings)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.settings(testDependencies)
.dependsOn(config)
.dependsOn(filesystem % "test->test")
lazy val `storage-aws` = (project in file("storage-aws"))
.settings(commonSettings)
.settings(assemblyJarName in assembly := "storage-aws.jar")
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
.settings(awsSdkDependencies)
.settings(testDependencies)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(storage)
.dependsOn(filesystem % "compile->compile;test->test")
.dependsOn(console)
.dependsOn(lib)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
lazy val lib = (project in file("lib"))
.settings(commonSettings)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.settings(assemblyJarName in assembly := "lib.jar")
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
.settings(testDependencies)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.enablePlugins(BuildInfoPlugin)
.settings(
buildInfoKeys := Seq[BuildInfoKey](name, version),
buildInfoPackage := "thorp"
)
.dependsOn(storage)
.dependsOn(console)
.dependsOn(config)
Use correct hash locally for comparing multi-part uploaded files (#82) * [storage-aws] ETagGenerator add stub * [core] MD5HashGenerator add hex and digest helpers * [domain] MD5Hash can always provide base64 and also digest Rather that store the base 64 digest some of the time, simply decode it from the hex hash. The same for the binary digest. MD5Hash is now cleaner now that it no longer has Option parameters. * [core] MD5HashGenerator add stubs to allow reading file chunks * [domain] MD5HashData add sub-objects * [domain] MD5HashData move back into test where it belongs * [sbt] add sbt-bloop plugin * [domain] MD5HashData Add hash of big-file * [domain] MD5HashData Add hash of big-file * [core] MD5HashGenerator find end of chunk correctly * [core] MD5HashGenerator offset is a Long * [core] MD5HashGenerator don't read past the end of the file * [storage-aws] ETagGenerator can reproduce ETags * [storage-aws] ETagGeneratorTest added * [storate-aws] ETagGenerator refactoring * [storage-aws] ETageGenerator refactoring * [core] SyncSuite remove redundant braces * [storage-api] HashService added * [storage-aws] S3HashService added * [core] LocalFileStream refactoring * [core] integrate HashService and ETagGenerator * Optimise imports * [domain] HexEncoder added to replace java 8 only DataTypeConverter * [core] MD5HashGenerator refactoring * [core] S3MetaDataEnricher refactoring * [core] S3MetaDataEnricherSuite refactoring * [storage-aws] ETagGeneratorTest refactoring * [storage-aws] StorageServiceSuite refactoring * [core] S3MetaDataEnricher refactoring * [core] refactoring * [storage-aws] refactoring
2019-06-29 19:07:51 +01:00
.dependsOn(domain % "compile->compile;test->test")
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(filesystem % "compile->compile;test->test")
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
lazy val storage = (project in file("storage"))
.settings(commonSettings)
.settings(zioDependencies)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.settings(assemblyJarName in assembly := "storage.jar")
.dependsOn(uishell)
.dependsOn(domain)
lazy val uishell = (project in file("uishell"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(eipDependencies)
.settings(assemblyJarName in assembly := "uishell.jar")
.dependsOn(config)
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(console)
.dependsOn(filesystem)
lazy val console = (project in file("console"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(assemblyJarName in assembly := "console.jar")
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(domain)
lazy val config = (project in file("config"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(testDependencies)
.settings(commandLineParsing)
.settings(assemblyJarName in assembly := "config.jar")
.dependsOn(domain % "compile->compile;test->test")
.dependsOn(filesystem)
lazy val filesystem = (project in file("filesystem"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(testDependencies)
.settings(assemblyJarName in assembly := "filesystem.jar")
Restructure using EIP-ZIO channels (#183) * [sbt] Rename storage-api as storage * [storage] remove dependency upon console * [storage] remove dependency upon config * [console] remove dependency upon config * [sbt] Add app module Make cli module actually cli, by moving CliArgs parser into it and Main and Program into app. * add app that depends on cli and thorp-lib * move non-cli specific to app * make cli depend on config * make cli not depend on thorp-lib * [sbt] make module dependencies more explicit * make app depend on storage-aws * make cli depend on filesystem's tests * make thorp-lib depend on core * make thorp-lib not depend on storage-aws * make storage-aws not depend on core's tests * make storage-aws depend on storage * make storage-aws depend on filesystem and its tests * make storage-aws depend on console * make storage-aws depend on core * make core depend on filesystem and its tests * make filesystem depend on domain and its tests * [sbt] merge thorp-lib with core as lib * [sbt] add zio streams * [lib] Add EIPTest * [sbt] Allow NonUnitStatements * [lib] EIPTest Message Channel rewritten using ZIO Stream * [sbt] Add eip-zip 0.2.0 as dependency in lib * Remove file counter and total upload size progress Simplifying UnversionedMirrorArchive so we can create it before we know what actions are needed. * Fetch Remote Data before preparing any plans * [domain] RemoteObjects only holds a single RemoteKey per Hash Having multiple keys for a hash is redundant. They are only used to create copy commands, and only one source remote key is needed for that. * [lib] Add a State trait * [lib] Add FileScanner * Add FileSystem.length(File) * Add Clock to the Environment * [domain] Sources update format * [domain] Asking for a path that isn't in any Sources is fatal There should never be any situation where are path not within a Source is supplied. If there is, then something is badly wrong. * [lib] Add test on use of zio.Ref * [uishell] Add stub module * [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0 * [uishell] Add UIEvent stub * [uishell] Add UIShell stub * [sbt] Add eip-zio dependencies to app module * [app] Wrap existing execution in simple point to point channel * [uishell] Add UIEvent.ShowValidConfig * [app] Remember to end the channel to allow prog to exit * [app] purify environment for showValidConfig * [app] Create type alias for pure effect free channel ref * [app] Program refactoring * [uishell] Add UIEvent.RemoteDataFetched * [domain] Move Counters from lib * [uishell] Add UIEvent.ShowSummary * [lib] Add stub for PushLocalChanges * [lib] Clean up FileScanner Environment types * [lib] End channel after scanning files * [lib] PushLocalChanges uses FileScanner Scans files and sends them to a dummy receiver. * [uishell] Add UIEvent.FileFound * [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload * [lib] FileScanner return LocalFile * [domain] add length to LocalFile * [domain] Add interogation queries to RemoteObjects * [domain] Remove RemoteObject.keyForHashes * [domain] RemoteObjects.remoteHasHash return the key and the hash * [lib] LocalFileSystem.scanCopyUpload create Actions * [domain] Move Action from lib * [uishell] Log actions * [lib] FileScanner respects Filters * [lib] Create remoteKey for files correctly * [lib] LocalFileSystem refactoring * [lib] ThorpArchive.update doesn't need Console * [uishell] Don't log choosen Action * [uishell] Add UIEvent.ActionFinished * [lib] LocalFileSystem refactoring * [lib] Switch to using LocalFileSystem to do Copy/Upload Todo or Broken: - [ ] Delete actions don't happen - [ ] Counters in summary are all zeros * [lib] LocalFileStream display summary counters correctly * [app] Restore ability to delete remote files * [lib] LocalFileSystem deletes remote when local does NOT exist * [filesystem] move hasLocalFile to FileSystem * [filesystem] fix detection of local files from a RemoteKey The configured Prefix wasn't being taken into account, meaning that the expected local file for a RemoteKey was wrong. * [filesystem] fix broken FileSystem test * [domain] fix RemoteKey test * [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility * [app] Program refactorting * [lib] Remove unused class * [lib] Remove test * [uishell] Refactor large method
2019-09-07 07:52:13 +01:00
.dependsOn(domain % "compile->compile;test->test")
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
lazy val domain = (project in file("domain"))
.settings(commonSettings)
.settings(assemblyJarName in assembly := "domain.jar")
Split into subprojects (#36) * [sbt] define existing single module project as legacyRoot * [sbt] add empty cli module depending on legacyRoot * [cli] move Main to cli module * [cli] move ParseArgs to cli module * [sbt] limit scope of scopt dependency to cli module * [cli] moved logging config to cli module * [cli] rename module directory * [aws-api] added empty module * [sbt] aggregate builds from cli * [aws-lib] add empty module * [core] add empty module * [sbt] add comment graphing module dependencies * [sbt] adjust module dependencies to reflect plan Include legacyRoot at the base until it can be redistributed * [legacy] make some awssdk classes non-private during this transition, these classes being private would cause problems * [aws-lib] create S3ClientBuilder This is copied from the legacy S3Client companion object * [domain] add empty module * [domain] move Bucket into module * [legacy] RemoteKey no longer has dependency on Config * [domain] move RemoteKey into module * [domain] move MD5Hash into module * [legacy] LocalFile no longer had dependency on MD5HashGenerator * [domain] move LocalFile into module * [domain] mode LastModified into module * [domain] move RemoteMetaData into module * [domain] move S3MetaData into module * [domain] move Exclude into module * [domain] move Filter into module * [domain] move KeyModified into module * [domain] move HashModified into module * [domain] RemoteKey.resolve added * [domain] add dependency on scalatest * [domain] LocalFile.resolve added * [legacy] Remove UnitTest * [legacy] optimise imports * [domain] move S3ObjectsData moved into module * [legacy] wrapper for using GeneralProgressListener * [domain] move Config into module * [sbt] move aws-api below legacyRoot in dependencies This will allow use to move S3Client into the aws-api module * [legacy] rename S3Client companion as S3ClientBuilder Preparation to move this into its own file. * Inject Logger via CLI (#34) * [S3Client] refactor defaultClient() * [S3Client] transfermanager explicitly uses the same s3client * [S3ClientPutObjectUploader] refactor putObjectRequest creation * [cli] copy in Logging trait as Logger class * [cli] Main uses Logger * [cli] simplify Logger and pass to Sync.run * [legacy] SyncLogging converted to companion * [cli] Logger info can more easily use levels again * [legacy] LocalFileStream uses injected info * [legacy] S3MetaDataEnricher remove unused Logging * [legacy] ActionGenerator remove unused Logging * [legacy] convert ActionGenerator to an object * [legacy] import log methods from SyncLogging * [legacy] move getS3Status from S3Client to S3MetaDataEnricher * [legact] convert ActionsSubmitter to an object * [legacy] convert LocalFileStream to an object * [legacy] move Action case classes inside companion * [legacy] move UploadEvent case classes inside companion and rename * [legacy] move S3Action case classes into companion * [legacy] convert Sync to an object * [cli] Logger takes verbosity level at construction No longer needs to be passed the whole Config implicitly for each info call. * [legacy] stop passing implicit Config for logging purposes Pass a more specific implicit info: Int => String => Unit instead * [legacy] remove DummyS3Client * [legacy] remove Logging * [legacy] convert MD5HashGenerator to an object * [aws-api] move S3Client into module * [legacy] convert KeyGenerator to an object * [legacy] don't use IO.unsafeRunSync directly * [legacy] refactor/rewrite Sync.run * [legacy] Rewrite sort using a for-comprehension * [legacy] Sync inline sorting * [legacy] SyncLogging rename method * [legacy] repair tests * [sbt] move core module to a dependency of legacyRoot * [sbt] add test dependencies to core module * [core] move classes into module * [aws-lib] move classes into module * [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
.settings(testDependencies)
.settings(zioDependencies)
.settings(eipDependencies)