kemitix/thorp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Convert to Java (domain, config, storage-aws and filesystem) (#446)

Browse source 
* Java rewrite domain (#438)

* domain.Bucket: convert to Java

* domain.LastModified: convert to Java

* domain.QuoteStripper: convert to Java

* domain.HexEncoder: convert to Java

* domain.MD5Hash: convert to Java

* remove unused import

* domain.RemoteKey: convert to Java

* domain.Action: convert to Java

* domain.Counters: convert to Java

* domain.HashType: convert to Java

* domain.Hashes: convert to Java

* domain.MD5HashData: convert to Java

* domain.Filter: convert to Java

* domain.LocalFile: convert to Java

* domain: make immutable field public

* domain.SizeTranslation: convert to Java

* domain.HashType: restrict access to contstructor

* domain.RemoteObjects: convert to Java

Introduce MapView and Tuple.

* domain.Sources: convert to Java

* domain.StorageEvent: convert to Java

* domain.Terminal: convert to Java

* domain => config: move SimpleLens to only module that uses it

* domain => filesystem: move TemporaryFolder

* domain.Implicits: removed

* parent: make junit, et al available

* domain: add testing dependencies

* domain.HexEncoder: convert test to Java and fix bugs

* domain.HexEncoderTest: replace with Java version

* domain.MD5HashTest: convert to Java

* domain.RemoteKeyTest: convert to Java

* domain.SizeTranslationTest: convert to Java

* domain.TerminalTest: convert to Java

* domain: remove unused dependencies

* parent: rollback zio-streams to match zio and pin them together

* storage-aws: resolve transitive dependency conflicts

* Java rewrite storage aws (#445)

* storage-aws.AmazonS3: convert to Java as AmazonS3Client

* storage-aws.S3Copier: convert to Java

* storage-aws.S3Uploader: convert to Java

* storage-aws.S3Deleter: convert to Java

* storage-aws.S3Lister: convert to Java

* filesystem: write cache data correctly (as supplied)

* domain,filesystem: fix MD5Hash generation

* filesystem: convert to Java (#450)

* remove legacy

* Rewrite config module in Java (#461)

* config.ParseConfigFile: convert to Java

* config.ParseConfigFile: convert to Java

* config.SourceConfigLoader: convert to Java

* WIP config.Configuration: convert to Java

* config.ConfigOption: convert to Java

* config.ConfigOptions: convert to Java

* config.ConfigValidation: convert to Java

* config.ConfigQuery: convert to Java

* config: move classes to correct location

* config.ConfigValidationException: convert to Java

* config.ConfigValidator: convert to Java

* config.ConfigurationBuilder: convert to Java

* config.SimpleLens: removed

* config.Config: remove environment

* config.ConfigOptionTest: convert to Java

* config.ConfigQueryTest: convert to Java

* config.ConfigurationBuilderTest: convert to Java

* config.ParseConfigFileTest: convert to Java

* config.ParseConfigLinesTest: convert to Java

* config: remove scala dependencies and plugin
This commit is contained in:
Paul Campbell 2020-06-21 07:21:21 +01:00 committed by GitHub
parent 823f50d75c
commit 319c46f403
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
170 changed files with 4602 additions and 4472 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
.travis.yml
View file

@ -1,37 +0,0 @@
language: scala
scala:
- 2.13.0
jdk:
- openjdk8
- openjdk11
env:
- AWS_REGION=eu-west-1
before_install:
- git fetch --tags
stages:
- name: test
- name: release
if: ((branch = master AND type = push) OR (tag IS present)) AND NOT fork
jobs:
include:
- stage: test
script: sbt ++$TRAVIS_SCALA_VERSION test
- stage: coverage
script:
- sbt clean coverage test coverageAggregate
- bash <(curl -s https://codecov.io/bash)
- stage: release
script: sbt ++$TRAVIS_SCALA_VERSION ci-release
cache:
directories:
- $HOME/.sbt/1.0/dependency
- $HOME/.sbt/boot/scala*
- $HOME/.sbt/launchers
- $HOME/.ivy2/cache
- $HOME/.coursier
before_cache:
- du -h -d 1 $HOME/.ivy2/cache
- du -h -d 2 $HOME/.sbt/
- find $HOME/.sbt -name "*.lock" -type f -delete
- find $HOME/.ivy2/cache -name "ivydata-*.properties" -type f -delete
- rm -rf $HOME/.ivy2/local

6
app/pom.xml
View file

@ -46,12 +46,6 @@
<artifactId>thorp-uishell</artifactId>
</dependency>
<!-- command line parsing -->
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_2.13</artifactId>
</dependency>
<!-- scala -->
<dependency>
<groupId>org.scala-lang</groupId>

2
app/src/main/resources/META-INF/services/net.kemitix.thorp.domain.HashGenerator Normal file
View file

@ -0,0 +1,2 @@
net.kemitix.thorp.filesystem.MD5HashGenerator
net.kemitix.thorp.storage.aws.S3ETagGenerator

6
app/src/main/scala/net/kemitix/thorp/Main.scala
View file

@ -1,11 +1,8 @@
package net.kemitix.thorp
import net.kemitix.thorp.config.Config
import net.kemitix.thorp.console.Console
import net.kemitix.thorp.filesystem.FileSystem
import net.kemitix.thorp.lib.FileScanner
import net.kemitix.thorp.storage.aws.S3Storage
import net.kemitix.thorp.storage.aws.hasher.S3Hasher
import zio.clock.Clock
import zio.{App, ZEnv, ZIO}
@ -15,9 +12,6 @@ object Main extends App {
extends S3Storage.Live
with Console.Live
with Clock.Live
with Config.Live
with FileSystem.Live
with S3Hasher.Live
with FileScanner.Live
override def run(args: List[String]): ZIO[ZEnv, Nothing, Int] =

87
app/src/main/scala/net/kemitix/thorp/Program.scala
View file

@ -5,89 +5,100 @@ import net.kemitix.eip.zio.{Message, MessageChannel}
import net.kemitix.thorp.cli.CliArgs
import net.kemitix.thorp.config._
import net.kemitix.thorp.console._
import net.kemitix.thorp.domain.{Counters, SimpleLens, StorageEvent}
import net.kemitix.thorp.domain.StorageEvent.{
CopyEvent,
DeleteEvent,
ErrorEvent,
UploadEvent
}
import net.kemitix.thorp.filesystem.{FileSystem, Hasher}
import net.kemitix.thorp.domain.{Counters, RemoteObjects, StorageEvent}
import net.kemitix.thorp.lib._
import net.kemitix.thorp.storage.Storage
import net.kemitix.thorp.uishell.{UIEvent, UIShell}
import zio.clock.Clock
import zio.{RIO, UIO, ZIO}
import scala.io.AnsiColor.{WHITE, RESET}
import zio.{IO, RIO, UIO, ZIO}
import scala.io.AnsiColor.{RESET, WHITE}
import scala.jdk.CollectionConverters._
trait Program {
val version = "0.11.0"
lazy val versionLabel = s"${WHITE}Thorp v${version}$RESET"
lazy val versionLabel = s"${WHITE}Thorp v$version$RESET"
def run(args: List[String]): ZIO[
Storage with Console with Config with Clock with FileSystem with Hasher with FileScanner,
Throwable,
Unit] = {
for {
def run(args: List[String])
: ZIO[Storage with Console with Clock with FileScanner, Nothing, Unit] = {
(for {
cli <- CliArgs.parse(args)
config <- ConfigurationBuilder.buildConfig(cli)
_ <- Config.set(config)
config <- IO(ConfigurationBuilder.buildConfig(cli))
_ <- Console.putStrLn(versionLabel)
_ <- ZIO.when(!showVersion(cli))(executeWithUI.catchAll(handleErrors))
} yield ()
_ <- ZIO.when(!showVersion(cli))(
executeWithUI(config).catchAll(handleErrors))
} yield ())
.catchAll(e => {
Console.putStrLn("An ERROR occurred:")
Console.putStrLn(e.getMessage)
})
}
private def showVersion: ConfigOptions => Boolean =
cli => ConfigQuery.showVersion(cli)
private def executeWithUI =
private def executeWithUI(configuration: Configuration) =
for {
uiEventSender <- execute
uiEventReceiver <- UIShell.receiver
uiEventSender <- execute(configuration)
uiEventReceiver <- UIShell.receiver(configuration)
_ <- MessageChannel.pointToPoint(uiEventSender)(uiEventReceiver).runDrain
} yield ()
type UIChannel = UChannel[Any, UIEvent]
private def execute
: ZIO[Any,
Nothing,
MessageChannel.ESender[
Storage with Config with FileSystem with Hasher with Clock with FileScanner with Console,
Throwable,
UIEvent]] = UIO { uiChannel =>
private def execute(configuration: Configuration): ZIO[
Any,
Nothing,
MessageChannel.ESender[Storage with Clock with FileScanner with Console,
Throwable,
UIEvent]] = UIO { uiChannel =>
(for {
_ <- showValidConfig(uiChannel)
remoteData <- fetchRemoteData(uiChannel)
remoteData <- fetchRemoteData(configuration, uiChannel)
archive <- UIO(UnversionedMirrorArchive)
copyUploadEvents <- LocalFileSystem.scanCopyUpload(uiChannel,
copyUploadEvents <- LocalFileSystem.scanCopyUpload(configuration,
uiChannel,
remoteData,
archive)
deleteEvents <- LocalFileSystem.scanDelete(uiChannel, remoteData, archive)
_ <- showSummary(uiChannel)(copyUploadEvents ++ deleteEvents)
deleteEvents <- LocalFileSystem.scanDelete(configuration,
uiChannel,
remoteData,
archive)
_ <- showSummary(uiChannel)(copyUploadEvents ++ deleteEvents)
} yield ()) <* MessageChannel.endChannel(uiChannel)
}
private def showValidConfig(uiChannel: UIChannel) =
Message.create(UIEvent.ShowValidConfig) >>= MessageChannel.send(uiChannel)
private def fetchRemoteData(uiChannel: UIChannel) =
private def fetchRemoteData(configuration: Configuration,
uiChannel: UIChannel)
: ZIO[Clock with Storage with Console, Throwable, RemoteObjects] = {
val bucket = configuration.bucket
val prefix = configuration.prefix
for {
bucket <- Config.bucket
prefix <- Config.prefix
objects <- Storage.list(bucket, prefix)
_ <- Message.create(UIEvent.RemoteDataFetched(objects.byKey.size)) >>= MessageChannel
.send(uiChannel)
} yield objects
}
private def handleErrors(throwable: Throwable) =
Console.putStrLn("There were errors:") *> logValidationErrors(throwable)
private def logValidationErrors(throwable: Throwable) =
throwable match {
case ConfigValidationException(errors) =>
ZIO.foreach_(errors)(error => Console.putStrLn(s"- $error"))
case validateError: ConfigValidationException =>
ZIO.foreach_(validateError.getErrors.asScala)(error =>
Console.putStrLn(s"- $error"))
}
private def showSummary(uiChannel: UIChannel)(
@ -99,13 +110,11 @@ trait Program {
private def countActivities: (Counters, StorageEvent) => Counters =
(counters: Counters, s3Action: StorageEvent) => {
def increment: SimpleLens[Counters, Int] => Counters =
_.modify(_ + 1)(counters)
s3Action match {
case _: UploadEvent => increment(Counters.uploaded)
case _: CopyEvent => increment(Counters.copied)
case _: DeleteEvent => increment(Counters.deleted)
case _: ErrorEvent => increment(Counters.errors)
case _: UploadEvent => counters.incrementUploaded()
case _: CopyEvent => counters.incrementCopied()
case _: DeleteEvent => counters.incrementDeleted()
case _: ErrorEvent => counters.incrementErrors()
case _ => counters
}
}

172
build.sbt
View file

@ -1,172 +0,0 @@
import sbtassembly.AssemblyPlugin.defaultShellScript
inThisBuild(List(
organization := "net.kemitix.thorp",
homepage := Some(url("https://github.com/kemitix/thorp")),
licenses := List("mit" -> url("https://opensource.org/licenses/MIT")),
developers := List(
Developer(
"kemitix",
"Paul Campbell",
"pcampbell@kemitix.net",
url("https://github.kemitix.net")
)
)
))
val commonSettings = Seq(
sonatypeProfileName := "net.kemitix",
scalaVersion := "2.13.0",
scalacOptions ++= Seq(
"-Ywarn-unused:imports",
"-Xfatal-warnings",
"-feature",
"-deprecation",
"-unchecked",
"-language:postfixOps",
"-language:higherKinds"),
wartremoverErrors ++= Warts.unsafe.filterNot(wart => List(
Wart.Any,
Wart.Nothing,
Wart.Serializable,
Wart.NonUnitStatements,
Wart.StringPlusAny
).contains(wart)),
test in assembly := {},
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
)
val applicationSettings = Seq(
name := "thorp",
)
val testDependencies = Seq(
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.8" % Test,
"org.scalamock" %% "scalamock" % "4.4.0" % Test
)
)
val commandLineParsing = Seq(
libraryDependencies ++= Seq(
"com.github.scopt" %% "scopt" % "4.0.0-RC2"
)
)
val awsSdkDependencies = Seq(
libraryDependencies ++= Seq(
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.797",
// override the versions AWS uses, which is they do to preserve Java 6 compatibility
"com.fasterxml.jackson.core" % "jackson-databind" % "2.10.4",
"com.fasterxml.jackson.dataformat" % "jackson-dataformat-cbor" % "2.10.4",
"javax.xml.bind" % "jaxb-api" % "2.3.1"
)
)
val zioDependencies = Seq(
libraryDependencies ++= Seq (
"dev.zio" %% "zio" % "1.0.0-RC16",
"dev.zio" %% "zio-streams" % "1.0.0-RC16"
)
)
val eipDependencies = Seq(
libraryDependencies ++= Seq(
"net.kemitix" %% "eip-zio" % "0.3.2"
)
)
lazy val thorp = (project in file("."))
.settings(commonSettings)
.aggregate(app, cli, config, console, domain, filesystem, lib, storage, `storage-aws`, uishell)
lazy val app = (project in file("app"))
.settings(commonSettings)
.settings(mainClass in assembly := Some("net.kemitix.thorp.Main"))
.settings(applicationSettings)
.settings(eipDependencies)
.settings(Seq(
assemblyOption in assembly := (
assemblyOption in assembly).value
.copy(prependShellScript =
Some(defaultShellScript)),
assemblyJarName in assembly := "thorp"
))
.dependsOn(cli)
.dependsOn(lib)
.dependsOn(`storage-aws`)
lazy val cli = (project in file("cli"))
.settings(commonSettings)
.settings(testDependencies)
.dependsOn(config)
.dependsOn(filesystem % "test->test")
lazy val `storage-aws` = (project in file("storage-aws"))
.settings(commonSettings)
.settings(assemblyJarName in assembly := "storage-aws.jar")
.settings(awsSdkDependencies)
.settings(testDependencies)
.dependsOn(storage)
.dependsOn(filesystem % "compile->compile;test->test")
.dependsOn(console)
.dependsOn(lib)
lazy val lib = (project in file("lib"))
.settings(commonSettings)
.settings(assemblyJarName in assembly := "lib.jar")
.settings(testDependencies)
.enablePlugins(BuildInfoPlugin)
.settings(
buildInfoKeys := Seq[BuildInfoKey](name, version),
buildInfoPackage := "thorp"
)
.dependsOn(storage)
.dependsOn(console)
.dependsOn(config)
.dependsOn(domain % "compile->compile;test->test")
.dependsOn(filesystem % "compile->compile;test->test")
lazy val storage = (project in file("storage"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(assemblyJarName in assembly := "storage.jar")
.dependsOn(uishell)
.dependsOn(domain)
lazy val uishell = (project in file("uishell"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(eipDependencies)
.settings(assemblyJarName in assembly := "uishell.jar")
.dependsOn(config)
.dependsOn(console)
.dependsOn(filesystem)
lazy val console = (project in file("console"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(assemblyJarName in assembly := "console.jar")
.dependsOn(domain)
lazy val config = (project in file("config"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(testDependencies)
.settings(commandLineParsing)
.settings(assemblyJarName in assembly := "config.jar")
.dependsOn(domain % "compile->compile;test->test")
.dependsOn(filesystem)
lazy val filesystem = (project in file("filesystem"))
.settings(commonSettings)
.settings(zioDependencies)
.settings(testDependencies)
.settings(assemblyJarName in assembly := "filesystem.jar")
.dependsOn(domain % "compile->compile;test->test")
lazy val domain = (project in file("domain"))
.settings(commonSettings)
.settings(assemblyJarName in assembly := "domain.jar")
.settings(testDependencies)
.settings(zioDependencies)
.settings(eipDependencies)

17
cli/pom.xml
View file

@ -22,23 +22,30 @@
<artifactId>thorp-filesystem</artifactId>
</dependency>
<!-- command line parsing -->
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_2.13</artifactId>
</dependency>
<!-- scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<!-- zio -->
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio_2.13</artifactId>
</dependency>
<!-- scala - testing -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.13</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalamock</groupId>
<artifactId>scalamock_2.13</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>

26
cli/src/main/scala/net/kemitix/thorp/cli/CliArgs.scala
View file

@ -2,6 +2,8 @@ package net.kemitix.thorp.cli
import java.nio.file.Paths
import scala.jdk.CollectionConverters._
import net.kemitix.thorp.config.{ConfigOption, ConfigOptions}
import scopt.OParser
import zio.Task
@ -11,7 +13,7 @@ object CliArgs {
def parse(args: List[String]): Task[ConfigOptions] = Task {
OParser
.parse(configParser, args, List())
.map(ConfigOptions(_))
.map(options => ConfigOptions.create(options.asJava))
.getOrElse(ConfigOptions.empty)
}
@ -22,40 +24,40 @@ object CliArgs {
programName("thorp"),
head("thorp"),
opt[Unit]('V', "version")
.action((_, cos) => ConfigOption.Version :: cos)
.action((_, cos) => ConfigOption.version() :: cos)
.text("Show version"),
opt[Unit]('B', "batch")
.action((_, cos) => ConfigOption.BatchMode :: cos)
.action((_, cos) => ConfigOption.batchMode() :: cos)
.text("Enable batch-mode"),
opt[String]('s', "source")
.unbounded()
.action((str, cos) => ConfigOption.Source(Paths.get(str)) :: cos)
.action((str, cos) => ConfigOption.source(Paths.get(str)) :: cos)
.text("Source directory to sync to destination"),
opt[String]('b', "bucket")
.action((str, cos) => ConfigOption.Bucket(str) :: cos)
.action((str, cos) => ConfigOption.bucket(str) :: cos)
.text("S3 bucket name"),
opt[String]('p', "prefix")
.action((str, cos) => ConfigOption.Prefix(str) :: cos)
.action((str, cos) => ConfigOption.prefix(str) :: cos)
.text("Prefix within the S3 Bucket"),
opt[Int]('P', "parallel")
.action((int, cos) => ConfigOption.Parallel(int) :: cos)
.action((int, cos) => ConfigOption.parallel(int) :: cos)
.text("Maximum Parallel uploads"),
opt[String]('i', "include")
.unbounded()
.action((str, cos) => ConfigOption.Include(str) :: cos)
.action((str, cos) => ConfigOption.include(str) :: cos)
.text("Include only matching paths"),
opt[String]('x', "exclude")
.unbounded()
.action((str, cos) => ConfigOption.Exclude(str) :: cos)
.action((str, cos) => ConfigOption.exclude(str) :: cos)
.text("Exclude matching paths"),
opt[Unit]('d', "debug")
.action((_, cos) => ConfigOption.Debug() :: cos)
.action((_, cos) => ConfigOption.debug() :: cos)
.text("Enable debug logging"),
opt[Unit]("no-global")
.action((_, cos) => ConfigOption.IgnoreGlobalOptions :: cos)
.action((_, cos) => ConfigOption.ignoreGlobalOptions() :: cos)
.text("Ignore global configuration"),
opt[Unit]("no-user")
.action((_, cos) => ConfigOption.IgnoreUserOptions :: cos)
.action((_, cos) => ConfigOption.ignoreUserOptions() :: cos)
.text("Ignore user configuration")
)
}

14
cli/src/test/scala/net/kemitix/thorp/cli/CliArgsTest.scala
View file

@ -2,19 +2,19 @@ package net.kemitix.thorp.cli
import java.nio.file.Paths
import net.kemitix.thorp.config.ConfigOption.Debug
import net.kemitix.thorp.config.{ConfigOptions, ConfigQuery}
import net.kemitix.thorp.config.{ConfigOption, ConfigOptions, ConfigQuery}
import net.kemitix.thorp.filesystem.Resource
import org.scalatest.FunSpec
import zio.DefaultRuntime
import scala.jdk.CollectionConverters._
import scala.util.Try
class CliArgsTest extends FunSpec {
private val runtime = new DefaultRuntime {}
val source = Resource(this, "")
val source = Resource.select(this, "")
describe("parse - source") {
def invokeWithSource(path: String) =
@ -36,7 +36,8 @@ class CliArgsTest extends FunSpec {
it("should get multiple sources") {
val expected = Some(Set("path1", "path2").map(Paths.get(_)))
val configOptions = invoke(args)
val result = configOptions.map(ConfigQuery.sources(_).paths.toSet)
val result =
configOptions.map(ConfigQuery.sources(_).paths.asScala.toSet)
assertResult(expected)(result)
}
}
@ -50,7 +51,8 @@ class CliArgsTest extends FunSpec {
maybeOptions.getOrElse(ConfigOptions.empty)
}
val containsDebug = ConfigOptions.contains(Debug())(_)
val containsDebug = (options: ConfigOptions) =>
options.options.stream().anyMatch(_.isInstanceOf[ConfigOption.Debug])
describe("when no debug flag") {
val configOptions = invokeWithArgument("")
@ -96,7 +98,7 @@ class CliArgsTest extends FunSpec {
}
private def pathTo(value: String): String =
Try(Resource(this, value))
Try(Resource.select(this, value))
.map(_.getCanonicalPath)
.getOrElse("[not-found]")

55
config/pom.xml
View file

@ -12,6 +12,19 @@
<name>config</name>
<dependencies>
<!-- mon -->
<dependency>
<groupId>net.kemitix</groupId>
<artifactId>mon</artifactId>
</dependency>
<!-- lombok -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- thorp -->
<dependency>
<groupId>net.kemitix.thorp</groupId>
@ -22,48 +35,16 @@
<artifactId>thorp-filesystem</artifactId>
</dependency>
<!-- command line parsing -->
<!-- testing -->
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_2.13</artifactId>
</dependency>
<!-- scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<!-- zio -->
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio_2.13</artifactId>
</dependency>
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio-streams_2.13</artifactId>
</dependency>
<!-- scala - testing -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.13</artifactId>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalamock</groupId>
<artifactId>scalamock_2.13</artifactId>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

178
config/src/main/java/net/kemitix/thorp/config/ConfigOption.java Normal file
View file

@ -0,0 +1,178 @@
package net.kemitix.thorp.config;
import lombok.EqualsAndHashCode;
import net.kemitix.mon.TypeAlias;
import net.kemitix.thorp.domain.Filter;
import net.kemitix.thorp.domain.RemoteKey;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public interface ConfigOption {
Configuration update(Configuration config);
static ConfigOption source(Path path) {
return new Source(path);
}
class Source extends TypeAlias<Path> implements ConfigOption {
private Source(Path value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
return config.withSources(config.sources.append(getValue()));
}
public Path path() {
return getValue();
}
}
static ConfigOption bucket(String name) {
return new Bucket(name);
}
class Bucket extends TypeAlias<String> implements ConfigOption {
private Bucket(String value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
return config.withBucket(
net.kemitix.thorp.domain.Bucket.named(getValue()));
}
}
static ConfigOption prefix(String path) {
return new Prefix(path);
}
class Prefix extends TypeAlias<String> implements ConfigOption {
private Prefix(String value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
return config.withPrefix(RemoteKey.create(getValue()));
}
}
static ConfigOption include(String pattern) {
return new Include(pattern);
}
class Include extends TypeAlias<String> implements ConfigOption {
private Include(String value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
List<Filter> filters = new ArrayList<>(config.filters);
filters.add(net.kemitix.thorp.domain.Filter.include(getValue()));
return config.withFilters(filters);
}
}
static ConfigOption exclude(String pattern) {
return new Exclude(pattern);
}
class Exclude extends TypeAlias<String> implements ConfigOption {
private Exclude(String value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
List<Filter> filters = new ArrayList<>(config.filters);
filters.add(net.kemitix.thorp.domain.Filter.exclude(getValue()));
return config.withFilters(filters);
}
}
static ConfigOption debug() {
return new Debug();
}
@EqualsAndHashCode
class Debug implements ConfigOption {
@Override
public Configuration update(Configuration config) {
return config.withDebug(true);
}
@Override
public String toString() {
return "Debug";
}
}
static ConfigOption batchMode() {
return new BatchMode();
}
class BatchMode implements ConfigOption {
@Override
public Configuration update(Configuration config) {
return config.withBatchMode(true);
}
@Override
public String toString() {
return "BatchMode";
}
}
static ConfigOption version() {
return new Version();
}
class Version implements ConfigOption {
@Override
public Configuration update(Configuration config) {
return config;
}
@Override
public String toString() {
return "Version";
}
}
static ConfigOption ignoreUserOptions() {
return new IgnoreUserOptions();
}
class IgnoreUserOptions implements ConfigOption {
@Override
public Configuration update(Configuration config) {
return config;
}
@Override
public String toString() {
return "Ignore User Options";
}
}
static ConfigOption ignoreGlobalOptions() {
return new IgnoreGlobalOptions();
}
class IgnoreGlobalOptions implements ConfigOption {
@Override
public Configuration update(Configuration config) {
return config;
}
@Override
public String toString() {
return "Ignore Global Options";
}
}
static ConfigOption parallel(int factor) {
return new Parallel(factor);
}
class Parallel extends TypeAlias<Integer> implements ConfigOption {
protected Parallel(Integer value) {
super(value);
}
@Override
public Configuration update(Configuration config) {
return config.withParallel(getValue());
}
public int factor() {
return getValue();
}
@Override
public String toString() {
return "Parallel: " + getValue();
}
}
}

60
config/src/main/java/net/kemitix/thorp/config/ConfigOptions.java Normal file
View file

@ -0,0 +1,60 @@
package net.kemitix.thorp.config;
import lombok.EqualsAndHashCode;
import lombok.RequiredArgsConstructor;
import java.util.*;
public interface ConfigOptions {
List<ConfigOption> options();
ConfigOptions merge(ConfigOptions other);
ConfigOptions prepend(ConfigOption configOption);
boolean containsInstanceOf(Class<? extends ConfigOption> type);
static int parallel(ConfigOptions configOptions) {
return configOptions.options()
.stream()
.filter(option -> option instanceof ConfigOption.Parallel)
.map(ConfigOption.Parallel.class::cast)
.findFirst()
.map(ConfigOption.Parallel::factor)
.orElse(1);
}
static ConfigOptions empty() {
return create(Collections.emptyList());
}
static ConfigOptions create(List<ConfigOption> options) {
return new ConfigOptionsImpl(options);
}
@EqualsAndHashCode
@RequiredArgsConstructor
class ConfigOptionsImpl implements ConfigOptions {
private final List<ConfigOption> options;
@Override
public List<ConfigOption> options() {
return new ArrayList<>(options);
}
@Override
public ConfigOptions merge(ConfigOptions other) {
List<ConfigOption> optionList = options();
other.options().stream()
.filter(o -> !optionList.contains(o))
.forEach(optionList::add);
return ConfigOptions.create(optionList);
}
@Override
public ConfigOptions prepend(ConfigOption configOption) {
List<ConfigOption> optionList = new ArrayList<>();
optionList.add(configOption);
options().stream()
.filter(o -> !optionList.contains(0))
.forEach(optionList::add);
return ConfigOptions.create(optionList);
}
@Override
public boolean containsInstanceOf(Class<? extends ConfigOption> type) {
return options.stream()
.anyMatch(option ->
type.isAssignableFrom(option.getClass()));
}
}
}

49
config/src/main/java/net/kemitix/thorp/config/ConfigQuery.java Normal file
View file

@ -0,0 +1,49 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Sources;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
public interface ConfigQuery {
static boolean showVersion(ConfigOptions configOptions) {
return configOptions.options().stream()
.anyMatch(configOption ->
configOption instanceof ConfigOption.Version);
}
static boolean batchMode(ConfigOptions configOptions) {
return configOptions.options().stream()
.anyMatch(configOption ->
configOption instanceof ConfigOption.BatchMode);
}
static boolean ignoreUserOptions(ConfigOptions configOptions) {
return configOptions.options().stream()
.anyMatch(configOption ->
configOption instanceof ConfigOption.IgnoreUserOptions);
}
static boolean ignoreGlobalOptions(ConfigOptions configOptions) {
return configOptions.options().stream()
.anyMatch(configOption ->
configOption instanceof ConfigOption.IgnoreGlobalOptions);
}
static Sources sources(ConfigOptions configOptions) {
List<Path> explicitPaths = configOptions.options().stream()
.filter(configOption ->
configOption instanceof ConfigOption.Source)
.map(ConfigOption.Source.class::cast)
.map(ConfigOption.Source::path)
.collect(Collectors.toList());
if (explicitPaths.isEmpty()) {
return Sources.create(Collections.singletonList(Paths.get(System.getenv("PWD"))));
}
return Sources.create(explicitPaths);
}
}

26
config/src/main/java/net/kemitix/thorp/config/ConfigValidation.java Normal file
View file

@ -0,0 +1,26 @@
package net.kemitix.thorp.config;
import java.io.File;
@FunctionalInterface
public interface ConfigValidation {
String errorMessage();
static ConfigValidation sourceIsNotADirectory(File file) {
return () -> "Source must be a directory: " + file;
}
static ConfigValidation sourceIsNotReadable(File file) {
return () -> "Source must be readable: " + file;
}
static ConfigValidation bucketNameIsMissing() {
return () -> "Bucket name is missing";
}
static ConfigValidation errorReadingFile(File file, String message) {
return () -> String.format(
"Error reading file '%s': %s",
file, message);
}
}

12
config/src/main/java/net/kemitix/thorp/config/ConfigValidationException.java Normal file
View file

@ -0,0 +1,12 @@
package net.kemitix.thorp.config;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import java.util.List;
@Getter
@RequiredArgsConstructor
public class ConfigValidationException extends Exception {
private final List<ConfigValidation> errors;
}

47
config/src/main/java/net/kemitix/thorp/config/ConfigValidator.java Normal file
View file

@ -0,0 +1,47 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Bucket;
import net.kemitix.thorp.domain.Sources;
import java.io.File;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
public interface ConfigValidator {
static Configuration validateConfig(Configuration config) throws ConfigValidationException {
validateSources(config.sources);
validateBucket(config.bucket);
return config;
}
static void validateBucket(Bucket bucket) throws ConfigValidationException {
if (bucket.name().isEmpty()) {
System.out.println("Bucket name is missing: " + bucket);
throw new ConfigValidationException(
Collections.singletonList(
ConfigValidation.bucketNameIsMissing()));
}
}
static void validateSources(Sources sources) throws ConfigValidationException {
List<ConfigValidation> errors = new ArrayList<>();
sources.paths().forEach(path ->
errors.addAll(validateAsSource(path.toFile())));
if (!errors.isEmpty()) {
throw new ConfigValidationException(errors);
}
}
static Collection<? extends ConfigValidation> validateAsSource(File file) {
if (!file.isDirectory())
return Collections.singletonList(
ConfigValidation.sourceIsNotADirectory(file));
if (!file.canRead())
return Collections.singletonList(
ConfigValidation.sourceIsNotReadable(file));
return Collections.emptyList();
}
}

35
config/src/main/java/net/kemitix/thorp/config/Configuration.java Normal file
View file

@ -0,0 +1,35 @@
package net.kemitix.thorp.config;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.With;
import net.kemitix.thorp.domain.Bucket;
import net.kemitix.thorp.domain.Filter;
import net.kemitix.thorp.domain.RemoteKey;
import net.kemitix.thorp.domain.Sources;
import java.util.Collections;
import java.util.List;
@With
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class Configuration {
public final Bucket bucket;
public final RemoteKey prefix;
public final List<Filter> filters;
public final boolean debug;
public final boolean batchMode;
public final int parallel;
public final Sources sources;
static Configuration create() {
return new Configuration(
Bucket.named(""),
RemoteKey.create(""),
Collections.emptyList(),
false,
false,
1,
Sources.emptySources
);
}
}

45
config/src/main/java/net/kemitix/thorp/config/ConfigurationBuilder.java Normal file
View file

@ -0,0 +1,45 @@
package net.kemitix.thorp.config;
import java.io.File;
import java.io.IOException;
public interface ConfigurationBuilder {
static Configuration buildConfig(ConfigOptions priorityOpts) throws IOException, ConfigValidationException {
return new ConfigurationBuilderImpl().buildConfig(priorityOpts);
}
class ConfigurationBuilderImpl implements ConfigurationBuilder {
private static final String userConfigFile = ".config/thorp.conf";
private static final File globalConfig = new File("/etc/thorp.conf");
private static final File userHome = new File(System.getProperty("user.home"));
Configuration buildConfig(ConfigOptions priorityOpts) throws IOException, ConfigValidationException {
return ConfigValidator.validateConfig(
collateOptions(getConfigOptions(priorityOpts)));
}
private ConfigOptions getConfigOptions(ConfigOptions priorityOpts) throws IOException {
ConfigOptions sourceOpts = SourceConfigLoader.loadSourceConfigs(ConfigQuery.sources(priorityOpts));
ConfigOptions userOpts = userOptions(priorityOpts.merge(sourceOpts));
ConfigOptions globalOpts = globalOptions(priorityOpts.merge(sourceOpts.merge(userOpts)));
return priorityOpts.merge(sourceOpts.merge(userOpts.merge(globalOpts)));
}
private ConfigOptions userOptions(ConfigOptions priorityOpts) throws IOException {
if (ConfigQuery.ignoreUserOptions(priorityOpts)) {
return ConfigOptions.empty();
}
return ParseConfigFile.parseFile(
new File(userHome, userConfigFile));
}
private ConfigOptions globalOptions(ConfigOptions priorityOpts) throws IOException {
if (ConfigQuery.ignoreGlobalOptions(priorityOpts)) {
return ConfigOptions.empty();
}
return ParseConfigFile.parseFile(globalConfig);
}
private Configuration collateOptions(ConfigOptions configOptions) {
Configuration config = Configuration.create();
for (ConfigOption configOption : configOptions.options()) {
config = configOption.update(config);
}
return config;
}
}
}

18
config/src/main/java/net/kemitix/thorp/config/ParseConfigFile.java Normal file
View file

@ -0,0 +1,18 @@
package net.kemitix.thorp.config;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.List;
public interface ParseConfigFile {
static ConfigOptions parseFile(File file) throws IOException {
if (file.exists()) {
System.out.println("Reading config: " + file);
ConfigOptions configOptions = new ParseConfigLines()
.parseLines(Files.readAllLines(file.toPath()));
return configOptions;
}
return ConfigOptions.empty();
}
}

71
config/src/main/java/net/kemitix/thorp/config/ParseConfigLines.java Normal file
View file

@ -0,0 +1,71 @@
package net.kemitix.thorp.config;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class ParseConfigLines {
private static final String pattern = "^\\s*(?<key>\\S*)\\s*=\\s*(?<value>\\S*)\\s*$";
private static final Pattern format = Pattern.compile(pattern);
ConfigOptions parseLines(List<String> lines) {
return ConfigOptions.create(
lines.stream()
.flatMap(this::parseLine)
.collect(Collectors.toList()));
}
private Stream<ConfigOption> parseLine(String str) {
Matcher m = format.matcher(str);
if (m.matches()) {
return parseKeyValue(m.group("key"), m.group("value"));
}
return Stream.empty();
}
private Stream<ConfigOption> parseKeyValue(String key, String value) {
switch (key.toLowerCase()) {
case "parallel":
return parseInt(value).map(ConfigOption::parallel);
case "source":
return Stream.of(ConfigOption.source(Paths.get(value)));
case "bucket":
return Stream.of(ConfigOption.bucket(value));
case "prefix":
return Stream.of(ConfigOption.prefix(value));
case "include":
return Stream.of(ConfigOption.include(value));
case "exclude":
return Stream.of(ConfigOption.exclude(value));
case "debug":
if (truthy(value))
return Stream.of(ConfigOption.debug());
// fall through to default
default:
return Stream.empty();
}
}
private Stream<Integer> parseInt(String value) {
try {
return Stream.of(Integer.parseInt(value));
} catch (NumberFormatException e) {
return Stream.empty();
}
}
private boolean truthy(String value) {
switch (value.toLowerCase()) {
case "true":
case "yes":
case "enabled":
return true;
default:
return false;
}
}
}

32
config/src/main/java/net/kemitix/thorp/config/SourceConfigLoader.java Normal file
View file

@ -0,0 +1,32 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Sources;
import java.io.File;
import java.io.IOException;
import java.nio.file.Path;
import java.util.stream.Collectors;
public interface SourceConfigLoader {
static ConfigOptions loadSourceConfigs(Sources sources) throws IOException {
// add each source as an option
ConfigOptions configOptions =
ConfigOptions.create(
sources.paths()
.stream()
.peek(path -> {
System.out.println("Using source: " + path);
})
.map(ConfigOption::source)
.collect(Collectors.toList()));
// add settings from each source as options
for (Path path : sources.paths()) {
configOptions = configOptions.merge(
ParseConfigFile.parseFile(
new File(path.toFile(), ".thorp.conf")));
}
return configOptions;
}
}

73
config/src/main/scala/net/kemitix/thorp/config/Config.scala
View file

@ -1,73 +0,0 @@
package net.kemitix.thorp.config
import java.util.concurrent.atomic.AtomicReference
import net.kemitix.thorp.domain.{Bucket, Filter, RemoteKey, Sources}
import zio.{UIO, ZIO}
trait Config {
val config: Config.Service
}
object Config {
trait Service {
def setConfiguration(config: Configuration): ZIO[Config, Nothing, Unit]
def isBatchMode: ZIO[Config, Nothing, Boolean]
def bucket: ZIO[Config, Nothing, Bucket]
def prefix: ZIO[Config, Nothing, RemoteKey]
def sources: ZIO[Config, Nothing, Sources]
def filters: ZIO[Config, Nothing, List[Filter]]
def parallel: UIO[Int]
}
trait Live extends Config {
val config: Service = new Service {
private val configRef = new AtomicReference(Configuration.empty)
override def setConfiguration(
config: Configuration): ZIO[Config, Nothing, Unit] =
UIO(configRef.set(config))
override def bucket: ZIO[Config, Nothing, Bucket] =
UIO(configRef.get).map(_.bucket)
override def sources: ZIO[Config, Nothing, Sources] =
UIO(configRef.get).map(_.sources)
override def prefix: ZIO[Config, Nothing, RemoteKey] =
UIO(configRef.get).map(_.prefix)
override def isBatchMode: ZIO[Config, Nothing, Boolean] =
UIO(configRef.get).map(_.batchMode)
override def filters: ZIO[Config, Nothing, List[Filter]] =
UIO(configRef.get).map(_.filters)
override def parallel: UIO[Int] = UIO(configRef.get).map(_.parallel)
}
}
object Live extends Live
final def set(config: Configuration): ZIO[Config, Nothing, Unit] =
ZIO.accessM(_.config setConfiguration config)
final def batchMode: ZIO[Config, Nothing, Boolean] =
ZIO.accessM(_.config isBatchMode)
final def bucket: ZIO[Config, Nothing, Bucket] =
ZIO.accessM(_.config bucket)
final def prefix: ZIO[Config, Nothing, RemoteKey] =
ZIO.accessM(_.config prefix)
final def sources: ZIO[Config, Nothing, Sources] =
ZIO.accessM(_.config sources)
final def filters: ZIO[Config, Nothing, List[Filter]] =
ZIO.accessM(_.config filters)
final def parallel: ZIO[Config, Nothing, Int] =
ZIO.accessM(_.config parallel)
}

73
config/src/main/scala/net/kemitix/thorp/config/ConfigOption.scala
View file

@ -1,73 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Path
import net.kemitix.thorp.config.Configuration._
import net.kemitix.thorp.domain
import net.kemitix.thorp.domain.RemoteKey
sealed trait ConfigOption {
def update(config: Configuration): Configuration
}
object ConfigOption {
final case class Source(path: Path) extends ConfigOption {
override def update(config: Configuration): Configuration =
sources.modify(_ + path)(config)
}
final case class Bucket(name: String) extends ConfigOption {
override def update(config: Configuration): Configuration =
if (config.bucket.name.isEmpty)
bucket.set(domain.Bucket(name))(config)
else
config
}
final case class Prefix(path: String) extends ConfigOption {
override def update(config: Configuration): Configuration =
if (config.prefix.key.isEmpty)
prefix.set(RemoteKey(path))(config)
else
config
}
final case class Include(pattern: String) extends ConfigOption {
override def update(config: Configuration): Configuration =
filters.modify(domain.Filter.Include(pattern) :: _)(config)
}
final case class Exclude(pattern: String) extends ConfigOption {
override def update(config: Configuration): Configuration =
filters.modify(domain.Filter.Exclude(pattern) :: _)(config)
}
final case class Debug() extends ConfigOption {
override def update(config: Configuration): Configuration =
debug.set(true)(config)
}
case object Version extends ConfigOption {
override def update(config: Configuration): Configuration = config
}
case object BatchMode extends ConfigOption {
override def update(config: Configuration): Configuration =
batchMode.set(true)(config)
}
case object IgnoreUserOptions extends ConfigOption {
override def update(config: Configuration): Configuration = config
}
case object IgnoreGlobalOptions extends ConfigOption {
override def update(config: Configuration): Configuration = config
}
case class Parallel(factor: Int) extends ConfigOption {
override def update(config: Configuration): Configuration =
parallel.set(factor)(config)
}
}

37
config/src/main/scala/net/kemitix/thorp/config/ConfigOptions.scala
View file

@ -1,37 +0,0 @@
package net.kemitix.thorp.config
import net.kemitix.thorp.domain.SimpleLens
final case class ConfigOptions(options: List[ConfigOption]) {
def ++(other: ConfigOptions): ConfigOptions =
ConfigOptions.combine(this, other)
def ::(head: ConfigOption): ConfigOptions =
ConfigOptions(head :: options)
}
object ConfigOptions {
val defaultParallel = 1
def parallel(configOptions: ConfigOptions): Int = {
configOptions.options
.collectFirst {
case ConfigOption.Parallel(factor) => factor
}
.getOrElse(defaultParallel)
}
val empty: ConfigOptions = ConfigOptions(List.empty)
val options: SimpleLens[ConfigOptions, List[ConfigOption]] =
SimpleLens[ConfigOptions, List[ConfigOption]](_.options,
c => a => c.copy(options = a))
def combine(
x: ConfigOptions,
y: ConfigOptions
): ConfigOptions = ConfigOptions(x.options ++ y.options)
def contains[A1 >: ConfigOption](elem: A1)(
configOptions: ConfigOptions): Boolean =
configOptions.options.contains(elem)
}

35
config/src/main/scala/net/kemitix/thorp/config/ConfigQuery.scala
View file

@ -1,35 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Paths
import net.kemitix.thorp.domain.Sources
trait ConfigQuery {
def showVersion(configOptions: ConfigOptions): Boolean =
ConfigOptions.contains(ConfigOption.Version)(configOptions)
def batchMode(configOptions: ConfigOptions): Boolean =
ConfigOptions.contains(ConfigOption.BatchMode)(configOptions)
def ignoreUserOptions(configOptions: ConfigOptions): Boolean =
ConfigOptions.contains(ConfigOption.IgnoreUserOptions)(configOptions)
def ignoreGlobalOptions(configOptions: ConfigOptions): Boolean =
ConfigOptions.contains(ConfigOption.IgnoreGlobalOptions)(configOptions)
def sources(configOptions: ConfigOptions): Sources = {
val explicitPaths = configOptions.options.flatMap {
case ConfigOption.Source(sourcePath) => List(sourcePath)
case _ => List.empty
}
val paths = explicitPaths match {
case List() => List(Paths.get(System.getenv("PWD")))
case _ => explicitPaths
}
Sources(paths)
}
}
object ConfigQuery extends ConfigQuery

31
config/src/main/scala/net/kemitix/thorp/config/ConfigValidation.scala
View file

@ -1,31 +0,0 @@
package net.kemitix.thorp.config
import java.io.File
sealed trait ConfigValidation {
def errorMessage: String
}
object ConfigValidation {
case object SourceIsNotADirectory extends ConfigValidation {
override def errorMessage: String = "Source must be a directory"
}
case object SourceIsNotReadable extends ConfigValidation {
override def errorMessage: String = "Source must be readable"
}
case object BucketNameIsMissing extends ConfigValidation {
override def errorMessage: String = "Bucket name is missing"
}
final case class ErrorReadingFile(
file: File,
message: String
) extends ConfigValidation {
override def errorMessage: String = s"Error reading file '$file': $message"
}
}

5
config/src/main/scala/net/kemitix/thorp/config/ConfigValidationException.scala
View file

@ -1,5 +0,0 @@
package net.kemitix.thorp.config
final case class ConfigValidationException(
errors: Seq[ConfigValidation]
) extends Exception

56
config/src/main/scala/net/kemitix/thorp/config/ConfigValidator.scala
View file

@ -1,56 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Path
import net.kemitix.thorp.domain.{Bucket, Sources}
import zio.IO
sealed trait ConfigValidator {
def validateConfig(
config: Configuration
): IO[List[ConfigValidation], Configuration] = IO.fromEither {
for {
_ <- validateSources(config.sources)
_ <- validateBucket(config.bucket)
} yield config
}
def validateBucket(bucket: Bucket): Either[List[ConfigValidation], Bucket] =
if (bucket.name.isEmpty) Left(List(ConfigValidation.BucketNameIsMissing))
else Right(bucket)
def validateSources(
sources: Sources): Either[List[ConfigValidation], Sources] =
sources.paths.foldLeft(List[ConfigValidation]()) {
(acc: List[ConfigValidation], path) =>
{
validateSource(path) match {
case Left(errors) => acc ++ errors
case Right(_) => acc
}
}
} match {
case Nil => Right(sources)
case errors => Left(errors)
}
def validateSource(source: Path): Either[List[ConfigValidation], Path] =
for {
_ <- validateSourceIsDirectory(source)
_ <- validateSourceIsReadable(source)
} yield source
def validateSourceIsDirectory(
source: Path): Either[List[ConfigValidation], Path] =
if (source.toFile.isDirectory) Right(source)
else Left(List(ConfigValidation.SourceIsNotADirectory))
def validateSourceIsReadable(
source: Path): Either[List[ConfigValidation], Path] =
if (source.toFile.canRead) Right(source)
else Left(List(ConfigValidation.SourceIsNotReadable))
}
object ConfigValidator extends ConfigValidator

41
config/src/main/scala/net/kemitix/thorp/config/Configuration.scala
View file

@ -1,41 +0,0 @@
package net.kemitix.thorp.config
import net.kemitix.thorp.domain.{Bucket, Filter, RemoteKey, SimpleLens, Sources}
private[config] final case class Configuration(
bucket: Bucket,
prefix: RemoteKey,
filters: List[Filter],
debug: Boolean,
batchMode: Boolean,
parallel: Int,
sources: Sources
)
private[config] object Configuration {
val empty: Configuration = Configuration(
bucket = Bucket(""),
prefix = RemoteKey(""),
filters = List.empty,
debug = false,
batchMode = false,
parallel = 1,
sources = Sources(List.empty)
)
val sources: SimpleLens[Configuration, Sources] =
SimpleLens[Configuration, Sources](_.sources, b => a => b.copy(sources = a))
val bucket: SimpleLens[Configuration, Bucket] =
SimpleLens[Configuration, Bucket](_.bucket, b => a => b.copy(bucket = a))
val prefix: SimpleLens[Configuration, RemoteKey] =
SimpleLens[Configuration, RemoteKey](_.prefix, b => a => b.copy(prefix = a))
val filters: SimpleLens[Configuration, List[Filter]] =
SimpleLens[Configuration, List[Filter]](_.filters,
b => a => b.copy(filters = a))
val debug: SimpleLens[Configuration, Boolean] =
SimpleLens[Configuration, Boolean](_.debug, b => a => b.copy(debug = a))
val batchMode: SimpleLens[Configuration, Boolean] =
SimpleLens[Configuration, Boolean](_.batchMode,
b => a => b.copy(batchMode = a))
val parallel: SimpleLens[Configuration, Int] =
SimpleLens[Configuration, Int](_.parallel, b => a => b.copy(parallel = a))
}

51
config/src/main/scala/net/kemitix/thorp/config/ConfigurationBuilder.scala
View file

@ -1,51 +0,0 @@
package net.kemitix.thorp.config
import java.io.File
import net.kemitix.thorp.filesystem.FileSystem
import zio.ZIO
/**
* Builds a configuration from settings in a file within the
* `source` directory and from supplied configuration options.
*/
trait ConfigurationBuilder {
private val userConfigFile = ".config/thorp.conf"
private val globalConfig = new File("/etc/thorp.conf")
private val userHome = new File(System.getProperty("user.home"))
def buildConfig(priorityOpts: ConfigOptions)
: ZIO[FileSystem, ConfigValidationException, Configuration] =
(getConfigOptions(priorityOpts).map(collateOptions) >>=
ConfigValidator.validateConfig)
.catchAll(errors => ZIO.fail(ConfigValidationException(errors)))
private def getConfigOptions(priorityOpts: ConfigOptions) =
for {
sourceOpts <- SourceConfigLoader.loadSourceConfigs(
ConfigQuery.sources(priorityOpts))
userOpts <- userOptions(priorityOpts ++ sourceOpts)
globalOpts <- globalOptions(priorityOpts ++ sourceOpts ++ userOpts)
} yield priorityOpts ++ sourceOpts ++ userOpts ++ globalOpts
private val emptyConfig = ZIO.succeed(ConfigOptions.empty)
private def userOptions(priorityOpts: ConfigOptions) =
if (ConfigQuery.ignoreUserOptions(priorityOpts)) emptyConfig
else ParseConfigFile.parseFile(new File(userHome, userConfigFile))
private def globalOptions(priorityOpts: ConfigOptions) =
if (ConfigQuery.ignoreGlobalOptions(priorityOpts)) emptyConfig
else ParseConfigFile.parseFile(globalConfig)
private def collateOptions(configOptions: ConfigOptions): Configuration =
ConfigOptions.options
.get(configOptions)
.foldLeft(Configuration.empty) { (config, configOption) =>
configOption.update(config)
}
}
object ConfigurationBuilder extends ConfigurationBuilder

23
config/src/main/scala/net/kemitix/thorp/config/ParseConfigFile.scala
View file

@ -1,23 +0,0 @@
package net.kemitix.thorp.config
import java.io.File
import net.kemitix.thorp.filesystem.FileSystem
import zio.{IO, RIO, ZIO}
trait ParseConfigFile {
def parseFile(
file: File): ZIO[FileSystem, Seq[ConfigValidation], ConfigOptions] =
(FileSystem.exists(file) >>= readLines(file) >>= ParseConfigLines.parseLines)
.catchAll(h =>
IO.fail(List(ConfigValidation.ErrorReadingFile(file, h.getMessage))))
private def readLines(file: File)(
exists: Boolean): RIO[FileSystem, Seq[String]] =
if (exists) FileSystem.lines(file)
else ZIO.succeed(Seq.empty)
}
object ParseConfigFile extends ParseConfigFile

48
config/src/main/scala/net/kemitix/thorp/config/ParseConfigLines.scala
View file

@ -1,48 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Paths
import java.util.regex.Pattern
import net.kemitix.thorp.config.ConfigOption._
import zio.UIO
trait ParseConfigLines {
private val pattern = "^\\s*(?<key>\\S*)\\s*=\\s*(?<value>\\S*)\\s*$"
private val format = Pattern.compile(pattern)
def parseLines(lines: Seq[String]): UIO[ConfigOptions] =
UIO(ConfigOptions(lines.flatMap(parseLine).toList))
private def parseLine(str: String) =
format.matcher(str) match {
case m if m.matches => parseKeyValue(m.group("key"), m.group("value"))
case _ => List.empty
}
private def parseKeyValue(
key: String,
value: String
): List[ConfigOption] =
key.toLowerCase match {
case "parallel" => value.toIntOption.map(Parallel).toList
case "source" => List(Source(Paths.get(value)))
case "bucket" => List(Bucket(value))
case "prefix" => List(Prefix(value))
case "include" => List(Include(value))
case "exclude" => List(Exclude(value))
case "debug" => if (truthy(value)) List(Debug()) else List.empty
case _ => List.empty
}
private def truthy(value: String): Boolean =
value.toLowerCase match {
case "true" => true
case "yes" => true
case "enabled" => true
case _ => false
}
}
object ParseConfigLines extends ParseConfigLines

26
config/src/main/scala/net/kemitix/thorp/config/SourceConfigLoader.scala
View file

@ -1,26 +0,0 @@
package net.kemitix.thorp.config
import java.io.File
import net.kemitix.thorp.domain.Sources
import net.kemitix.thorp.filesystem.FileSystem
import zio.ZIO
trait SourceConfigLoader {
val thorpConfigFileName = ".thorp.conf"
def loadSourceConfigs(
sources: Sources): ZIO[FileSystem, Seq[ConfigValidation], ConfigOptions] =
ZIO
.foreach(sources.paths) { path =>
ParseConfigFile.parseFile(new File(path.toFile, thorpConfigFileName))
}
.map(_.foldLeft(ConfigOptions(sources.paths.map(ConfigOption.Source))) {
(acc, co) =>
acc ++ co
})
}
object SourceConfigLoader extends SourceConfigLoader

37
config/src/test/java/net/kemitix/thorp/config/ConfigOptionTest.java Normal file
View file

@ -0,0 +1,37 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Sources;
import net.kemitix.thorp.filesystem.TemporaryFolder;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.List;
public class ConfigOptionTest
implements TemporaryFolder, WithAssertions {
@Test
@DisplayName("when more the one source then preserve their order")
public void whenMultiSource_PreserveOrder() {
withDirectory(path1 -> {
withDirectory(path2 -> {
ConfigOptions configOptions = ConfigOptions.create(
Arrays.asList(
ConfigOption.source(path1),
ConfigOption.source(path2),
ConfigOption.bucket("bucket"),
ConfigOption.ignoreGlobalOptions(),
ConfigOption.ignoreUserOptions()
));
List<Path> expected = Arrays.asList(path1, path2);
assertThatCode(() -> {
Configuration result =
ConfigurationBuilder.buildConfig(configOptions);
assertThat(result.sources.paths()).isEqualTo(expected);
}).doesNotThrowAnyException();
});
});
}
}

141
config/src/test/java/net/kemitix/thorp/config/ConfigQueryTest.java Normal file
View file

@ -0,0 +1,141 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Sources;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class ConfigQueryTest
implements WithAssertions {
@Nested
@DisplayName("show version")
public class ShowVersionTest{
@Test
@DisplayName("when set then show")
public void whenSet_thenShow() {
assertThat(ConfigQuery.showVersion(
ConfigOptions.create(
Collections.singletonList(
ConfigOption.version()))
)).isTrue();
}
@Test
@DisplayName("when not set then do not show")
public void whenNotSet_thenDoNotShow() {
assertThat(ConfigQuery.showVersion(
ConfigOptions.create(
Collections.emptyList())
)).isFalse();
}
}
@Nested
@DisplayName("batch mode")
public class BatchModeTest{
@Test
@DisplayName("when set then show")
public void whenSet_thenShow() {
assertThat(ConfigQuery.batchMode(
ConfigOptions.create(
Collections.singletonList(
ConfigOption.batchMode()))
)).isTrue();
}
@Test
@DisplayName("when not set then do not show")
public void whenNotSet_thenDoNotShow() {
assertThat(ConfigQuery.batchMode(
ConfigOptions.create(
Collections.emptyList())
)).isFalse();
}
}
@Nested
@DisplayName("ignore user options")
public class IgnoreUserOptionsTest{
@Test
@DisplayName("when set then show")
public void whenSet_thenShow() {
assertThat(ConfigQuery.ignoreUserOptions(
ConfigOptions.create(
Collections.singletonList(
ConfigOption.ignoreUserOptions()))
)).isTrue();
}
@Test
@DisplayName("when not set then do not show")
public void whenNotSet_thenDoNotShow() {
assertThat(ConfigQuery.ignoreUserOptions(
ConfigOptions.create(
Collections.emptyList())
)).isFalse();
}
}
@Nested
@DisplayName("ignore global options")
public class IgnoreGlobalOptionsTest{
@Test
@DisplayName("when set then show")
public void whenSet_thenShow() {
assertThat(ConfigQuery.ignoreGlobalOptions(
ConfigOptions.create(
Collections.singletonList(
ConfigOption.ignoreGlobalOptions()))
)).isTrue();
}
@Test
@DisplayName("when not set then do not show")
public void whenNotSet_thenDoNotShow() {
assertThat(ConfigQuery.ignoreGlobalOptions(
ConfigOptions.create(
Collections.emptyList())
)).isFalse();
}
}
@Nested
@DisplayName("source")
public class SourcesTest {
Path pathA = Paths.get("a-path");
Path pathB = Paths.get("b-path");
@Test
@DisplayName("when not set then use current directory")
public void whenNoSet_thenCurrentDir() {
Sources expected = Sources.create(
Collections.singletonList(
Paths.get(
System.getenv("PWD")
)));
assertThat(ConfigQuery.sources(ConfigOptions.empty()))
.isEqualTo(expected);
}
@Test
@DisplayName("when one source then have one source")
public void whenOneSource_thenOneSource() {
List<Path> expected = Collections.singletonList(pathA);
assertThat(ConfigQuery.sources(
ConfigOptions.create(
Collections.singletonList(
ConfigOption.source(pathA)))).paths())
.isEqualTo(expected);
}
@Test
@DisplayName("when two sources then have two sources")
public void whenTwoSources_thenTwoSources() {
List<Path> expected = Arrays.asList(pathA, pathB);
assertThat(
ConfigQuery.sources(
ConfigOptions.create(
Arrays.asList(
ConfigOption.source(pathA),
ConfigOption.source(pathB))
)).paths())
.isEqualTo(expected);
}
}
}

214
config/src/test/java/net/kemitix/thorp/config/ConfigurationBuilderTest.java Normal file
View file

@ -0,0 +1,214 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.domain.Bucket;
import net.kemitix.thorp.domain.RemoteKey;
import net.kemitix.thorp.filesystem.TemporaryFolder;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class ConfigurationBuilderTest
implements WithAssertions {
Path pwd = Paths.get(System.getenv("PWD"));
Bucket aBucket = Bucket.named("aBucket");
ConfigOption coBucket = ConfigOption.bucket(aBucket.name());
String thorpConfigFileName = ".thorp.conf";
ConfigOptions configOptions(List<ConfigOption> options) {
List<ConfigOption> optionList = new ArrayList<>(options);
optionList.add(ConfigOption.ignoreUserOptions());
optionList.add(ConfigOption.ignoreGlobalOptions());
return ConfigOptions.create(optionList);
}
@Test
@DisplayName("when no source then user current directory")
public void whenNoSource_thenUseCurrentDir() throws IOException, ConfigValidationException {
Configuration result = ConfigurationBuilder.buildConfig(
configOptions(Collections.singletonList(coBucket)));
assertThat(result.sources.paths()).containsExactly(pwd);
}
@Nested
@DisplayName("default source")
public class DefaultSourceTests {
@Nested
@DisplayName("with .thorp.conf")
public class WithThorpConfTests implements TemporaryFolder {
@Test
@DisplayName("with settings")
public void WithSettingsTests() {
withDirectory(source -> {
//given
List<String> settings = Arrays.asList(
"bucket = a-bucket",
"prefix = a-prefix",
"include = an-inclusion",
"exclude = an-exclusion"
);
createFile(source, thorpConfigFileName, settings);
//when
Configuration result =
invoke(configOptions(Collections.singletonList(
ConfigOption.source(source))));
//then
assertThat(result.bucket).isEqualTo(Bucket.named("a-bucket"));
assertThat(result.prefix).isEqualTo(RemoteKey.create("a-prefix"));
assertThat(result.filters).hasSize(2)
.anySatisfy(filter ->
assertThat(filter.predicate()
.test("an-exclusion")).isTrue())
.anySatisfy(filter ->
assertThat(filter.predicate()
.test("an-inclusion")).isTrue());
});
}
}
}
@Nested
@DisplayName("single source")
public class SingleSourceTests implements TemporaryFolder {
@Test
@DisplayName("has single source")
public void hasSingleSource() {
withDirectory(aSource -> {
Configuration result =
invoke(
configOptions(Arrays.asList(
ConfigOption.source(aSource),
coBucket)));
assertThat(result.sources.paths()).containsExactly(aSource);
});
}
}
@Nested
@DisplayName("multiple sources")
public class MultipleSources implements TemporaryFolder {
@Test
@DisplayName("included in order")
public void hasBothSourcesInOrder() {
withDirectory(currentSource -> {
withDirectory(previousSource -> {
Configuration result =
invoke(configOptions(Arrays.asList(
ConfigOption.source(currentSource),
ConfigOption.source(previousSource),
coBucket)));
assertThat(result.sources.paths())
.containsExactly(
currentSource,
previousSource);
});
});
}
}
@Nested
@DisplayName("config file includes another source")
public class ConfigLinkedSourceTests implements TemporaryFolder {
@Test
@DisplayName("include the linked source")
public void configIncludeOtherSource() {
withDirectory(currentSource -> {
withDirectory(previousSource -> {
createFile(currentSource,
thorpConfigFileName,
Collections.singletonList(
"source = " + previousSource));
Configuration result = invoke(configOptions(Arrays.asList(
ConfigOption.source(currentSource),
coBucket)));
assertThat(result.sources.paths())
.containsExactly(
currentSource,
previousSource);
});
});
}
@Test
@DisplayName("when linked source has config file")
public void whenSettingsFileInBothSources() {
withDirectory(currentSource -> {
withDirectory(previousSource -> {
//given
createFile(currentSource,
thorpConfigFileName,
Arrays.asList(
"source = " + previousSource,
"bucket = current-bucket",
"prefix = current-prefix",
"include = current-include",
"exclude = current-exclude"));
createFile(previousSource,
thorpConfigFileName,
Arrays.asList(
"bucket = previous-bucket",
"prefix = previous-prefix",
"include = previous-include",
"exclude = previous-exclude"));
//when
Configuration result = invoke(configOptions(Arrays.asList(
ConfigOption.source(currentSource),
coBucket)));
//then
assertThat(result.sources.paths()).containsExactly(currentSource, previousSource);
assertThat(result.bucket.name()).isEqualTo("current-bucket");
assertThat(result.prefix.key()).isEqualTo("current-prefix");
assertThat(result.filters).anyMatch(filter -> filter.predicate().test("current-include"));
assertThat(result.filters).anyMatch(filter -> filter.predicate().test("current-exclude"));
assertThat(result.filters).noneMatch(filter -> filter.predicate().test("previous-include"));
assertThat(result.filters).noneMatch(filter -> filter.predicate().test("previous-exclude"));
});
});
}
}
@Nested
@DisplayName("linked source links to third source")
public class LinkedSourceLinkedSourceTests implements TemporaryFolder {
@Test
@DisplayName("ignore third source")
public void ignoreThirdSource() {
withDirectory(currentSource -> {
withDirectory(parentSource -> {
createFile(currentSource, thorpConfigFileName,
Collections.singletonList("source = " + parentSource));
withDirectory(grandParentSource -> {
createFile(parentSource, thorpConfigFileName,
Collections.singletonList("source = " + grandParentSource));
//when
Configuration result = invoke(configOptions(Arrays.asList(
ConfigOption.source(currentSource), coBucket)));
//then
assertThat(result.sources.paths())
.containsExactly(currentSource, parentSource)
.doesNotContain(grandParentSource);
});
});
});
}
}
@Test
@DisplayName("when batch mode option then batch mode in configuration")
public void whenBatchMode_thenBatchMode() {
Configuration result= invoke(configOptions(Arrays.asList(
ConfigOption.batchMode(),
coBucket)));
assertThat(result.batchMode).isTrue();
}
public Configuration invoke(ConfigOptions configOptions) {
try {
return ConfigurationBuilder.buildConfig(configOptions);
} catch (IOException | ConfigValidationException e) {
throw new RuntimeException(e);
}
}
}

63
config/src/test/java/net/kemitix/thorp/config/ParseConfigFileTest.java Normal file
View file

@ -0,0 +1,63 @@
package net.kemitix.thorp.config;
import net.kemitix.thorp.filesystem.TemporaryFolder;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Collections;
public class ParseConfigFileTest
implements WithAssertions, TemporaryFolder {
@Test
@DisplayName("when file is missing then no options")
public void whenFileMissing_thenNoOptions() throws IOException {
assertThat(invoke(new File("/path/to/missing/file")))
.isEqualTo(ConfigOptions.empty());
}
@Test
@DisplayName("when file is empty then no options")
public void whenEmptyFile_thenNoOptions() {
withDirectory(dir -> {
File file = createFile(dir, "empty-file", Collections.emptyList());
assertThat(invoke(file)).isEqualTo(ConfigOptions.empty());
});
}
@Test
@DisplayName("when no valid entried then no options")
public void whenNoValidEntries_thenNoOptions() {
withDirectory(dir -> {
File file = createFile(dir, "invalid-config",
Arrays.asList("no valid = config items", "invalid line"));
assertThat(invoke(file)).isEqualTo(ConfigOptions.empty());
});
}
@Test
@DisplayName("when file is valid then parse options")
public void whenValidFile_thenOptions() {
withDirectory(dir -> {
File file = createFile(dir, "simple-config", Arrays.asList(
"source = /path/to/source",
"bucket = bucket-name"));
assertThat(invoke(file)).isEqualTo(
ConfigOptions.create(
Arrays.asList(
ConfigOption.source(Paths.get("/path/to/source")),
ConfigOption.bucket("bucket-name"))));
});
}
ConfigOptions invoke(File file) {
try {
return ParseConfigFile.parseFile(file);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}

94
config/src/test/java/net/kemitix/thorp/config/ParseConfigLinesTest.java Normal file
View file

@ -0,0 +1,94 @@
package net.kemitix.thorp.config;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.List;
public class ParseConfigLinesTest
implements WithAssertions {
private final ParseConfigLines parser = new ParseConfigLines();
@Test
@DisplayName("source")
public void source() {
testParser("source = /path/to/source",
ConfigOption.source(Paths.get("/path/to/source")));
}
@Test
@DisplayName("bucket")
public void bucket() {
testParser("bucket = bucket-name",
ConfigOption.bucket("bucket-name"));
}
@Test
@DisplayName("prefix")
public void prefix() {
testParser("prefix = prefix/to/files",
ConfigOption.prefix("prefix/to/files"));
}
@Test
@DisplayName("include")
public void include() {
testParser("include = path/to/include",
ConfigOption.include("path/to/include"));
}
@Test
@DisplayName("exclude")
public void exclude() {
testParser("exclude = path/to/exclude",
ConfigOption.exclude("path/to/exclude"));
}
@Test
@DisplayName("parallel")
public void parallel() {
testParser("parallel = 3",
ConfigOption.parallel(3));
}
@Test
@DisplayName("parallel - invalid")
public void parallelInvalid() {
testParserIgnores("parallel = invalid");
}
@Test
@DisplayName("debug - true")
public void debugTrue() {
testParser("debug = true",
ConfigOption.debug());
}
@Test
@DisplayName("debug - false")
public void debugFalse() {
testParserIgnores("debug = false");
}
@Test
@DisplayName("comment")
public void comment() {
testParserIgnores("# ignore name");
}
@Test
@DisplayName("unrecognised option")
public void unrecognised() {
testParserIgnores("unsupported = option");
}
public void testParser(String line, ConfigOption configOption) {
assertThat(invoke(Collections.singletonList(line))).isEqualTo(
ConfigOptions.create(
Collections.singletonList(configOption)));
}
public void testParserIgnores(String line) {
assertThat(invoke(Collections.singletonList(line))).isEqualTo(
ConfigOptions.create(
Collections.emptyList()));
}
private ConfigOptions invoke(List<String> lines) {
return parser.parseLines(lines);
}
}

38
config/src/test/scala/net/kemitix/thorp/config/ConfigOptionTest.scala
View file

@ -1,38 +0,0 @@
package net.kemitix.thorp.config
import net.kemitix.thorp.domain.{Sources, TemporaryFolder}
import net.kemitix.thorp.filesystem.FileSystem
import org.scalatest.FunSpec
import zio.DefaultRuntime
class ConfigOptionTest extends FunSpec with TemporaryFolder {
describe("when more than one source") {
it("should preserve their order") {
withDirectory(path1 => {
withDirectory(path2 => {
val configOptions = ConfigOptions(
List[ConfigOption](
ConfigOption.Source(path1),
ConfigOption.Source(path2),
ConfigOption.Bucket("bucket"),
ConfigOption.IgnoreGlobalOptions,
ConfigOption.IgnoreUserOptions
))
val expected = Sources(List(path1, path2))
val result = invoke(configOptions)
assert(result.isRight, result)
assertResult(expected)(ConfigQuery.sources(configOptions))
})
})
}
}
private def invoke(configOptions: ConfigOptions) = {
new DefaultRuntime {}.unsafeRunSync {
ConfigurationBuilder
.buildConfig(configOptions)
.provide(FileSystem.Live)
}.toEither
}
}

100
config/src/test/scala/net/kemitix/thorp/config/ConfigQueryTest.scala
View file

@ -1,100 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Paths
import net.kemitix.thorp.domain.Sources
import org.scalatest.FreeSpec
class ConfigQueryTest extends FreeSpec {
"show version" - {
"when is set" - {
"should be true" in {
val result =
ConfigQuery.showVersion(ConfigOptions(List(ConfigOption.Version)))
assertResult(true)(result)
}
}
"when not set" - {
"should be false" in {
val result = ConfigQuery.showVersion(ConfigOptions(List()))
assertResult(false)(result)
}
}
}
"batch mode" - {
"when is set" - {
"should be true" in {
val result =
ConfigQuery.batchMode(ConfigOptions(List(ConfigOption.BatchMode)))
assertResult(true)(result)
}
}
"when not set" - {
"should be false" in {
val result = ConfigQuery.batchMode(ConfigOptions(List()))
assertResult(false)(result)
}
}
}
"ignore user options" - {
"when is set" - {
"should be true" in {
val result = ConfigQuery.ignoreUserOptions(
ConfigOptions(List(ConfigOption.IgnoreUserOptions)))
assertResult(true)(result)
}
}
"when not set" - {
"should be false" in {
val result = ConfigQuery.ignoreUserOptions(ConfigOptions(List()))
assertResult(false)(result)
}
}
}
"ignore global options" - {
"when is set" - {
"should be true" in {
val result = ConfigQuery.ignoreGlobalOptions(
ConfigOptions(List(ConfigOption.IgnoreGlobalOptions)))
assertResult(true)(result)
}
}
"when not set" - {
"should be false" in {
val result = ConfigQuery.ignoreGlobalOptions(ConfigOptions(List()))
assertResult(false)(result)
}
}
}
"sources" - {
val pathA = Paths.get("a-path")
val pathB = Paths.get("b-path")
"when not set" - {
"should have current dir" - {
val pwd = Paths.get(System.getenv("PWD"))
val expected = Sources(List(pwd))
val result = ConfigQuery.sources(ConfigOptions(List()))
assertResult(expected)(result)
}
}
"when is set once" - {
"should have one source" in {
val expected = Sources(List(pathA))
val result =
ConfigQuery.sources(ConfigOptions(List(ConfigOption.Source(pathA))))
assertResult(expected)(result)
}
}
"when is set twice" - {
"should have two sources" in {
val expected = Sources(List(pathA, pathB))
val result = ConfigQuery.sources(
ConfigOptions(
List(ConfigOption.Source(pathA), ConfigOption.Source(pathB))))
assertResult(expected)(result)
}
}
}
}

174
config/src/test/scala/net/kemitix/thorp/config/ConfigurationBuilderTest.scala
View file

@ -1,174 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.{Path, Paths}
import net.kemitix.thorp.domain.Filter.{Exclude, Include}
import net.kemitix.thorp.domain._
import net.kemitix.thorp.filesystem.FileSystem
import org.scalatest.FunSpec
import zio.DefaultRuntime
class ConfigurationBuilderTest extends FunSpec with TemporaryFolder {
private val pwd: Path = Paths.get(System.getenv("PWD"))
private val aBucket = Bucket("aBucket")
private val coBucket: ConfigOption.Bucket = ConfigOption.Bucket(aBucket.name)
private val thorpConfigFileName = ".thorp.conf"
private def configOptions(options: ConfigOption*): ConfigOptions =
ConfigOptions(
List[ConfigOption](
ConfigOption.IgnoreUserOptions,
ConfigOption.IgnoreGlobalOptions
) ++ options)
describe("when no source") {
it("should use the current (PWD) directory") {
val expected = Right(Sources(List(pwd)))
val options = configOptions(coBucket)
val result = invoke(options).map(_.sources)
assertResult(expected)(result)
}
}
describe("a source") {
describe("with .thorp.conf") {
describe("with settings") {
withDirectory(source => {
writeFile(source,
thorpConfigFileName,
"bucket = a-bucket",
"prefix = a-prefix",
"include = an-inclusion",
"exclude = an-exclusion")
val result = invoke(configOptions(ConfigOption.Source(source)))
it("should have bucket") {
val expected = Right(Bucket("a-bucket"))
assertResult(expected)(result.map(_.bucket))
}
it("should have prefix") {
val expected = Right(RemoteKey("a-prefix"))
assertResult(expected)(result.map(_.prefix))
}
it("should have filters") {
val expected =
Right(
List[Filter](Exclude("an-exclusion"), Include("an-inclusion")))
assertResult(expected)(result.map(_.filters))
}
})
}
}
}
describe("when has a single source with no .thorp.conf") {
it("should only include the source once") {
withDirectory(aSource => {
val expected = Right(Sources(List(aSource)))
val options = configOptions(ConfigOption.Source(aSource), coBucket)
val result = invoke(options).map(_.sources)
assertResult(expected)(result)
})
}
}
describe("when has two sources") {
it("should include both sources in order") {
withDirectory(currentSource => {
withDirectory(previousSource => {
val expected = Right(List(currentSource, previousSource))
val options = configOptions(ConfigOption.Source(currentSource),
ConfigOption.Source(previousSource),
coBucket)
val result = invoke(options).map(_.sources.paths)
assertResult(expected)(result)
})
})
}
}
describe("when current source has .thorp.conf with source to another") {
it("should include both sources in order") {
withDirectory(currentSource => {
withDirectory(previousSource => {
writeFile(currentSource,
thorpConfigFileName,
s"source = $previousSource")
val expected = Right(List(currentSource, previousSource))
val options =
configOptions(ConfigOption.Source(currentSource), coBucket)
val result = invoke(options).map(_.sources.paths)
assertResult(expected)(result)
})
})
}
describe("when settings are in current and previous") {
it("should include settings from only current") {
withDirectory(previousSource => {
withDirectory(currentSource => {
writeFile(
currentSource,
thorpConfigFileName,
s"source = $previousSource",
"bucket = current-bucket",
"prefix = current-prefix",
"include = current-include",
"exclude = current-exclude"
)
writeFile(previousSource,
thorpConfigFileName,
"bucket = previous-bucket",
"prefix = previous-prefix",
"include = previous-include",
"exclude = previous-exclude")
// should have both sources in order
val expectedSources =
Right(Sources(List(currentSource, previousSource)))
// should have bucket from current only
val expectedBuckets = Right(Bucket("current-bucket"))
// should have prefix from current only
val expectedPrefixes = Right(RemoteKey("current-prefix"))
// should have filters from both sources
val expectedFilters = Right(
List[Filter](Filter.Exclude("current-exclude"),
Filter.Include("current-include")))
val options = configOptions(ConfigOption.Source(currentSource))
val result = invoke(options)
assertResult(expectedSources)(result.map(_.sources))
assertResult(expectedBuckets)(result.map(_.bucket))
assertResult(expectedPrefixes)(result.map(_.prefix))
assertResult(expectedFilters)(result.map(_.filters))
})
})
}
}
}
describe(
"when source has thorp.config source to another source that does the same") {
it("should only include first two sources") {
withDirectory(currentSource => {
withDirectory(parentSource => {
writeFile(currentSource,
thorpConfigFileName,
s"source = $parentSource")
withDirectory(grandParentSource => {
writeFile(parentSource,
thorpConfigFileName,
s"source = $grandParentSource")
val expected = Right(List(currentSource, parentSource))
val options =
configOptions(ConfigOption.Source(currentSource), coBucket)
val result = invoke(options).map(_.sources.paths)
assertResult(expected)(result)
})
})
})
}
}
private def invoke(configOptions: ConfigOptions) = {
new DefaultRuntime {}.unsafeRunSync {
ConfigurationBuilder
.buildConfig(configOptions)
.provide(FileSystem.Live)
}.toEither
}
}

60
config/src/test/scala/net/kemitix/thorp/config/ParseConfigFileTest.scala
View file

@ -1,60 +0,0 @@
package net.kemitix.thorp.config
import java.io.File
import java.nio.file.Paths
import net.kemitix.thorp.domain.TemporaryFolder
import net.kemitix.thorp.filesystem.FileSystem
import org.scalatest.FunSpec
import zio.DefaultRuntime
class ParseConfigFileTest extends FunSpec with TemporaryFolder {
private val empty = Right(ConfigOptions.empty)
describe("parse a missing file") {
val file = new File("/path/to/missing/file")
it("should return no options") {
assertResult(empty)(invoke(file))
}
}
describe("parse an empty file") {
it("should return no options") {
withDirectory(dir => {
val file = createFile(dir, "empty-file")
assertResult(empty)(invoke(file))
})
}
}
describe("parse a file with no valid entries") {
it("should return no options") {
withDirectory(dir => {
val file = createFile(dir, "invalid-config", "no valid = config items")
assertResult(empty)(invoke(file))
})
}
}
describe("parse a file with properties") {
it("should return some options") {
val expected = Right(
ConfigOptions(
List[ConfigOption](ConfigOption.Source(Paths.get("/path/to/source")),
ConfigOption.Bucket("bucket-name"))))
withDirectory(dir => {
val file = createFile(dir,
"simple-config",
"source = /path/to/source",
"bucket = bucket-name")
assertResult(expected)(invoke(file))
})
}
}
private def invoke(file: File) = {
new DefaultRuntime {}.unsafeRunSync {
ParseConfigFile
.parseFile(file)
.provide(FileSystem.Live)
}.toEither
}
}

106
config/src/test/scala/net/kemitix/thorp/config/ParseConfigLinesTest.scala
View file

@ -1,106 +0,0 @@
package net.kemitix.thorp.config
import java.nio.file.Paths
import org.scalatest.FunSpec
import zio.DefaultRuntime
class ParseConfigLinesTest extends FunSpec {
describe("parse single lines") {
describe("source") {
it("should parse") {
val expected =
Right(
ConfigOptions(
List(ConfigOption.Source(Paths.get("/path/to/source")))))
val result = invoke(List("source = /path/to/source"))
assertResult(expected)(result)
}
}
describe("bucket") {
it("should parse") {
val expected =
Right(ConfigOptions(List(ConfigOption.Bucket("bucket-name"))))
val result = invoke(List("bucket = bucket-name"))
assertResult(expected)(result)
}
}
describe("prefix") {
it("should parse") {
val expected =
Right(ConfigOptions(List(ConfigOption.Prefix("prefix/to/files"))))
val result = invoke(List("prefix = prefix/to/files"))
assertResult(expected)(result)
}
}
describe("include") {
it("should parse") {
val expected =
Right(ConfigOptions(List(ConfigOption.Include("path/to/include"))))
val result = invoke(List("include = path/to/include"))
assertResult(expected)(result)
}
}
describe("exclude") {
it("should parse") {
val expected =
Right(ConfigOptions(List(ConfigOption.Exclude("path/to/exclude"))))
val result = invoke(List("exclude = path/to/exclude"))
assertResult(expected)(result)
}
}
describe("parallel") {
describe("when valid") {
it("should parse") {
val expected =
Right(ConfigOptions(List(ConfigOption.Parallel(3))))
val result = invoke(List("parallel = 3"))
assertResult(expected)(result)
}
}
describe("when invalid") {
it("should ignore") {
val expected =
Right(ConfigOptions(List.empty))
val result = invoke(List("parallel = invalid"))
assertResult(expected)(result)
}
}
}
describe("debug - true") {
it("should parse") {
val expected = Right(ConfigOptions(List(ConfigOption.Debug())))
val result = invoke(List("debug = true"))
assertResult(expected)(result)
}
}
describe("debug - false") {
it("should parse") {
val expected = Right(ConfigOptions.empty)
val result = invoke(List("debug = false"))
assertResult(expected)(result)
}
}
describe("comment line") {
it("should be ignored") {
val expected = Right(ConfigOptions.empty)
val result = invoke(List("# ignore me"))
assertResult(expected)(result)
}
}
describe("unrecognised option") {
it("should be ignored") {
val expected = Right(ConfigOptions.empty)
val result = invoke(List("unsupported = option"))
assertResult(expected)(result)
}
}
def invoke(lines: List[String]) = {
new DefaultRuntime {}.unsafeRunSync {
ParseConfigLines.parseLines(lines)
}.toEither
}
}
}

3
console/src/main/scala/net/kemitix/thorp/console/ConsoleOut.scala
View file

@ -1,5 +1,6 @@
package net.kemitix.thorp.console
import scala.jdk.CollectionConverters._
import net.kemitix.thorp.domain.StorageEvent.ActionSummary
import net.kemitix.thorp.domain.Terminal._
import net.kemitix.thorp.domain.{Bucket, RemoteKey, Sources}
@ -27,7 +28,7 @@ object ConsoleOut {
prefix: RemoteKey,
sources: Sources
) extends ConsoleOut {
private val sourcesList = sources.paths.mkString(", ")
private val sourcesList = sources.paths.asScala.mkString(", ")
override def en: String =
List(s"Bucket: ${bucket.name}",
s"Prefix: ${prefix.key}",

BIN
docs/images/reactor-graph.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 299 KiB

After

Width:  |  Height:  |  Size: 249 KiB

42
domain/pom.xml
View file

@ -1,6 +1,5 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>net.kemitix.thorp</groupId>
<artifactId>thorp-parent</artifactId>
@ -12,48 +11,29 @@
<name>domain</name>
<dependencies>
<!-- scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<!-- eip-zio -->
<!-- mon -->
<dependency>
<groupId>net.kemitix</groupId>
<artifactId>eip-zio_2.13</artifactId>
<artifactId>mon</artifactId>
</dependency>
<!-- zio -->
<!-- lombok -->
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio_2.13</artifactId>
</dependency>
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio-streams_2.13</artifactId>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- scala - testing -->
<!-- testing -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.13</artifactId>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalamock</groupId>
<artifactId>scalamock_2.13</artifactId>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

107
domain/src/main/java/net/kemitix/thorp/domain/Action.java Normal file
View file

@ -0,0 +1,107 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public abstract class Action {
public final Bucket bucket;
public final Long size;
public final RemoteKey remoteKey;
public abstract String asString();
public static DoNothing doNothing(
Bucket body,
RemoteKey remoteKey,
Long size
) {
return new DoNothing(body, size, remoteKey);
}
public static ToUpload toUpload(
Bucket body,
LocalFile localFile,
Long size
) {
return new ToUpload(body, localFile, size);
}
public static ToCopy toCopy(
Bucket body,
RemoteKey sourceKey,
MD5Hash hash,
RemoteKey targetKey,
Long size
) {
return new ToCopy(body, sourceKey, hash, targetKey, size);
}
public static ToDelete toDelete(
Bucket body,
RemoteKey remoteKey,
Long size
) {
return new ToDelete(body, size, remoteKey);
}
public static class DoNothing extends Action {
private DoNothing(
Bucket body,
Long size,
RemoteKey remoteKey
) {
super(body, size, remoteKey);
}
@Override
public String asString() {
return String.format("Do nothing: %s", remoteKey.key());
}
}
public static class ToUpload extends Action {
public final LocalFile localFile;
private ToUpload(
Bucket body,
LocalFile localFile,
Long size
) {
super(body, size, localFile.remoteKey);
this.localFile = localFile;
}
@Override
public String asString() {
return String.format("Upload: %s", localFile.remoteKey.key());
}
}
public static class ToCopy extends Action {
public final RemoteKey sourceKey;
public final MD5Hash hash;
public final RemoteKey targetKey;
private ToCopy(
Bucket body,
RemoteKey sourceKey,
MD5Hash hash,
RemoteKey targetKey,
Long size
) {
super(body, size, targetKey);
this.sourceKey = sourceKey;
this.hash = hash;
this.targetKey = targetKey;
}
@Override
public String asString() {
return String.format("Copy: %s => %s",
sourceKey.key(), targetKey.key());
}
}
public static class ToDelete extends Action {
private ToDelete(
Bucket body,
Long size,
RemoteKey remoteKey
) {
super(body, size, remoteKey);
}
@Override
public String asString() {
return String.format("Delete: %s", remoteKey.key());
}
}
}

15
domain/src/main/java/net/kemitix/thorp/domain/Bucket.java Normal file
View file

@ -0,0 +1,15 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
public class Bucket extends TypeAlias<String> {
private Bucket(String value) {
super(value);
}
public String name() {
return getValue();
}
public static Bucket named(String name) {
return new Bucket(name);
}
}

27
domain/src/main/java/net/kemitix/thorp/domain/Counters.java Normal file
View file

@ -0,0 +1,27 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.With;
@With
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class Counters {
public final int uploaded;
public final int deleted;
public final int copied;
public final int errors;
public static Counters empty = new Counters(0, 0, 0, 0);
public Counters incrementUploaded() {
return withUploaded(uploaded + 1);
}
public Counters incrementDeleted() {
return withDeleted(deleted + 1);
}
public Counters incrementCopied() {
return withCopied(copied + 1);
}
public Counters incrementErrors() {
return withErrors(errors + 1);
}
}

45
domain/src/main/java/net/kemitix/thorp/domain/Filter.java Normal file
View file

@ -0,0 +1,45 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import net.kemitix.mon.TypeAlias;
import java.util.function.Predicate;
import java.util.regex.Pattern;
public interface Filter {
static Include include(String include) {
return Include.create(include);
}
static Exclude exclude(String exclude) {
return Exclude.create(exclude);
}
Predicate<String> predicate();
class Include extends TypeAlias<Pattern> implements Filter {
private Include(Pattern value) {
super(value);
}
public static Include create(String include) {
return new Include(Pattern.compile(include));
}
public static Include all() {
return Include.create(".*");
}
@Override
public Predicate<String> predicate() {
return getValue().asPredicate();
}
}
class Exclude extends TypeAlias<Pattern> implements Filter {
private Exclude(Pattern value) {
super(value);
}
public static Exclude create(String exclude) {
return new Exclude(Pattern.compile(exclude));
}
@Override
public Predicate<String> predicate() {
return getValue().asPredicate();
}
}
}

44
domain/src/main/java/net/kemitix/thorp/domain/HashGenerator.java Normal file
View file

@ -0,0 +1,44 @@
package net.kemitix.thorp.domain;
import java.io.IOException;
import java.nio.file.Path;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.List;
import java.util.ServiceLoader;
public interface HashGenerator {
HashType hashType();
String label();
String hashFile(Path path) throws IOException, NoSuchAlgorithmException;
Hashes hash(Path path) throws IOException, NoSuchAlgorithmException;
MD5Hash hashChunk(Path path, Long index, long partSize) throws IOException, NoSuchAlgorithmException;
static List<HashGenerator> all() {
ServiceLoader<HashGenerator> hashGenerators = ServiceLoader.load(HashGenerator.class);
List<HashGenerator> list = new ArrayList<>();
for(HashGenerator hashGenerator: hashGenerators) {
list.add(hashGenerator);
}
return list;
}
static HashGenerator generatorFor(String label) {
return all()
.stream()
.filter(g -> g.label().equals(label))
.findFirst()
.orElseThrow(() -> new RuntimeException("Unknown hash type: " + label));
}
static HashType typeFrom(String label) {
return generatorFor(label).hashType();
}
static Hashes hashObject(Path path) throws IOException, NoSuchAlgorithmException {
List<Hashes> hashesList = new ArrayList<>();
for (HashGenerator hashGenerator : all()) {
hashesList.add(hashGenerator.hash(path));
}
return Hashes.mergeAll(hashesList);
}
}

12
domain/src/main/java/net/kemitix/thorp/domain/HashType.java Normal file
View file

@ -0,0 +1,12 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
@RequiredArgsConstructor(access = AccessLevel.PROTECTED)
public class HashType {
public final String label;
public static HashType MD5 = new HashType("MD5");
public static HashType DUMMY = new HashType("Dummy"); // testing only
}

48
domain/src/main/java/net/kemitix/thorp/domain/Hashes.java Normal file
View file

@ -0,0 +1,48 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
import java.util.*;
public class Hashes extends TypeAlias<Map<HashType, MD5Hash>> {
private Hashes() {
super(new HashMap<>());
}
public static Hashes create() {
return new Hashes();
}
public static Hashes create(HashType key, MD5Hash value) {
Hashes hashes = Hashes.create();
hashes.getValue().put(key, value);
return hashes;
}
public static Hashes mergeAll(List<Hashes> hashesList) {
Hashes hashes = Hashes.create();
Map<HashType, MD5Hash> values = hashes.getValue();
hashesList.stream().map(TypeAlias::getValue).forEach(values::putAll);
return hashes;
}
public Hashes withKeyValue(HashType key, MD5Hash value) {
Hashes hashes = Hashes.create();
hashes.getValue().putAll(getValue());
hashes.getValue().put(key, value);
return hashes;
}
public Set<HashType> keys() {
return getValue().keySet();
}
public Collection<MD5Hash> values() {
return getValue().values();
}
public Optional<MD5Hash> get(HashType key) {
return Optional.ofNullable(getValue().get(key));
}
public Hashes merge(Hashes other) {
Hashes hashes = Hashes.create();
hashes.getValue().putAll(getValue());
hashes.getValue().putAll(other.getValue());
return hashes;
}
}

38
domain/src/main/java/net/kemitix/thorp/domain/HexEncoder.java Normal file
View file

@ -0,0 +1,38 @@
package net.kemitix.thorp.domain;
import java.io.ByteArrayOutputStream;
import java.math.BigInteger;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class HexEncoder {
public static String encode(byte[] bytes) {
return String.format("%0" + (bytes.length << 1) + "x",
new BigInteger(1, bytes))
.toUpperCase();
}
public static byte[] decode(String hexString) {
ByteArrayOutputStream bytes =
new ByteArrayOutputStream(hexString.length() * 4);
List<String> hexBytes = Arrays.stream(hexString
.replaceAll("[^0-9A-Fa-f]", "")
.split("")).collect(Collectors.toList());
sliding(hexBytes, 2)
.map(hb -> String.join("", hb))
.mapToInt(hex -> Integer.parseInt(hex, 16))
.forEach(bytes::write);
return bytes.toByteArray();
}
public static <T> Stream<List<T>> sliding(List<T> list, int size) {
if(size > list.size())
return Stream.empty();
return IntStream.range(0, list.size()-size+1)
.filter(i -> i % size == 0)
.mapToObj(start -> list.subList(start, start+size));
}
}

17
domain/src/main/java/net/kemitix/thorp/domain/LastModified.java Normal file
View file

@ -0,0 +1,17 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
import java.time.Instant;
public class LastModified extends TypeAlias<Instant> {
private LastModified(Instant value) {
super(value);
}
public static LastModified at(Instant instant) {
return new LastModified(instant);
}
public Instant at() {
return getValue();
}
}

31
domain/src/main/java/net/kemitix/thorp/domain/LocalFile.java Normal file
View file

@ -0,0 +1,31 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import java.io.File;
import java.util.Optional;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class LocalFile {
public final File file;
public final File source;
public final Hashes hashes;
public final RemoteKey remoteKey;
public final Long length;
public static LocalFile create(
File file,
File source,
Hashes hashes,
RemoteKey remoteKey,
Long length
) {
return new LocalFile(file, source, hashes, remoteKey, length);
}
public boolean matchesHash(MD5Hash hash) {
return hashes.values().contains(hash);
}
public Optional<String> md5base64() {
return hashes.get(HashType.MD5).map(MD5Hash::hash64);
}
}

42
domain/src/main/java/net/kemitix/thorp/domain/MD5Hash.java Normal file
View file

@ -0,0 +1,42 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.LongStream;
import java.util.stream.Stream;
public class MD5Hash extends TypeAlias<String> {
private MD5Hash(String value) {
super(value);
}
public static MD5Hash create(String in) {
return new MD5Hash(in);
}
public static MD5Hash fromDigest(byte[] digest) {
return new MD5Hash(digestAsString(digest));
}
public static String digestAsString(byte[] digest) {
return IntStream.range(0, digest.length)
.map(i -> digest[i])
.mapToObj(b -> String.format("%02x", b))
.map(s -> s.substring(s.length() - 2, s.length()))
.flatMap(x -> Stream.of(x.split("")))
.collect(Collectors.joining());
}
public String hash() {
return QuoteStripper.stripQuotes(String.join("", getValue()));
}
public byte[] digest() {
return HexEncoder.decode(hash());
}
public String hash64() {
return Base64.getEncoder().encodeToString(digest());
}
}

37
domain/src/main/java/net/kemitix/thorp/domain/MD5HashData.java Normal file
View file

@ -0,0 +1,37 @@
package net.kemitix.thorp.domain;
public interface MD5HashData {
class Root {
public static final String hashString = "a3a6ac11a0eb577b81b3bb5c95cc8a6e";
public static final MD5Hash hash = MD5Hash.create(hashString);
public static final String base64 = "o6asEaDrV3uBs7tclcyKbg==";
public static final RemoteKey remoteKey = RemoteKey.create("root-file");
public static final Long size = 55L;
}
class Leaf {
public static final String hashString = "208386a650bdec61cfcd7bd8dcb6b542";
public static final MD5Hash hash = MD5Hash.create(hashString);
public static final String base64 = "IIOGplC97GHPzXvY3La1Qg==";
public static final RemoteKey remoteKey = RemoteKey.create("subdir/leaf-file");
public static final Long size = 58L;
}
class BigFile {
public static final String hashString = "b1ab1f7680138e6db7309200584e35d8";
public static final MD5Hash hash = MD5Hash.create(hashString);
public static class Part1 {
public static final int offset = 0;
public static final int size = 1048576;
public static final String hashString = "39d4a9c78b9cfddf6d241a201a4ab726";
public static final MD5Hash hash = MD5Hash.create(hashString);
}
public static class Part2 {
public static final int offset = 1048576;
public static final int size = 1048576;
public static final String hashString = "af5876f3a3bc6e66f4ae96bb93d8dae0";
public static final MD5Hash hash = MD5Hash.create(hashString);
}
}
}

40
domain/src/main/java/net/kemitix/thorp/domain/MapView.java Normal file
View file

@ -0,0 +1,40 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import java.util.function.BiFunction;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class MapView<K, V> {
private final Map<K, V> map;
public static <K, V> MapView<K, V> empty(){
return MapView.of(new HashMap<>());
}
public static <K, V> MapView<K, V> of(Map<K, V> map) {
return new MapView<>(map);
}
public boolean contains(K key) {
return map.containsKey(key);
}
public Optional<V> get(K key) {
return Optional.ofNullable(map.get(key));
}
public Collection<K> keys() { return map.keySet(); }
public Optional<Tuple<K, V>> collectFirst(BiFunction<K, V, Boolean> test) {
return map.entrySet().stream()
.filter(e -> test.apply(e.getKey(), e.getValue()))
.findFirst()
.map(e -> Tuple.create(e.getKey(), e.getValue()));
}
public Map<K, V> asMap() {
return new HashMap<>(map);
}
public int size() {
return map.size();
}
}

12
domain/src/main/java/net/kemitix/thorp/domain/QuoteStripper.java Normal file
View file

@ -0,0 +1,12 @@
package net.kemitix.thorp.domain;
import java.util.Arrays;
import java.util.stream.Collectors;
public interface QuoteStripper {
static String stripQuotes(String in) {
return Arrays.stream(in.split(""))
.filter(c -> !c.equals("\""))
.collect(Collectors.joining());
}
}

49
domain/src/main/java/net/kemitix/thorp/domain/RemoteKey.java Normal file
View file

@ -0,0 +1,49 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class RemoteKey extends TypeAlias<String> {
private RemoteKey(String value) {
super(value);
}
public static RemoteKey create(String key) {
return new RemoteKey(key);
}
public String key() {
return getValue();
}
public Optional<File> asFile(Path source, RemoteKey prefix) {
if (key().length() == 0 || !key().startsWith(prefix.key())) {
return Optional.empty();
}
return Optional.of(
source.resolve(relativeTo(prefix))
.toFile());
}
public Path relativeTo(RemoteKey prefix) {
if (prefix.key().equals("")) {
return Paths.get(key());
}
return Paths.get(prefix.key()).relativize(Paths.get(key()));
}
public RemoteKey resolve(String path) {
return RemoteKey.create(
Stream.of(key(), path)
.filter(s -> !s.isEmpty())
.collect(Collectors.joining("/")));
}
public static RemoteKey fromSourcePath(Path source, Path path) {
return RemoteKey.create(
source.relativize(path).toString());
}
public static RemoteKey from(Path source, RemoteKey prefix, File file) {
return prefix.resolve(source.relativize(file.toPath()).toString());
}
}

37
domain/src/main/java/net/kemitix/thorp/domain/RemoteObjects.java Normal file
View file

@ -0,0 +1,37 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import java.util.Map;
import java.util.Optional;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class RemoteObjects {
public final MapView<MD5Hash, RemoteKey> byHash;
public final MapView<RemoteKey, MD5Hash> byKey;
public static final RemoteObjects empty =
new RemoteObjects(MapView.empty(), MapView.empty());
public static RemoteObjects create(
Map<MD5Hash, RemoteKey> byHash,
Map<RemoteKey, MD5Hash> byKey
) {
return new RemoteObjects(
MapView.of(byHash),
MapView.of(byKey)
);
}
public boolean remoteKeyExists(RemoteKey remoteKey) {
return byKey.contains(remoteKey);
}
public boolean remoteMatchesLocalFile(LocalFile localFile) {
return byKey.get(localFile.remoteKey)
.map(localFile::matchesHash)
.orElse(false);
}
public Optional<Tuple<RemoteKey, MD5Hash>> remoteHasHash(Hashes hashes) {
return byHash.collectFirst(
(hash, key) -> hashes.values().contains(hash))
.map(Tuple::swap);
}
}

20
domain/src/main/java/net/kemitix/thorp/domain/SizeTranslation.java Normal file
View file

@ -0,0 +1,20 @@
package net.kemitix.thorp.domain;
public class SizeTranslation {
static long kbLimit = 10240L;
static long mbLimit = kbLimit * 1024;
static long gbLimit = mbLimit * 1024;
public static String sizeInEnglish(long length) {
double bytes = length;
if (length > gbLimit) {
return String.format("%.3fGb", bytes / 1024 / 1024 / 1024);
}
if (length > mbLimit) {
return String.format("%.2fMb", bytes / 1024 / 1024);
}
if (length > kbLimit) {
return String.format("%.0fKb", bytes / 1024);
}
return String.format("%db", length);
}
}

36
domain/src/main/java/net/kemitix/thorp/domain/Sources.java Normal file
View file

@ -0,0 +1,36 @@
package net.kemitix.thorp.domain;
import net.kemitix.mon.TypeAlias;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
public class Sources extends TypeAlias<List<Path>> {
private Sources(List<Path> value) { super(value); }
public static final Sources emptySources = new Sources(Collections.emptyList());
public static Sources create(List<Path> paths) {
return new Sources(paths);
}
public List<Path> paths() {
return new ArrayList<>(getValue());
}
public Path forPath(Path path) {
return getValue().stream()
.filter(path::startsWith)
.findFirst()
.orElseThrow(() ->
new RuntimeException(
"Path is not within any known source"));
}
public Sources append(Path path) {
return append(Collections.singletonList(path));
}
public Sources append(List<Path> paths) {
List<Path> collected = new ArrayList<>();
collected.addAll(getValue());
collected.addAll(paths);
return Sources.create(collected);
}
}

101
domain/src/main/java/net/kemitix/thorp/domain/StorageEvent.java Normal file
View file

@ -0,0 +1,101 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
public class StorageEvent {
public static DoNothingEvent doNothingEvent(RemoteKey remoteKey) {
return new DoNothingEvent(remoteKey);
}
public static StorageEvent copyEvent(RemoteKey sourceKey, RemoteKey targetKey) {
return new CopyEvent(sourceKey, targetKey);
}
public static UploadEvent uploadEvent(RemoteKey remoteKey, MD5Hash md5Hash) {
return new UploadEvent(remoteKey, md5Hash);
}
public static DeleteEvent deleteEvent(RemoteKey remoteKey) {
return new DeleteEvent(remoteKey);
}
public static ErrorEvent errorEvent(ActionSummary action, RemoteKey remoteKey, Throwable e) {
return new ErrorEvent(action, remoteKey, e);
}
public static ShutdownEvent shutdownEvent() {
return new ShutdownEvent();
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class DoNothingEvent extends StorageEvent {
public final RemoteKey remoteKey;
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class CopyEvent extends StorageEvent {
public final RemoteKey sourceKey;
public final RemoteKey targetKey;
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class UploadEvent extends StorageEvent {
public final RemoteKey remoteKey;
public final MD5Hash md5Hash;
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class DeleteEvent extends StorageEvent {
public final RemoteKey remoteKey;
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public static class ErrorEvent extends StorageEvent {
public final ActionSummary action;
public final RemoteKey remoteKey;
public final Throwable e;
}
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public static class ShutdownEvent extends StorageEvent {}
public interface ActionSummary {
String name();
String keys();
static Copy copy(String keys) {
return new Copy(keys);
}
static Upload upload(String keys) {
return new Upload(keys);
}
static Delete delete(String keys) {
return new Delete(keys);
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
class Copy implements ActionSummary {
public final String keys;
@Override
public String name() {
return "Copy";
}
@Override
public String keys() {
return keys;
}
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
class Upload implements ActionSummary {
public final String keys;
@Override
public String name() {
return "Upload";
}
@Override
public String keys() {
return keys;
}
}
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
class Delete implements ActionSummary {
public final String keys;
@Override
public String name() {
return "Delete";
}
@Override
public String keys() {
return keys;
}
}
}
}

200
domain/src/main/java/net/kemitix/thorp/domain/Terminal.java Normal file
View file

@ -0,0 +1,200 @@
package net.kemitix.thorp.domain;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import java.util.stream.IntStream;
public class Terminal {
public static String esc = "\u001B";
public static String csi = esc + "[";
/**
* Clear from cursor to end of screen.
*/
public static String eraseToEndOfScreen = csi + "0J";
/**
* Clear from cursor to beginning of screen.
*/
public static String eraseToStartOfScreen = csi + "1J";
/**
* Clear screen and move cursor to top-left.
*
* On DOS the "2J" command also moves to 1,1, so we force that behaviour for all.
*/
public static String eraseScreen = csi + "2J" + cursorPosition(1, 1);
/**
* Clear screen and scrollback buffer then move cursor to top-left.
*
* Anticipate that same DOS behaviour here, and to maintain consistency with {@link #eraseScreen}.
*/
public static String eraseScreenAndBuffer = csi + "3J";
/**
* Clears the terminal line to the right of the cursor.
*
* Does not move the cursor.
*/
public static String eraseLineForward = csi + "0K";
/**
* Clears the terminal line to the left of the cursor.
*
* Does not move the cursor.
*/
public static String eraseLineBack = csi + "1K";
/**
* Clears the whole terminal line.
*
* Does not move the cursor.
*/
public static String eraseLine = csi + "2K";
/**
* Saves the cursor position/state.
*/
public static String saveCursorPosition = csi + "s";
/**
* Restores the cursor position/state.
*/
public static String restoreCursorPosition = csi + "u";
public static String enableAlternateBuffer = csi + "?1049h";
public static String disableAlternateBuffer = csi + "?1049l";
private static Map<Integer, String> getSubBars() {
Map<Integer, String> subBars = new HashMap<>();
subBars.put(0, " ");
subBars.put(1, "▏");
subBars.put(2, "▎");
subBars.put(3, "▍");
subBars.put(4, "▌");
subBars.put(5, "▋");
subBars.put(6, "▊");
subBars.put(7, "▉");
return subBars;
}
/**
* Move the cursor up, default 1 line.
*
* Stops at the edge of the screen.
*/
public static String cursorUp(int lines) {
return csi + lines + "A";
}
/**
* Move the cursor down, default 1 line.
*
* Stops at the edge of the screen.
*/
public static String cursorDown(int lines) {
return csi + lines + "B";
}
/**
* Move the cursor forward, default 1 column.
*
* Stops at the edge of the screen.
*/
public static String cursorForward(int cols) {
return csi + cols + "C";
}
/**
* Move the cursor back, default 1 column,
*
* Stops at the edge of the screen.
*/
public static String cursorBack(int cols) {
return csi + cols + "D";
}
/**
* Move the cursor to the beginning of the line, default 1, down.
*/
public static String cursorNextLine(int lines) {
return csi + lines + "E";
}
/**
* Move the cursor to the beginning of the line, default 1, up.
*/
public static String cursorPrevLine(int lines) {
return csi + lines + "F";
}
/**
* Move the cursor to the column on the current line.
*/
public static String cursorHorizAbs(int col) {
return csi + col + "G";
}
/**
* Move the cursor to the position on screen (1,1 is the top-left).
*/
public static String cursorPosition(int row, int col) {
return csi + row + ";" + col + "H";
}
/**
* Scroll page up, default 1, lines.
*/
public static String scrollUp(int lines) {
return csi + lines + "S";
}
/**
* Scroll page down, default 1, lines.
*/
public static String scrollDown(int lines) {
return csi + lines + "T";
}
/**
* The Width of the terminal, as reported by the COLUMNS environment variable.
*
* N.B. Not all environment will update this value when the terminal is resized.
*
* @return the number of columns in the terminal
*/
public static int width() {
return Optional.ofNullable(System.getenv("COLUMNS"))
.map(Integer::parseInt)
.map(x -> Math.max(x, 10))
.orElse(80);
}
public static String progressBar(
double pos,
double max,
int width
) {
Map<Integer, String> subBars = getSubBars();
int barWidth = width - 2;
int phases = subBars.values().size();
int pxWidth = barWidth * phases;
double ratio = pos / max;
int pxDone = (int) (ratio * pxWidth);
int fullHeadSize = pxDone / phases;
int part = pxDone % phases;
String partial = part != 0 ? subBars.getOrDefault(part, "") : "";
String head = repeat("█", fullHeadSize) + partial;
int tailSize = barWidth - head.length();
String tail = repeat(" ", tailSize);
return "[" + head + tail + "]";
}
private static String repeat(String s, int times) {
StringBuilder sb = new StringBuilder();
IntStream.range(0, times).forEach(x -> sb.append(s));
return sb.toString();
}
}

16
domain/src/main/java/net/kemitix/thorp/domain/Tuple.java Normal file
View file

@ -0,0 +1,16 @@
package net.kemitix.thorp.domain;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class Tuple<A, B> {
public final A a;
public final B b;
public static <A, B> Tuple<A, B> create(A a, B b) {
return new Tuple<>(a, b);
}
public Tuple<B, A> swap() {
return Tuple.create(b, a);
}
}

40
domain/src/main/scala/net/kemitix/thorp/domain/Action.scala
View file

@ -1,40 +0,0 @@
package net.kemitix.thorp.domain
sealed trait Action {
def bucket: Bucket
def size: Long
def remoteKey: RemoteKey
}
object Action {
final case class DoNothing(
bucket: Bucket,
remoteKey: RemoteKey,
size: Long
) extends Action
final case class ToUpload(
bucket: Bucket,
localFile: LocalFile,
size: Long
) extends Action {
override def remoteKey: RemoteKey = localFile.remoteKey
}
final case class ToCopy(
bucket: Bucket,
sourceKey: RemoteKey,
hash: MD5Hash,
targetKey: RemoteKey,
size: Long
) extends Action {
override def remoteKey: RemoteKey = targetKey
}
final case class ToDelete(
bucket: Bucket,
remoteKey: RemoteKey,
size: Long
) extends Action
}

5
domain/src/main/scala/net/kemitix/thorp/domain/Bucket.scala
View file

@ -1,5 +0,0 @@
package net.kemitix.thorp.domain
final case class Bucket(
name: String
)

20
domain/src/main/scala/net/kemitix/thorp/domain/Counters.scala
View file

@ -1,20 +0,0 @@
package net.kemitix.thorp.domain
final case class Counters(
uploaded: Int,
deleted: Int,
copied: Int,
errors: Int
)
object Counters {
val empty: Counters = Counters(0, 0, 0, 0)
val uploaded: SimpleLens[Counters, Int] =
SimpleLens[Counters, Int](_.uploaded, b => a => b.copy(uploaded = a))
val deleted: SimpleLens[Counters, Int] =
SimpleLens[Counters, Int](_.deleted, b => a => b.copy(deleted = a))
val copied: SimpleLens[Counters, Int] =
SimpleLens[Counters, Int](_.copied, b => a => b.copy(copied = a))
val errors: SimpleLens[Counters, Int] =
SimpleLens[Counters, Int](_.errors, b => a => b.copy(errors = a))
}

21
domain/src/main/scala/net/kemitix/thorp/domain/Filter.scala
View file

@ -1,21 +0,0 @@
package net.kemitix.thorp.domain
import java.util.function.Predicate
import java.util.regex.Pattern
sealed trait Filter {
def predicate: Predicate[String]
}
object Filter {
final case class Include(include: String) extends Filter {
lazy val predicate: Predicate[String] = Pattern.compile(include).asPredicate
}
object Include {
def all: Include = Include(".*")
}
final case class Exclude(exclude: String) extends Filter {
lazy val predicate: Predicate[String] =
Pattern.compile(exclude).asPredicate()
}
}

7
domain/src/main/scala/net/kemitix/thorp/domain/HashType.scala
View file

@ -1,7 +0,0 @@
package net.kemitix.thorp.domain
trait HashType
object HashType {
case object MD5 extends HashType
}

23
domain/src/main/scala/net/kemitix/thorp/domain/HexEncoder.scala
View file

@ -1,23 +0,0 @@
package net.kemitix.thorp.domain
import java.math.BigInteger
trait HexEncoder {
def encode(bytes: Array[Byte]): String =
String
.format(s"%0${bytes.length << 1}x", new BigInteger(1, bytes))
.toUpperCase
def decode(hexString: String): Array[Byte] =
hexString
.replaceAll("[^0-9A-Fa-f]", "")
.toSeq
.sliding(2, 2)
.map(_.unwrap)
.toArray
.map(Integer.parseInt(_, 16).toByte)
}
object HexEncoder extends HexEncoder

11
domain/src/main/scala/net/kemitix/thorp/domain/Implicits.scala
View file

@ -1,11 +0,0 @@
package net.kemitix.thorp.domain
object Implicits {
@SuppressWarnings(Array("org.wartremover.warts.Equals"))
implicit final class AnyOps[A](self: A) {
def ===(other: A): Boolean = self == other
def =/=(other: A): Boolean = self != other
}
}

24
domain/src/main/scala/net/kemitix/thorp/domain/LocalFile.scala
View file

@ -1,24 +0,0 @@
package net.kemitix.thorp.domain
import java.io.File
import net.kemitix.thorp.domain.HashType.MD5
import net.kemitix.thorp.domain.Implicits._
final case class LocalFile private (
file: File,
source: File,
hashes: Hashes,
remoteKey: RemoteKey,
length: Long
)
object LocalFile {
val remoteKey: SimpleLens[LocalFile, RemoteKey] =
SimpleLens[LocalFile, RemoteKey](_.remoteKey,
b => a => b.copy(remoteKey = a))
def matchesHash(localFile: LocalFile)(other: MD5Hash): Boolean =
localFile.hashes.values.exists(other === _)
def md5base64(localFile: LocalFile): Option[String] =
localFile.hashes.get(MD5).map(MD5Hash.hash64)
}

17
domain/src/main/scala/net/kemitix/thorp/domain/MD5Hash.scala
View file

@ -1,17 +0,0 @@
package net.kemitix.thorp.domain
import java.util.Base64
import net.kemitix.thorp.domain.QuoteStripper.stripQuotes
final case class MD5Hash(in: String)
object MD5Hash {
def fromDigest(digest: Array[Byte]): MD5Hash =
MD5Hash((digest map ("%02x" format _)).mkString)
def hash(md5Hash: MD5Hash): String = md5Hash.in.filter(stripQuotes)
def digest(md5Hash: MD5Hash): Array[Byte] =
HexEncoder.decode(MD5Hash.hash(md5Hash))
def hash64(md5Hash: MD5Hash): String =
Base64.getEncoder.encodeToString(MD5Hash.digest(md5Hash))
}

31
domain/src/main/scala/net/kemitix/thorp/domain/MD5HashData.scala
View file

@ -1,31 +0,0 @@
package net.kemitix.thorp.domain
object MD5HashData {
object Root {
val hash: MD5Hash = MD5Hash("a3a6ac11a0eb577b81b3bb5c95cc8a6e")
val base64: String = "o6asEaDrV3uBs7tclcyKbg=="
val remoteKey = RemoteKey("root-file")
val size: Long = 55
}
object Leaf {
val hash: MD5Hash = MD5Hash("208386a650bdec61cfcd7bd8dcb6b542")
val base64: String = "IIOGplC97GHPzXvY3La1Qg=="
val remoteKey = RemoteKey("subdir/leaf-file")
val size: Long = 58
}
object BigFile {
val hash: MD5Hash = MD5Hash("b1ab1f7680138e6db7309200584e35d8")
object Part1 {
val offset: Int = 0
val size: Int = 1048576
val hash: MD5Hash = MD5Hash("39d4a9c78b9cfddf6d241a201a4ab726")
}
object Part2 {
val offset: Int = 1048576
val size: Int = 1048576
val hash: MD5Hash = MD5Hash("af5876f3a3bc6e66f4ae96bb93d8dae0")
}
}
}

9
domain/src/main/scala/net/kemitix/thorp/domain/QuoteStripper.scala
View file

@ -1,9 +0,0 @@
package net.kemitix.thorp.domain
import Implicits._
object QuoteStripper {
def stripQuotes: Char => Boolean = _ =/= '"'
}

35
domain/src/main/scala/net/kemitix/thorp/domain/RemoteKey.scala
View file

@ -1,35 +0,0 @@
package net.kemitix.thorp.domain
import java.io.File
import java.nio.file.{Path, Paths}
import Implicits._
import zio.UIO
final case class RemoteKey(key: String)
object RemoteKey {
val key: SimpleLens[RemoteKey, String] =
SimpleLens[RemoteKey, String](_.key, b => a => b.copy(key = a))
def asFile(source: Path, prefix: RemoteKey)(
remoteKey: RemoteKey): Option[File] =
if (remoteKey.key.length === 0 || !remoteKey.key.startsWith(prefix.key))
None
else Some(source.resolve(RemoteKey.relativeTo(prefix)(remoteKey)).toFile)
def relativeTo(prefix: RemoteKey)(remoteKey: RemoteKey): Path = prefix match {
case RemoteKey("") => Paths.get(remoteKey.key)
case _ => Paths.get(prefix.key).relativize(Paths.get(remoteKey.key))
}
def resolve(path: String)(remoteKey: RemoteKey): RemoteKey =
RemoteKey(List(remoteKey.key, path).filterNot(_.isEmpty).mkString("/"))
def fromSourcePath(source: Path, path: Path): RemoteKey =
RemoteKey(source.relativize(path).toString)
def from(source: Path, prefix: RemoteKey, file: File): UIO[RemoteKey] =
UIO(RemoteKey.resolve(source.relativize(file.toPath).toString)(prefix))
}

45
domain/src/main/scala/net/kemitix/thorp/domain/RemoteObjects.scala
View file

@ -1,45 +0,0 @@
package net.kemitix.thorp.domain
import zio.UIO
import scala.collection.MapView
/**
* A list of objects and their MD5 hash values.
*/
final case class RemoteObjects private (
byHash: MapView[MD5Hash, RemoteKey],
byKey: MapView[RemoteKey, MD5Hash]
)
object RemoteObjects {
val empty: RemoteObjects = RemoteObjects(MapView.empty, MapView.empty)
def create(byHash: MapView[MD5Hash, RemoteKey],
byKey: MapView[RemoteKey, MD5Hash]): RemoteObjects =
RemoteObjects(byHash, byKey)
def remoteKeyExists(
remoteObjects: RemoteObjects,
remoteKey: RemoteKey
): UIO[Boolean] = UIO(remoteObjects.byKey.contains(remoteKey))
def remoteMatchesLocalFile(
remoteObjects: RemoteObjects,
localFile: LocalFile
): UIO[Boolean] =
UIO(
remoteObjects.byKey
.get(localFile.remoteKey)
.exists(LocalFile.matchesHash(localFile)))
def remoteHasHash(
remoteObjects: RemoteObjects,
hashes: Hashes
): UIO[Option[(RemoteKey, MD5Hash)]] =
UIO(remoteObjects.byHash.collectFirst {
case (hash, key) if (hashes.values.exists(h => h == hash)) => (key, hash)
})
}

18
domain/src/main/scala/net/kemitix/thorp/domain/SimpleLens.scala
View file

@ -1,18 +0,0 @@
package net.kemitix.thorp.domain
final case class SimpleLens[A, B](field: A => B, update: A => B => A) {
def composeLens[C](other: SimpleLens[B, C]): SimpleLens[A, C] =
SimpleLens[A, C](
a => other.field(field(a)),
a => c => update(a)(other.update(field(a))(c))
)
def ^|->[C](other: SimpleLens[B, C]): SimpleLens[A, C] = composeLens(other)
def set(b: B)(a: A): A = update(a)(b)
def get(a: A): B = field(a)
def modify(f: B => B)(a: A): A = update(a)(f(field(a)))
}

17
domain/src/main/scala/net/kemitix/thorp/domain/SizeTranslation.scala
View file

@ -1,17 +0,0 @@
package net.kemitix.thorp.domain
object SizeTranslation {
val kbLimit: Long = 10240L
val mbLimit: Long = kbLimit * 1024
val gbLimit: Long = mbLimit * 1024
def sizeInEnglish(length: Long): String =
length.toDouble match {
case bytes if bytes > gbLimit => f"${bytes / 1024 / 1024 / 1024}%.3fGb"
case bytes if bytes > mbLimit => f"${bytes / 1024 / 1024}%.2fMb"
case bytes if bytes > kbLimit => f"${bytes / 1024}%.0fKb"
case bytes => s"${length}b"
}
}

41
domain/src/main/scala/net/kemitix/thorp/domain/Sources.scala
View file

@ -1,41 +0,0 @@
package net.kemitix.thorp.domain
import java.nio.file.Path
import zio.{UIO, ZIO}
/**
* The paths to synchronise with target.
*
* The first source path takes priority over those later in the list,
* etc. Where there is any file with the same relative path within
* more than one source, the file in the first listed path is
* uploaded, and the others are ignored.
*
* A path should only occur once in paths.
*/
final case class Sources(paths: List[Path]) {
def +(path: Path): Sources = this ++ List(path)
def ++(otherPaths: List[Path]): Sources =
Sources(
otherPaths.foldLeft(paths)(
(acc, path) =>
if (acc contains path) acc
else acc ++ List(path)
)
)
}
object Sources {
val emptySources: Sources = Sources(List.empty)
/**
* Returns the source path for the given path.
*/
def forPath(path: Path)(sources: Sources): UIO[Path] =
ZIO
.fromOption(sources.paths.find(s => path.startsWith(s)))
.orDieWith { _ =>
new RuntimeException("Path is not within any known source")
}
}

49
domain/src/main/scala/net/kemitix/thorp/domain/StorageEvent.scala
View file

@ -1,49 +0,0 @@
package net.kemitix.thorp.domain
sealed trait StorageEvent
object StorageEvent {
final case class DoNothingEvent(
remoteKey: RemoteKey
) extends StorageEvent
final case class CopyEvent(
sourceKey: RemoteKey,
targetKey: RemoteKey
) extends StorageEvent
final case class UploadEvent(
remoteKey: RemoteKey,
md5Hash: MD5Hash
) extends StorageEvent
final case class DeleteEvent(
remoteKey: RemoteKey
) extends StorageEvent
final case class ErrorEvent(
action: ActionSummary,
remoteKey: RemoteKey,
e: Throwable
) extends StorageEvent
final case class ShutdownEvent() extends StorageEvent
sealed trait ActionSummary {
val name: String
val keys: String
}
object ActionSummary {
final case class Copy(keys: String) extends ActionSummary {
override val name: String = "Copy"
}
final case class Upload(keys: String) extends ActionSummary {
override val name: String = "Upload"
}
final case class Delete(keys: String) extends ActionSummary {
override val name: String = "Delete"
}
}
}

50
domain/src/main/scala/net/kemitix/thorp/domain/TemporaryFolder.scala
View file

@ -1,50 +0,0 @@
package net.kemitix.thorp.domain
import java.io.{File, IOException, PrintWriter}
import java.nio.file.attribute.BasicFileAttributes
import java.nio.file.{FileVisitResult, Files, Path, SimpleFileVisitor}
import scala.util.Try
trait TemporaryFolder {
@SuppressWarnings(Array("org.wartremover.warts.TryPartial"))
def withDirectory(testCode: Path => Any): Unit = {
val dir: Path = Files.createTempDirectory("thorp-temp")
val t = Try(testCode(dir))
remove(dir)
t.get
()
}
def remove(root: Path): Unit = {
Files.walkFileTree(
root,
new SimpleFileVisitor[Path] {
override def visitFile(file: Path,
attrs: BasicFileAttributes): FileVisitResult = {
Files.delete(file)
FileVisitResult.CONTINUE
}
override def postVisitDirectory(dir: Path,
exc: IOException): FileVisitResult = {
Files.delete(dir)
FileVisitResult.CONTINUE
}
}
)
}
def createFile(directory: Path, name: String, contents: String*): File = {
val _ = directory.toFile.mkdirs
val file = directory.resolve(name).toFile
val writer = new PrintWriter(file, "UTF-8")
contents.foreach(writer.println)
writer.close()
file
}
def writeFile(directory: Path, name: String, contents: String*): Unit =
createFile(directory, name, contents: _*)
}

166
domain/src/main/scala/net/kemitix/thorp/domain/Terminal.scala
View file

@ -1,166 +0,0 @@
package net.kemitix.thorp.domain
import Implicits._
object Terminal {
val esc: String = "\u001B"
val csi: String = esc + "["
/**
* Clear from cursor to end of screen.
*/
val eraseToEndOfScreen: String = csi + "0J"
/**
* Clear from cursor to beginning of screen.
*/
val eraseToStartOfScreen: String = csi + "1J"
/**
* Clear screen and move cursor to top-left.
*
* On DOS the "2J" command also moves to 1,1, so we force that behaviour for all.
*/
val eraseScreen: String = csi + "2J" + cursorPosition(1, 1)
/**
* Clear screen and scrollback buffer then move cursor to top-left.
*
* Anticipate that same DOS behaviour here, and to maintain consistency with {@link #eraseScreen}.
*/
val eraseScreenAndBuffer: String = csi + "3J"
/**
* Clears the terminal line to the right of the cursor.
*
* Does not move the cursor.
*/
val eraseLineForward: String = csi + "0K"
/**
* Clears the terminal line to the left of the cursor.
*
* Does not move the cursor.
*/
val eraseLineBack: String = csi + "1K"
/**
* Clears the whole terminal line.
*
* Does not move the cursor.
*/
val eraseLine: String = csi + "2K"
/**
* Saves the cursor position/state.
*/
val saveCursorPosition: String = csi + "s"
/**
* Restores the cursor position/state.
*/
val restoreCursorPosition: String = csi + "u"
val enableAlternateBuffer: String = csi + "?1049h"
val disableAlternateBuffer: String = csi + "?1049l"
private val subBars = Map(0 -> " ",
1 -> "▏",
2 -> "▎",
3 -> "▍",
4 -> "▌",
5 -> "▋",
6 -> "▊",
7 -> "▉")
/**
* Move the cursor up, default 1 line.
*
* Stops at the edge of the screen.
*/
def cursorUp(lines: Int): String = s"${csi}${lines}A"
/**
* Move the cursor down, default 1 line.
*
* Stops at the edge of the screen.
*/
def cursorDown(lines: Int): String = s"${csi}${lines}B"
/**
* Move the cursor forward, default 1 column.
*
* Stops at the edge of the screen.
*/
def cursorForward(cols: Int): String = s"${csi}${cols}C"
/**
* Move the cursor back, default 1 column,
*
* Stops at the edge of the screen.
*/
def cursorBack(cols: Int): String = s"${csi}${cols}D"
/**
* Move the cursor to the beginning of the line, default 1, down.
*/
def cursorNextLine(lines: Int): String = s"${csi}${lines}E"
/**
* Move the cursor to the beginning of the line, default 1, up.
*/
def cursorPrevLine(lines: Int): String = s"${csi}${lines}F"
/**
* Move the cursor to the column on the current line.
*/
def cursorHorizAbs(col: Int): String = s"${csi}${col}G"
/**
* Move the cursor to the position on screen (1,1 is the top-left).
*/
def cursorPosition(row: Int, col: Int): String = s"${csi}${row};${col}H"
/**
* Scroll page up, default 1, lines.
*/
def scrollUp(lines: Int): String = s"${csi}${lines}S"
/**
* Scroll page down, default 1, lines.
*/
def scrollDown(lines: Int): String = s"${csi}${lines}T"
/**
* The Width of the terminal, as reported by the COLUMNS environment variable.
*
* N.B. Not all environment will update this value when the terminal is resized.
*
* @return the number of columns in the terminal
*/
def width: Int = {
Option(System.getenv("COLUMNS"))
.map(_.toInt)
.map(Math.max(_, 10))
.getOrElse(80)
}
def progressBar(
pos: Double,
max: Double,
width: Int
): String = {
val barWidth = width - 2
val phases = subBars.values.size
val pxWidth = barWidth * phases
val ratio = pos / max
val pxDone = pxWidth * ratio
val fullHeadSize: Int = (pxDone / phases).toInt
val part = (pxDone % phases).toInt
val partial = if (part =/= 0) subBars.getOrElse(part, "") else ""
val head = ("█" * fullHeadSize) + partial
val tailSize = barWidth - head.length
val tail = " " * tailSize
s"[$head$tail]"
}
}

8
domain/src/main/scala/net/kemitix/thorp/domain/package.scala
View file

@ -1,8 +0,0 @@
package net.kemitix.thorp
import java.time.Instant
package object domain {
type Hashes = Map[HashType, MD5Hash]
type LastModified = Instant
}

33
domain/src/test/java/net/kemitix/thorp/domain/HashesTest.java Normal file
View file

@ -0,0 +1,33 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Nested;
import java.util.Arrays;
public class HashesTest
implements WithAssertions {
@Nested
@DisplayName("mergeAll")
public class MergeAll {
@Test
@DisplayName("")
public void mergeAll() {
//given
HashType key1 = HashType.MD5;
HashType key2 = HashType.DUMMY;
MD5Hash value1 = MD5Hash.create("1");
MD5Hash value2 = MD5Hash.create("2");
Hashes hashes1 = Hashes.create(key1, value1);
Hashes hashes2 = Hashes.create(key2, value2);
//when
Hashes result = Hashes.mergeAll(Arrays.asList(hashes1,hashes2));
//then
assertThat(result.keys()).containsExactlyInAnyOrder(key1, key2);
assertThat(result.values()).containsExactlyInAnyOrder(value1, value2);
}
}
}

29
domain/src/test/java/net/kemitix/thorp/domain/HexEncoderTest.java Normal file
View file

@ -0,0 +1,29 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.nio.charset.StandardCharsets;
public class HexEncoderTest
implements WithAssertions {
private String text = "test text to encode to hex";
private String hex = "74657374207465787420746F20656E636F646520746F20686578";
@Test
@DisplayName("can round trip a hash decode then encode")
public void roundTripDecodeEncode() {
String result = HexEncoder.encode(HexEncoder.decode(hex));
assertThat(result).isEqualTo(hex);
}
@Test
@DisplayName("can round trip a hash encode then decode")
public void roundTripEncodeDecode() {
byte[] input = hex.getBytes(StandardCharsets.UTF_8);
byte[] result = HexEncoder.decode(HexEncoder.encode(input));
assertThat(result).isEqualTo(input);
}
}

30
domain/src/test/java/net/kemitix/thorp/domain/MD5HashTest.java Normal file
View file

@ -0,0 +1,30 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
public class MD5HashTest
implements WithAssertions {
@Test
@DisplayName("recover base64 hash")
public void recoverBase64Hash() {
assertThat(MD5HashData.Root.hash.hash64())
.isEqualTo(MD5HashData.Root.base64);
assertThat(MD5HashData.Leaf.hash.hash64())
.isEqualTo(MD5HashData.Leaf.base64);
}
@Test
@DisplayName("hash() strips quotes")
public void hashStripsQuotes() {
//given
String dQuote = "\"";
MD5Hash md5Hash = MD5Hash.create(dQuote + MD5HashData.Root.hashString + dQuote);
//when
String result = md5Hash.hash();
//then
assertThat(result).isEqualTo(MD5HashData.Root.hashString);
}
}

201
domain/src/test/java/net/kemitix/thorp/domain/RemoteKeyTest.java Normal file
View file

@ -0,0 +1,201 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Optional;
public class RemoteKeyTest
implements WithAssertions {
private RemoteKey emptyKey = RemoteKey.create("");
@Nested
@DisplayName("Create a RemoteKey")
public class CreateRemoteKey {
@Nested
@DisplayName("resolve()")
public class ResolvePath {
@Test
@DisplayName("key is empty")
public void keyIsEmpty() {
//given
RemoteKey expected = RemoteKey.create("path");
RemoteKey key = emptyKey;
//when
RemoteKey result = key.resolve("path");
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("path is empty")
public void pathIsEmpty() {
//given
RemoteKey expected = RemoteKey.create("key");
RemoteKey key = RemoteKey.create("key");
String path = "";
//when
RemoteKey result = key.resolve(path);
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("key and path are empty")
public void keyAndPathEmpty() {
//given
RemoteKey expected = RemoteKeyTest.this.emptyKey;
String path = "";
RemoteKey key = emptyKey;
//when
RemoteKey result = key.resolve(path);
//then
assertThat(result).isEqualTo(expected);
}
}
@Nested
@DisplayName("asFile()")
public class AsFile {
@Test
@DisplayName("key and prefix are non-empty")
public void keyAndPrefixNonEmpty() {
//given
Optional<File> expected = Optional.of(new File("source/key"));
RemoteKey key = RemoteKey.create("prefix/key");
Path source = Paths.get("source");
RemoteKey prefix = RemoteKey.create("prefix");
//when
Optional<File> result = key.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("prefix is empty")
public void prefixEmpty() {
//given
Optional<File> expected = Optional.of(new File("source/key"));
RemoteKey key = RemoteKey.create("key");
Path source = Paths.get("source");
RemoteKey prefix = emptyKey;
//when
Optional<File> result = key.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("key is empty")
public void keyEmpty() {
//given
Optional<File> expected = Optional.empty();
RemoteKey key = emptyKey;
Path source = Paths.get("source");
RemoteKey prefix = RemoteKey.create("source/key");
//when
Optional<File> result = key.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("key and prefix are empty")
public void keyAndPrefixEmpty() {
//given
Optional<File> expected = Optional.empty();
RemoteKey key = emptyKey;
Path source = Paths.get("source");
RemoteKey prefix = emptyKey;
//when
Optional<File> result = key.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
}
@Nested
@DisplayName("fromSourcePath()")
public class FromSourcePath {
@Test
@DisplayName("path is in source")
public void pathInSource() {
//given
RemoteKey expected = RemoteKey.create("child");
Path source = Paths.get("/source");
Path path = source.resolve("/source/child");
//when
RemoteKey result = RemoteKey.fromSourcePath(source, path);
//then
assertThat(result).isEqualTo(expected);
}
}
@Nested
@DisplayName("from(source, prefix, file)")
public class FromSourcePrefixFile {
@Test
@DisplayName("file in source")
public void fileInSource() {
//given
RemoteKey expected = RemoteKey.create("prefix/dir/filename");
Path source = Paths.get("/source");
RemoteKey prefix = RemoteKey.create("prefix");
File file = new File("/source/dir/filename");
//when
RemoteKey result = RemoteKey.from(source, prefix, file);
//then
assertThat(result).isEqualTo(expected);
}
}
}
@Nested
@DisplayName("asFile()")
public class AsFile {
@Test
@DisplayName("remoteKey is empty")
public void remoteKeyEmpty() {
//given
Optional<File> expected = Optional.empty();
Path source = Paths.get("/source");
RemoteKey prefix = RemoteKey.create("prefix");
RemoteKey remoteKey = emptyKey;
//when
Optional<File> result = remoteKey.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
@Nested
@DisplayName("remoteKey is not empty")
public class RemoteKeyNotEmpty {
@Test
@DisplayName("remoteKey is within prefix")
public void remoteKeyWithinPrefix() {
//given
Optional<File> expected = Optional.of(new File("/source/key"));
Path source = Paths.get("/source");
RemoteKey prefix = RemoteKey.create("prefix");
RemoteKey remoteKey = RemoteKey.create("prefix/key");
//when
Optional<File> result = remoteKey.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
@Test
@DisplayName("remoteKey is outwith prefix")
public void remoteKeyIsOutwithPrefix() {
//given
Optional<File> expected = Optional.empty();
Path source = Paths.get("/source");
RemoteKey prefix = RemoteKey.create("prefix");
RemoteKey remoteKey = RemoteKey.create("elsewhere/key");
//when
Optional<File> result = remoteKey.asFile(source, prefix);
//then
assertThat(result).isEqualTo(expected);
}
}
}
}

53
domain/src/test/java/net/kemitix/thorp/domain/SizeTranslationTest.java Normal file
View file

@ -0,0 +1,53 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
public class SizeTranslationTest
implements WithAssertions {
@Nested
@DisplayName("sizeInEnglish()")
public class SizeInEnglish {
@Test
@DisplayName("when size is less the 1Kb")
public void sizeLessThan1Kb() {
//should be in bytes
assertThat(SizeTranslation.sizeInEnglish(512))
.isEqualTo("512b");
}
@Test
@DisplayName("when size is a less than 10Kb")
public void sizeLessThan10Kb() {
//should still be in bytes
assertThat(SizeTranslation.sizeInEnglish(2000))
.isEqualTo("2000b");
}
@Test
@DisplayName("when size is over 10Kb and less than 10Mb")
public void sizeBetween10KbAnd10Mb() {
//should be in Kb with zero decimal places
assertThat(SizeTranslation.sizeInEnglish(5599232))
.isEqualTo("5468Kb");
}
@Test
@DisplayName("when size is over 10Mb and less than 10Gb")
public void sizeBetween10Mb10Gb() {
//should be in Mb with two decimal place
assertThat(SizeTranslation.sizeInEnglish(5733789833L))
.isEqualTo("5468.17Mb");
}
@Test@DisplayName("when size is over 10Gb")
public void sizeOver10Gb() {
//should be in Gb with three decimal place
assertThat(SizeTranslation.sizeInEnglish(5871400857278L))
.isEqualTo("5468.168Gb");
}
}
}

81
domain/src/test/java/net/kemitix/thorp/domain/TerminalTest.java Normal file
View file

@ -0,0 +1,81 @@
package net.kemitix.thorp.domain;
import org.assertj.core.api.WithAssertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
public class TerminalTest
implements WithAssertions {
@Nested
@DisplayName("progressBar()")
public class ProgressBar {
@Test
@DisplayName("width 10 - 0%")
public void width10at0() {
String bar = Terminal.progressBar(0d, 10d, 12);
assertThat(bar).isEqualTo("[ ]");
}
@Test
@DisplayName("width 10 - 10%")
public void width10at10() {
String bar = Terminal.progressBar(1d, 10d, 12);
assertThat(bar).isEqualTo("[█ ]");
}
@Test
@DisplayName("width 10 - 50%")
public void width10at50() {
String bar = Terminal.progressBar(5d, 10d, 12);
assertThat(bar).isEqualTo("[█████ ]");
}
@Test
@DisplayName("width 1 - 8/8th")
public void width8of8() {
String bar = Terminal.progressBar(8d, 8d, 3);
assertThat(bar).isEqualTo("[█]");
}
@Test
@DisplayName("width 1 - 7/8th")
public void width7of8() {
String bar = Terminal.progressBar(7d, 8d, 3);
assertThat(bar).isEqualTo("[▉]");
}
@Test
@DisplayName("width 1 - 6/8th")
public void width6of8() {
String bar = Terminal.progressBar(6d, 8d, 3);
assertThat(bar).isEqualTo("[▊]");
}
@Test
@DisplayName("width 1 - 5/8th")
public void width5of8() {
String bar = Terminal.progressBar(5d, 8d, 3);
assertThat(bar).isEqualTo("[▋]");
}
@Test
@DisplayName("width 1 - 4/8th")
public void width4of8() {
String bar = Terminal.progressBar(4d, 8d, 3);
assertThat(bar).isEqualTo("[▌]");
}
@Test
@DisplayName("width 1 - 3/8th")
public void width3of8() {
String bar = Terminal.progressBar(3d, 8d, 3);
assertThat(bar).isEqualTo("[▍]");
}
@Test
@DisplayName("width 1 - 2/8th")
public void width2of8() {
String bar = Terminal.progressBar(2d, 8d, 3);
assertThat(bar).isEqualTo("[▎]");
}
@Test
@DisplayName("width 1 - 1/8th")
public void width1of8() {
String bar = Terminal.progressBar(1d, 8d, 3);
assertThat(bar).isEqualTo("[▏]");
}
}
}

23
domain/src/test/scala/net/kemitix/thorp/domain/HexEncoderTest.scala
View file

@ -1,23 +0,0 @@
package net.kemitix.thorp.domain
import java.nio.charset.StandardCharsets
import org.scalatest.FreeSpec
class HexEncoderTest extends FreeSpec {
val text = "test text to encode to hex"
val hex = "74657374207465787420746F20656E636F646520746F20686578"
"can round trip a hash decode then encode" in {
val input = hex
val result = HexEncoder.encode(HexEncoder.decode(input))
assertResult(input)(result)
}
"can round trip a hash encode then decode" in {
val input = hex.getBytes(StandardCharsets.UTF_8)
val result = HexEncoder.decode(HexEncoder.encode(input))
assertResult(input)(result)
}
}

17
domain/src/test/scala/net/kemitix/thorp/domain/MD5HashTest.scala
View file

@ -1,17 +0,0 @@
package net.kemitix.thorp.domain
import org.scalatest.FunSpec
class MD5HashTest extends FunSpec {
describe("recover base64 hash") {
it("should recover base 64 #1") {
val rootHash = MD5HashData.Root.hash
assertResult(MD5HashData.Root.base64)(MD5Hash.hash64(rootHash))
}
it("should recover base 64 #2") {
val leafHash = MD5HashData.Leaf.hash
assertResult(MD5HashData.Leaf.base64)(MD5Hash.hash64(leafHash))
}
}
}

130
domain/src/test/scala/net/kemitix/thorp/domain/RemoteKeyTest.scala
View file

@ -1,130 +0,0 @@
package net.kemitix.thorp.domain
import java.io.File
import java.nio.file.Paths
import org.scalatest.FreeSpec
import zio.DefaultRuntime
class RemoteKeyTest extends FreeSpec {
private val emptyKey = RemoteKey("")
"create a RemoteKey" - {
"can resolve a path" - {
"when key is empty" in {
val key = emptyKey
val path = "path"
val expected = RemoteKey("path")
val result = RemoteKey.resolve(path)(key)
assertResult(expected)(result)
}
"when path is empty" in {
val key = RemoteKey("key")
val path = ""
val expected = RemoteKey("key")
val result = RemoteKey.resolve(path)(key)
assertResult(expected)(result)
}
"when key and path are empty" in {
val key = emptyKey
val path = ""
val expected = emptyKey
val result = RemoteKey.resolve(path)(key)
assertResult(expected)(result)
}
}
"asFile" - {
"when key and prefix are non-empty" in {
val key = RemoteKey("prefix/key")
val source = Paths.get("source")
val prefix = RemoteKey("prefix")
val expected = Some(new File("source/key"))
val result = RemoteKey.asFile(source, prefix)(key)
assertResult(expected)(result)
}
"when prefix is empty" in {
val key = RemoteKey("key")
val source = Paths.get("source")
val prefix = emptyKey
val expected = Some(new File("source/key"))
val result = RemoteKey.asFile(source, prefix)(key)
assertResult(expected)(result)
}
"when key is empty" in {
val key = emptyKey
val source = Paths.get("source")
val prefix = RemoteKey("prefix")
val expected = None
val result = RemoteKey.asFile(source, prefix)(key)
assertResult(expected)(result)
}
"when key and prefix are empty" in {
val key = emptyKey
val source = Paths.get("source")
val prefix = emptyKey
val expected = None
val result = RemoteKey.asFile(source, prefix)(key)
assertResult(expected)(result)
}
}
"fromSourcePath" - {
"when path in source" in {
val source = Paths.get("/source")
val path = source.resolve("/source/child")
val expected = RemoteKey("child")
val result = RemoteKey.fromSourcePath(source, path)
assertResult(expected)(result)
}
}
"from source, prefix, file" - {
"when file in source" in {
val source = Paths.get("/source")
val prefix = RemoteKey("prefix")
val file = new File("/source/dir/filename")
val expected = RemoteKey("prefix/dir/filename")
val program = RemoteKey.from(source, prefix, file)
val result = new DefaultRuntime {}.unsafeRunSync(program).toEither
assertResult(Right(expected))(result)
}
}
}
"asFile" - {
"remoteKey is empty" in {
val source = Paths.get("/source")
val prefix = RemoteKey("prefix")
val remoteKey = RemoteKey("")
val expected = None
val result = RemoteKey.asFile(source, prefix)(remoteKey)
assertResult(expected)(result)
}
"remoteKey is not empty" - {
"remoteKey is within prefix" in {
val source = Paths.get("/source")
val prefix = RemoteKey("prefix")
val remoteKey = RemoteKey("prefix/key")
val expected = Some(Paths.get("/source/key").toFile)
val result = RemoteKey.asFile(source, prefix)(remoteKey)
assertResult(expected)(result)
}
"remoteKey is outwith prefix" in {
val source = Paths.get("/source")
val prefix = RemoteKey("prefix")
val remoteKey = RemoteKey("elsewhere/key")
val expected = None
val result = RemoteKey.asFile(source, prefix)(remoteKey)
assertResult(expected)(result)
}
}
}
}

64
domain/src/test/scala/net/kemitix/thorp/domain/SimpleLensTest.scala
View file

@ -1,64 +0,0 @@
package net.kemitix.thorp.domain
import org.scalatest.FreeSpec
class SimpleLensTest extends FreeSpec {
"lens" - {
val subject = Subject(0, "s")
"modify" in {
val expected = Subject(1, "s")
val result = Subject.anIntLens.modify(_ + 1)(subject)
assertResult(expected)(result)
}
"get" in {
val expected = "s"
val result = Subject.aStringLens.get(subject)
assertResult(expected)(result)
}
"set" in {
val expected = Subject(0, "k")
val result = Subject.aStringLens.set("k")(subject)
assertResult(expected)(result)
}
}
"lens composed" - {
val wrapper = Wrapper(1, Subject(2, "x"))
val subjectStringLens = Wrapper.aSubjectLens ^|-> Subject.aStringLens
"modify" in {
val expected = Wrapper(1, Subject(2, "X"))
val result = subjectStringLens.modify(_.toUpperCase)(wrapper)
assertResult(expected)(result)
}
"get" in {
val expected = "x"
val result = subjectStringLens.get(wrapper)
assertResult(expected)(result)
}
"set" in {
val expected = Wrapper(1, Subject(2, "k"))
val result = subjectStringLens.set("k")(wrapper)
assertResult(expected)(result)
}
}
case class Subject(anInt: Int, aString: String)
object Subject {
val anIntLens: SimpleLens[Subject, Int] =
SimpleLens[Subject, Int](_.anInt, subject => i => subject.copy(anInt = i))
val aStringLens: SimpleLens[Subject, String] =
SimpleLens[Subject, String](_.aString,
subject => str => subject.copy(aString = str))
}
case class Wrapper(anInt: Int, aSubject: Subject)
object Wrapper {
val anIntLens: SimpleLens[Wrapper, Int] =
SimpleLens[Wrapper, Int](_.anInt, wrapper => i => wrapper.copy(anInt = i))
val aSubjectLens: SimpleLens[Wrapper, Subject] =
SimpleLens[Wrapper, Subject](
_.aSubject,
wrapper => subject => wrapper.copy(aSubject = subject))
}
}

36
domain/src/test/scala/net/kemitix/thorp/domain/SizeTranslationTest.scala
View file

@ -1,36 +0,0 @@
package net.kemitix.thorp.domain
import org.scalatest.FunSpec
class SizeTranslationTest extends FunSpec {
describe("sizeInEnglish") {
describe("when size is less the 1Kb") {
it("should in in bytes") {
assertResult("512b")(SizeTranslation.sizeInEnglish(512))
}
}
describe("when size is a less than 10Kb") {
it("should still be in bytes") {
assertResult("2000b")(SizeTranslation.sizeInEnglish(2000))
}
}
describe("when size is over 10Kb and less than 10Mb") {
it("should be in Kb with zero decimal places") {
assertResult("5468Kb")(SizeTranslation.sizeInEnglish(5599232))
}
}
describe("when size is over 10Mb and less than 10Gb") {
it("should be in Mb with two decimal place") {
assertResult("5468.17Mb")(SizeTranslation.sizeInEnglish(5733789833L))
}
}
describe("when size is over 10Gb") {
it("should be in Gb with three decimal place") {
assertResult("5468.168Gb")(
SizeTranslation.sizeInEnglish(5871400857278L))
}
}
}
}

69
domain/src/test/scala/net/kemitix/thorp/domain/TerminalTest.scala
View file

@ -1,69 +0,0 @@
package net.kemitix.thorp.domain
import org.scalatest.FunSpec
class TerminalTest extends FunSpec {
describe("progressBar") {
describe("width 10 - 0%") {
it("should match") {
val bar = Terminal.progressBar(0d, 10d, 12)
assertResult("[ ]")(bar)
}
}
describe("width 10 - 10%") {
it("should match") {
val bar = Terminal.progressBar(1d, 10d, 12)
assertResult("[█ ]")(bar)
}
}
describe("width 1 - 8/8th") {
it("should match") {
val bar = Terminal.progressBar(8d, 8d, 3)
assertResult("[█]")(bar)
}
}
describe("width 1 - 7/8th") {
it("should match") {
val bar = Terminal.progressBar(7d, 8d, 3)
assertResult("[▉]")(bar)
}
}
describe("width 1 - 6/8th") {
it("should match") {
val bar = Terminal.progressBar(6d, 8d, 3)
assertResult("[▊]")(bar)
}
}
describe("width 1 - 5/8th") {
it("should match") {
val bar = Terminal.progressBar(5d, 8d, 3)
assertResult("[▋]")(bar)
}
}
describe("width 1 - 4/8th") {
it("should match") {
val bar = Terminal.progressBar(4d, 8d, 3)
assertResult("[▌]")(bar)
}
}
describe("width 1 - 3/8th") {
it("should match") {
val bar = Terminal.progressBar(3d, 8d, 3)
assertResult("[▍]")(bar)
}
}
describe("width 1 - 2/8th") {
it("should match") {
val bar = Terminal.progressBar(2d, 8d, 3)
assertResult("[▎]")(bar)
}
}
describe("width 1 - 1/8th") {
it("should match") {
val bar = Terminal.progressBar(1d, 8d, 3)
assertResult("[▏]")(bar)
}
}
}
}

33
filesystem/pom.xml
View file

@ -12,37 +12,28 @@
<name>filesystem</name>
<dependencies>
<!-- lombok -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- thorp -->
<dependency>
<groupId>net.kemitix.thorp</groupId>
<artifactId>thorp-domain</artifactId>
</dependency>
<!-- scala -->
<!-- testing -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<!-- zio -->
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio_2.13</artifactId>
</dependency>
<dependency>
<groupId>dev.zio</groupId>
<artifactId>zio-streams_2.13</artifactId>
</dependency>
<!-- scala - testing -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.13</artifactId>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalamock</groupId>
<artifactId>scalamock_2.13</artifactId>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

Some files were not shown because too many files have changed in this diff Show more