Commit graph

227 commits

Author SHA1 Message Date
899e0724c9
Refactor Copier.copyObject(AmazonS3.Client,Bucket,RemoteKey,MD5Hash,RemoteKey) (#137)
* [core] Convert HashService into an effect Hasher

* [core] LocalFileStreamSuite fix prep of test env

* [core] LocalFileStreamSuite Refactor

* [core] Add Hasher.hashObjectChunk

* [core] make MD5HashGenerator private

Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.

Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.

* [domain] Add Monoid

* [domain] Sources use Monoid

* [core] ActionGenerator Refactor

* [core] Use >>= and *> operators

* [config] Refactoring

* [storage-aws] Refactoring

* [domain] Refactoring

* [core] Refactoring

* [config] refactoring

* Refactoring

* [core] PlanBuilder Refactoring

* [core] Make PlanBuilder into an object

* [storate-aws] Copier Reduce the number of parameters
2019-08-01 08:34:58 +01:00
a749b1f5b0
[filesystem] Merge FS object into FileSystem object (#139) 2019-07-30 15:30:12 +01:00
e5de73e705
Merge config package object with Config object (#138)
* [config] merge config package object with Config object

* [config] Only extract batchMode value where used

Don't pull it out early then pass a boolean around
2019-07-30 10:33:06 +01:00
a93781007d
Create FileSystem as full ZIO effect module (#135)
* [changelog] updated

* [sbt] Add filesystem module

* [filesystem] stub effect module

* [filesystem] Add fileExists

* [filesystem] Add openFile

* [filesystem] Add fileLines

* [config] Use FileSystem.fileExists

* [filesystem] FileSystem.Test implement test methods

* [filesystem] Replace package object with FS object

* [storage] merge storage package object into Storage object

* [console] merge console package with Console object
2019-07-30 08:07:26 +01:00
Scala Steward
f6ce262f2b Update jackson-databind to 2.9.9.2 (#132) 2019-07-28 21:48:24 +01:00
ccefd286f9
Convert Config to full ZIO effect module (#134)
* [config] new module

* [config] stub module

* [domain] Rename domain.Config as domain.LegacyConfig

* [config] Move LegacyConfig to config module

* [config] Move config parsing and validation into module

* [config] Complete migration to module for Config

* [config] Config You should not name methods after their defining object

* [config] Rename LegacyConfig as Configuration

Also remove redundant uses

* [core] LocalFileStream Refactoring

* [changelog] update
2019-07-28 21:47:01 +01:00
96a83e6c3e
Convert Storage to full ZIO effect module (#133)
* [console] Rename MyConsole as Console

* [console] break infinite loop

* [console] fix typo

* [console] clean up helpers

* [cli] Main use ZIO#provide to run program

* [cli] Main define Program type alias

* [cli] Program handle cli args in Program

* [cli] Program doesn't extend PlanBuilder

* [cli] refactoring

* [cli] rename ParseArgs as CliArgs

* [cli] CliArgs#apply renamed a parse

* [storage-aws] S3StorageService renamed as S3Storage

* [storage-api] Rename StorageService as Storage.Service

* [storage-api] make Storage.copy effectTotal

* [storage-api] make Storage.delete effectTotal

* [storage-api] make Storage.shutdown effectTotal

* [storage-api] make Storage.upload effectTotal

* [storage-aws] Lister refactoring

* [storage-aws] make Lister into a trait

* [storage-aws] make Copier into a trait

* [storage-aws] make Deleter into a trait

* [storate-aws] make Uploader into a trait

* [storage-aws] AmazonS3 move error handling out of client wrapper

* [storage-aws] DeleterTest added

* [storage-aws] ListerTest added

* [storage-aws] Uploader refactoring

* [storage-aws] CopierTest test Copier directly

* [storage-aws] DeleterTest test Deleter directly

* [storate-aws] ListerTest test Lister directly

* [storage-aws] UploaderTest added

* [storage-aws] S3Storage.Live replaces S3StorageServiceBuilder

* Complete migration to Module for Storage

* [cli] Main define LiveThorpApp object

* [core] Add CoreTypes

* [cli] Program Refactoring

* [core] PlanBuilding Refactoring

* [changelog] updated

* [console] Console.Live Usage of get on optional type

* [storage-aws] AmazonS3ClientTestFixture Use wildcards when selecting more than 6 elements
2019-07-28 20:11:03 +01:00
985cc9f147
Don't use String as key in Map for hashes (#124)
* Don't use String as key in Map for hashes

* [domain] HashType remote redundant braces
2019-07-24 19:50:28 +01:00
ad0f848bed
Ensure file handles are always closed (#122)
* [core] Ensure files are closed even when interrupted

* [changelog] updated
2019-07-24 18:38:19 +01:00
cad152379e
Remove Monocle dependency (#121)
* Replace Monocle with local SimpleLens implementation

* [domain] SimpleLensTest avoid short variables and remove field imports
2019-07-24 09:40:56 +01:00
ad0585e9a9 [changelog] updated 2019-07-23 23:16:08 +01:00
Scala Steward
763c813a1f Update aws-java-sdk-s3 to 1.11.596 (#118) 2019-07-23 23:14:23 +01:00
f277b5e789
Improve logging consistency (#119)
* [console] create new effect MyConsole

* [core] SyncLogging use ConsoleOut.ValidConfig

* [storage-aws] Extract AmazonS3 trait to help testing

* [console] MyConsole use UIO

* [storage-aws] S3StorageServiceSuite convert to FreeSpec

* [storage-aws] S3StorageServiceSuite extract S3ClientTest trait

* [storage-aws] remove incomplete test UploaderSuite

* [storage-aws] extract and rename AmazonS3ClientTestFixture

* [storage-aws] Copier handle and log errors

* [core] ThropArchive log completed uploads here

* [core] ThorpArchive log copies and deletes

* Improve consistency of logging

* [storage-aws] CopierTest extract from S3StorageServiceSuite

* [storage-aws] Copier handle hash match errors properly

* [core] ThropArchive display erros in red

* [core] SequencePlan extracted

* [core] Clear line after deletion log

* [changelog] updated

* [cli] Program tidy up imports

* [storage-aws] Copier replace null with Option

* [storage-aws] AmazonS3 copyObject returns Some[CopyObjectResult]
2019-07-23 23:13:09 +01:00
8cca46340c
Replace cats-effect with zio (#117)
* [sbt] Add ZIO dependency to storage-api

* Convert to use ZIO

* [sbt] remove cats-effect dependency

* [changelog] updated

* [cli] Program restore actions to correct order (copy, upload, delete)

* [cli] Program You should not name methods after their defining object

* [core] ConfigValidationException Redundant braces after class definition

* [core] LocalFileStreamSuite Usage of get on optional type.

* [core] PlanBuilderTest Usage of get on optional type.
2019-07-21 21:02:04 +01:00
32ef58ff11
NullPointerException creating scaladoc (#115)
* [domain] Drop use of Lenses macro and only create used lenses

* [core] Drop use of Lenses macro and only create used lenses

* [sbt] drop paradise plugin

* [changelog] updated
2019-07-19 23:30:20 +01:00
c33fa05d19
Creates incorrect MD5 hash for some files (#103)
* [core] Add test file that we create incorrect hash for

By 'incorrect' we mean "not what AWS S3 think is correct" for this
file.

* [domain] HexEncoder rewritten to correctly decode hex

* [domain] HexEncoder encode to uppercase
2019-07-19 20:54:39 +01:00
515b896993
Use Lenses (#113)
* [sbt] Add monocle for support for lenses

* [domain] UploadEventListener make a case class

* [domain] Add @Lenses to all case classes

* [core] Add @Lenses to case classes

* [core] ActionGenerator use lense

* [core] ConfigOption use Lenses

* [core] ConfigurationBuilder remove unused fields

* [core] ConfigurationBuilder refactoring

* [core] SyncLogging use lenses

* [core] syncLogging refactoring
2019-07-18 08:57:14 +01:00
0fbae945a7 [domain] Sources remove broken javadoc 2019-07-16 09:05:13 +01:00
afc55354e7
Apply scalafmt (#108)
* [scalafmt] Add .scalafmt

* [sbt] Add unused import warning and make them errors

* [core] ConfigurationBuilder remove unused import

* [core] ConfigurationBuilder apply scalafmt

* [domain] reformat

* [storage-api] reformat

* [core] reformat and some refactoring

* [storage-aws] reformat

* [cli] reformat

* [core] post rebase fix up

* [storage-aws] UploaderSuite tidy up

* [domain] MD5Hash tweak

Without doing this we have a file that we don't create a valid md5
hash for. See #103

* [storage-aws] UploaderSuite remove unused import

* [storage-aws] StorageServiceSuite reformatted

* [sbt] consistent settings for all modules

Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.

* [storage-aws] Add AmazonTransferManager

This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.

* [storage-aws] Add AmazonUpload

Prevents deprecation errors due to deprecated parameters on Transfer.

* [sbt] mode import to top of file
2019-07-16 07:56:54 +01:00
dfb885b76d [changelog] update v0.7.1 release date 2019-07-15 07:02:26 +01:00
6a55e74047
Not reading .thorp.conf file (#111)
* [domain] Config defaults to an empty Sources list

* [core] ConfigurationBuilterTest add test for parsing .thorp.conf

* [core] ConfigQuery if no Sources given returns current dir

* [core] rewrote config loader

- Only settings from explicit sources are used

* [changelog] updated

* [core] Remove stray println statements

* [core] SyncLogging tidy up multi-source messages
2019-07-15 07:01:06 +01:00
f2131ab7fc
Not reading .thorp.conf file (#110)
* [core] ConfigurationBuilder read the correct source config file

* [changelog] updated
2019-07-13 15:43:40 +01:00
775cba9cc0 [changelog] fix typo 2019-07-12 08:15:45 +01:00
73fc4546ed [changelog] updated 2019-07-12 08:09:02 +01:00
b15350d959
Sync more than one source directory into a single bucket/prefix (#25)
* [changelog] updated

* [readme] updated

* [core] ConfigQuery added sources()

* [cli] ParseArgs allow specifying multiple sources

* [domain,core,cli] Source datatype changed to Path

* [domain] Sources added to hold multiple paths in order

* [domain] Config sources change datatype to Sources

* [core] Scan sources for .thorp.config and include any sources listed

This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.

* [core] ConfigurationBuilderTest add more pending tests

* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures

* [core] ConfigOptionTest use TemporaryFolder

* [core] ConfigOptionTest remove unused fields

* [cli] ParseArgsTest don't use get on an Option

* [core] ConfigurationBuilderTest don't use get on Either

* [core] TemporaryFolder Move import to top of file

* [core] TemporaryFolder use Try over try-finally

* [core] ConfigurationBuilderTest don't use get on Either

* [core] TemporaryFolders.withDirectory propogate errors

* [core] TemporaryFolders add writeFile and createFile

* [core] PlanBuilderTest create a plan with two sources with unique files in both

* [core] ActionGenerator only upload file by name in first source

create a plan
  two sources
    same filename in both
    - only upload file in first source

* [domain] LastModified with no params is now()

* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing

* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing

* [domain] RemoteKey map to a file when prefix is empty

* [domain] S3ObjectData defaults to empty

* [core] KeyGenerator Avoid delimiter when empty prefix key

* [core] PlanBuilderTest when remote not in sources delete from remote

* [core] PlanBuilderTest extract helper md5Hash()

* [core] PlanBuilderTest one source a file no matching remote key

* [core] PlanBuildingTest file with matching key and hash do nothing

* [core] PlanBuilderTest file w/matching remote key and different hash

* [core] PlanBuilderTest a remote key with and without local file

* [core] DummyStorageService Use wildcards when selecting more than 6 elements
2019-07-12 07:42:42 +01:00
a6c767f16f
[readme] add maven version badge 2019-07-06 17:20:33 +01:00
Scala Steward
21638d9530 Update sbt-assembly to 0.14.10 (#105) 2019-07-06 13:47:24 +01:00
Scala Steward
2b284a9649 Update aws-java-sdk-s3 to 1.11.587 (#104) 2019-07-06 13:46:14 +01:00
00c04187e8
Display total size and progress for entire run (#94)
* [changelog] updated

* [core] Wrap Stream[LocalFile] as LocalFiles

* [core] LocalFiles counts files

* [core] LocalFiles sums file lengths

* [core] Restore logFileScan

* [storage-aws] Lister logs when fetching object summaries

* [storage-aws] Extract ListerLogger

* [core] Synchronise use leftMap

* [core] Syncronise extract assemblePlan

* [core] Wrap Stream[Action] in SyncPlan

* [core] Copy the file count and totalSizeBytes across to SyncPlan

* [cli] Program rename actions as syncPlan

* [cli] Program extract thorpArchive def

* [cli] Program extract createPlan def

* [cli] Program refactoring

* [cli] Program remove println showing version

* [cli] Program rename actions parameter as syncPlan

* [core] ThorpArchive add an index to each action

* [cli] Program make SyncPlan available to ThorpArchive

* [core] Pass SyncTotals to Archive

* [domain] Move SyncTotals into module

* [domain] Pass index and SyncTotals to UploadEventListener

* [domain] UploadEventLogger add file count a size progress bars

* [domain] UploadEventLogger better display stability and add file index

* [cli] Index files in correct order

* [cli] Program extends Synchronise

* [core] Rename Synchronise as PlanBuilder

* [cli] Program add test to check actions don't get reordered from plan

* [core] collect file size totals

* [domain] UploadEventLogger include percentage

* [cli] ProgramTest Use wildcards when selecting more than 6 elements
2019-07-04 18:58:31 +01:00
26b4f6f794 Merge branch 'v0.6.x'
* v0.6.x:
  Release to sonatype using correct profile name (#102)
2019-07-03 18:42:40 +01:00
Scala Steward
6cda33a94a Update jackson-databind to 2.9.9.1 (#101) 2019-07-03 14:18:32 +01:00
8585023c3e
Release to sonatype using correct profile name (#102)
* [changelog] updated

* [sbt] Specify sonatypeProfileName in commonSettings

This prevents the value being overridded by the `organization` value
from `inThisBuild`.
2019-07-03 13:50:28 +01:00
1267b6e313
Add a batch mode that provides a simple log output (#85)
* [changelog] Updated

* [readme] Updated

* [domain] Config Add batch-mode flag

* [core] ConfigOption Add BatchMode option

* [core] ConfigQuery Add batchMode query

Also replaced verbose exists case clauses with a simple contains.

* [core] ConfigOptions added to replace Seq[ConfigOption]

* [core] Syncronise rename method to createPlan

* [cli] Program rename apply as run

* [storage-aws] S3StorageServiceBuilder stop using IO to create object

* [storage-aws] S3StorageServiceBuilder make default service lazy

* [storage-aws] Rename S3ClientCopier => Copier

* [storage-aws] Rename S3ClientDeleter => Deleter

* [storage-aws] Rename S3ClientObjectLister => Lister

* [storage-aws] Only attach upload listener when in batch mode

Only detects batch mode when selected as a command line option

* [core] Synchronise use leftMap rather than swap.map.swap

* [cli] ParseArgs add `-B` and `--batch` options to enable batch mode

* [core] ThorpArchive logs file uploaded when in batch mode
2019-07-02 08:43:52 +01:00
1440990d79
Update CHANGELOG.org
Start section for 0.7.0
2019-06-30 22:05:34 +01:00
5df752047f
Add a version command-line option (#99)
* [sbt] make name and version available within Scala

Available as `thorp.BuildInfo._`

* [core] ConfigOption Add Version option

* [cli] ParseArgs add command-line options `-V` and `--version`

* [core] ConfigQuery add showVersion query

* [cli] Program shows version then exit

* [cli] Main don't show "Done"
2019-06-30 22:01:02 +01:00
561966218c
Remove repo badges
Until releases are made, these are just distractions.
2019-06-30 19:49:22 +01:00
8120fca0ae [sbt] Only specify organization once 2019-06-30 19:00:10 +01:00
88dd2f37a4 [sbt] specify sonatype profile name 2019-06-30 17:42:16 +01:00
3f54519a1b [changelog] updated 2019-06-30 15:32:25 +01:00
cff67401be [readme] updated 2019-06-30 15:31:00 +01:00
a2e651218e [changelog] updated 2019-06-30 15:22:11 +01:00
619424a416
Embed launch script in fat-jar (#98)
* [sbt] enable executable jar for shell

* [sbt] rename assembled jar to not include file suffix
2019-06-30 15:04:23 +01:00
124c269322
[sbt,travis] revert most of "publish fat-jar", keeping cli jar name (#97) 2019-06-30 14:38:50 +01:00
4574e2c535
Include assembly jar in release (#96)
* [sbt] set name for assembly jar

* [travis] run assembly prior to release

* [sbt] disable publishing non-cli or thorp-lib modules

* [travis] alternate format for multiple commands in script
2019-06-30 13:58:19 +01:00
708f3697ee
Release snapshots to sonatype from TravisCI (#92)
* [sbt,travis] Deploy releases and snapshots to sonatype

* [travis] remove oracle JDK builds

* [travis] remove 'compile' job

* [travis] Try adding scala 2.13

* [travis] test using openjdk8 rather than 10

* [travis] force use of scala version, overriding sbt version

* Revert "[travis] Try adding scala 2.13"

This reverts commit 6725199e009ae1f809247bbc0851b62d63121993.

* Revert "[travis] force use of scala version, overriding sbt version"

This reverts commit 21c94992f40cf2ef39c5126e67b66feaabe1b2c0.

* [sbt] Group modules under n.k.thorp organisation
2019-06-30 12:23:04 +01:00
0fa8382fa3
[cli] Program handle actions list in stack safe manner (#93) 2019-06-30 08:47:51 +01:00
c23376a037
Shutdown storage service once completed (#88) 2019-06-29 20:04:36 +01:00
ac9a52f93f
Use correct hash locally for comparing multi-part uploaded files (#82)
* [storage-aws] ETagGenerator add stub

* [core] MD5HashGenerator add hex and digest helpers

* [domain] MD5Hash can always provide base64 and also digest

Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.

MD5Hash is now cleaner now that it no longer has Option parameters.

* [core] MD5HashGenerator add stubs to allow reading file chunks

* [domain] MD5HashData add sub-objects

* [domain] MD5HashData move back into test where it belongs

* [sbt] add sbt-bloop plugin

* [domain] MD5HashData Add hash of big-file

* [domain] MD5HashData Add hash of big-file

* [core] MD5HashGenerator find end of chunk correctly

* [core] MD5HashGenerator offset is a Long

* [core] MD5HashGenerator don't read past the end of the file

* [storage-aws] ETagGenerator can reproduce ETags

* [storage-aws] ETagGeneratorTest added

* [storate-aws] ETagGenerator refactoring

* [storage-aws] ETageGenerator refactoring

* [core] SyncSuite remove redundant braces

* [storage-api] HashService added

* [storage-aws] S3HashService added

* [core] LocalFileStream refactoring

* [core] integrate HashService and ETagGenerator

* Optimise imports

* [domain] HexEncoder added to replace java 8 only DataTypeConverter

* [core] MD5HashGenerator refactoring

* [core] S3MetaDataEnricher refactoring

* [core] S3MetaDataEnricherSuite refactoring

* [storage-aws] ETagGeneratorTest refactoring

* [storage-aws] StorageServiceSuite refactoring

* [core] S3MetaDataEnricher refactoring

* [core] refactoring

* [storage-aws] refactoring
2019-06-29 19:07:51 +01:00
9418744136
Display simple error message when bucket is invalid (#81)
* [core,storage-{api,aws}] Allow Lister errors to be reported to user

* [cli,core,storage-*] Display simple error message when bucket is invalid

* [core] optimise imports

* [storate-aws] optimise imports

* [core] SyncSuite don't use get on an optional type
2019-06-27 07:34:15 +01:00
Scala Steward
f0783e4419 Update aws-java-sdk-s3 to 1.11.580 (#86) 2019-06-26 22:14:15 +01:00