Commit graph

55 commits

Author SHA1 Message Date
515b896993
Use Lenses (#113)
* [sbt] Add monocle for support for lenses

* [domain] UploadEventListener make a case class

* [domain] Add @Lenses to all case classes

* [core] Add @Lenses to case classes

* [core] ActionGenerator use lense

* [core] ConfigOption use Lenses

* [core] ConfigurationBuilder remove unused fields

* [core] ConfigurationBuilder refactoring

* [core] SyncLogging use lenses

* [core] syncLogging refactoring
2019-07-18 08:57:14 +01:00
afc55354e7
Apply scalafmt (#108)
* [scalafmt] Add .scalafmt

* [sbt] Add unused import warning and make them errors

* [core] ConfigurationBuilder remove unused import

* [core] ConfigurationBuilder apply scalafmt

* [domain] reformat

* [storage-api] reformat

* [core] reformat and some refactoring

* [storage-aws] reformat

* [cli] reformat

* [core] post rebase fix up

* [storage-aws] UploaderSuite tidy up

* [domain] MD5Hash tweak

Without doing this we have a file that we don't create a valid md5
hash for. See #103

* [storage-aws] UploaderSuite remove unused import

* [storage-aws] StorageServiceSuite reformatted

* [sbt] consistent settings for all modules

Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.

* [storage-aws] Add AmazonTransferManager

This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.

* [storage-aws] Add AmazonUpload

Prevents deprecation errors due to deprecated parameters on Transfer.

* [sbt] mode import to top of file
2019-07-16 07:56:54 +01:00
Scala Steward
2b284a9649 Update aws-java-sdk-s3 to 1.11.587 (#104) 2019-07-06 13:46:14 +01:00
26b4f6f794 Merge branch 'v0.6.x'
* v0.6.x:
  Release to sonatype using correct profile name (#102)
2019-07-03 18:42:40 +01:00
Scala Steward
6cda33a94a Update jackson-databind to 2.9.9.1 (#101) 2019-07-03 14:18:32 +01:00
8585023c3e
Release to sonatype using correct profile name (#102)
* [changelog] updated

* [sbt] Specify sonatypeProfileName in commonSettings

This prevents the value being overridded by the `organization` value
from `inThisBuild`.
2019-07-03 13:50:28 +01:00
5df752047f
Add a version command-line option (#99)
* [sbt] make name and version available within Scala

Available as `thorp.BuildInfo._`

* [core] ConfigOption Add Version option

* [cli] ParseArgs add command-line options `-V` and `--version`

* [core] ConfigQuery add showVersion query

* [cli] Program shows version then exit

* [cli] Main don't show "Done"
2019-06-30 22:01:02 +01:00
8120fca0ae [sbt] Only specify organization once 2019-06-30 19:00:10 +01:00
88dd2f37a4 [sbt] specify sonatype profile name 2019-06-30 17:42:16 +01:00
619424a416
Embed launch script in fat-jar (#98)
* [sbt] enable executable jar for shell

* [sbt] rename assembled jar to not include file suffix
2019-06-30 15:04:23 +01:00
124c269322
[sbt,travis] revert most of "publish fat-jar", keeping cli jar name (#97) 2019-06-30 14:38:50 +01:00
4574e2c535
Include assembly jar in release (#96)
* [sbt] set name for assembly jar

* [travis] run assembly prior to release

* [sbt] disable publishing non-cli or thorp-lib modules

* [travis] alternate format for multiple commands in script
2019-06-30 13:58:19 +01:00
708f3697ee
Release snapshots to sonatype from TravisCI (#92)
* [sbt,travis] Deploy releases and snapshots to sonatype

* [travis] remove oracle JDK builds

* [travis] remove 'compile' job

* [travis] Try adding scala 2.13

* [travis] test using openjdk8 rather than 10

* [travis] force use of scala version, overriding sbt version

* Revert "[travis] Try adding scala 2.13"

This reverts commit 6725199e009ae1f809247bbc0851b62d63121993.

* Revert "[travis] force use of scala version, overriding sbt version"

This reverts commit 21c94992f40cf2ef39c5126e67b66feaabe1b2c0.

* [sbt] Group modules under n.k.thorp organisation
2019-06-30 12:23:04 +01:00
ac9a52f93f
Use correct hash locally for comparing multi-part uploaded files (#82)
* [storage-aws] ETagGenerator add stub

* [core] MD5HashGenerator add hex and digest helpers

* [domain] MD5Hash can always provide base64 and also digest

Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.

MD5Hash is now cleaner now that it no longer has Option parameters.

* [core] MD5HashGenerator add stubs to allow reading file chunks

* [domain] MD5HashData add sub-objects

* [domain] MD5HashData move back into test where it belongs

* [sbt] add sbt-bloop plugin

* [domain] MD5HashData Add hash of big-file

* [domain] MD5HashData Add hash of big-file

* [core] MD5HashGenerator find end of chunk correctly

* [core] MD5HashGenerator offset is a Long

* [core] MD5HashGenerator don't read past the end of the file

* [storage-aws] ETagGenerator can reproduce ETags

* [storage-aws] ETagGeneratorTest added

* [storate-aws] ETagGenerator refactoring

* [storage-aws] ETageGenerator refactoring

* [core] SyncSuite remove redundant braces

* [storage-api] HashService added

* [storage-aws] S3HashService added

* [core] LocalFileStream refactoring

* [core] integrate HashService and ETagGenerator

* Optimise imports

* [domain] HexEncoder added to replace java 8 only DataTypeConverter

* [core] MD5HashGenerator refactoring

* [core] S3MetaDataEnricher refactoring

* [core] S3MetaDataEnricherSuite refactoring

* [storage-aws] ETagGeneratorTest refactoring

* [storage-aws] StorageServiceSuite refactoring

* [core] S3MetaDataEnricher refactoring

* [core] refactoring

* [storage-aws] refactoring
2019-06-29 19:07:51 +01:00
Scala Steward
f0783e4419 Update aws-java-sdk-s3 to 1.11.580 (#86) 2019-06-26 22:14:15 +01:00
Scala Steward
3a0402d568 Update scalamock to 4.3.0 (#84) 2019-06-26 22:13:34 +01:00
0f8708e19f
Restructure sync to use a State with foldLeft around actions (#74)
* [changelog] updated

* [cli] Program rename parameter

* [core] Add AppState

* [core] Synchronise rought draft replacement for Sync

Uses the AppState

* [core] Synchronise as sequential for-comprehensions

* [core] Synchronise as nested for-comprehensions

* [sbt] thorp(root) depends on cli moduke

* [core] Synchronise extract methods

* [core] Synchronise rewritten

* [core] Synchronise generates actions

* [core] Remove AppState

* [core] ActionSubmitter remove unused implicit config parameter

* [cli] Program rewritten to use Synchronise

* [core] Synchronise useValidConfig accepts Logger implicitly

* [core] Synchronise reorder methods

* [core] Synchronise refactor errorMessages

* [core] SyncLogging logRunStart accepts explicit parameters

* [core] remove old Sync

* [core] Synchronise restore logRunStart

* [domain] Terminal add types to public methods and values

* [domain] UploadEventLogger force flush to terminal

Also make part of the progress message in green.

Not flushing, by using println, cause odd behaviour. Works on normal
terminal, but not great in an emacs terminal. Oh well.

* [core] SyncLogging.logRunFinished remove unused parameters

* [cli] Program restore final summary

* [storage-aws] remove logging from module

* [core] ThorpArchive replaces ActionSubmitter

ActionSubmitter implementation becomes UnversionedMirrorArchive

* [domain] cleaner upload progress messages

* [cli] Program remove unused Logger

* [cli] Program rename parameter

* [core] SyncSuite use Synchronise

* [sbt] Allow storage-aws to share core test classes

* [domain] LocalFile stop storing a lambda

The lambda breaks the equality test between LocalFile instances.

* [core] MD4HashData add missing base64 digest for leafFile

* [core] Synchronise drop DoNothing actions

* [core] SyncSuite update tests

* [sbt] aggregate modules from root module
2019-06-25 08:27:38 +01:00
9d2271fdcf
Introduce backend abstraction to hide S3 (#76)
Add backend abstraction to hide S3
2019-06-22 07:20:59 +01:00
910688ee32
Add support for per directory configuration files (#71)
* [core] rename the config supplied from CLI as such

This distinguishes it as config supplied from the command line.

* [core] add ConfigOption

* [core] ConfigOption can update a Config

* [core] new validator for config

* [domain] Config doesn't validate source any more

* [cli] PrintLogger default to not print debug messages

* [cli] Use ConfigOptions and new ConfigValidator

* [sbt] Use common settings for project root

* [domain] RemoteKey can handle when prefix is empty

* [cli] remove banner

* [domain] Logger can create version of itself with debug flipped

* [core] Build and validate Config within core module

This means that the `thorp-lib` module can validate its input from a
list of `ConfigOption`s.

* [core] refactor ConfigurationBuilder

* [core] refactor ConfigurationBuilder

* [sbt] starting back from tagless-final by using IO where needed

* [core] Add ParseConfigFile

* [sbt] Make cats-effect available from the domain

* Roll back from tagless-final to just use cat-effect's IO

* [core] extract ParseConfigLines

* [core] ConfigurationBuilder rename apply as buildConfig

* [core] ParseConfig[Files,Lines] rename apply methods

* [core] refactor ParseConfigFile and add tests

* [core] Sync fix call to run

* [core] SyncSuite update tests to use ConfigOptions
2019-06-20 17:18:46 +01:00
afd28e7100
[sbt] add thorp-lib module (#66) 2019-06-17 16:47:01 +01:00
9196dd623f
Rename project to Thorp (#75)
* [sbt] change application name

* [cli] rename package

* [cli] Change displayed application name and description

* [domain] rename package

* [core] fix bad package directory structure

* [core] rename package

* [aws-lib] rename package

* [aws-api] rename package

* [cli] rename programe for usage message

* [bin] rename and update script

* [gitignore] update

* [readme] update
2019-06-17 15:33:49 +01:00
Scala Steward
6450bbeff5 Update aws-java-sdk-s3 to 1.11.573 (#65) 2019-06-15 11:34:34 +01:00
Scala Steward
0b60d52db5 Update aws-java-sdk-s3 to 1.11.572 (#62) 2019-06-14 20:17:53 +01:00
a780c5fdfe
[cli] Display "Source is not a directory" if so (#51) 2019-06-14 19:01:46 +01:00
Scala Steward
fffe156cc1 Update aws-java-sdk-s3 to 1.11.570 (#57) 2019-06-14 08:52:05 +01:00
8d0c3e23c9
Improve purity by moving all IO out to the edge (#52)
* [aws-api] TTFI UploadProgressLogging#logTransfer

* [aws-api] TTFI UploadProgressLogging#logRequestCycle

* [aws-api] TTFI UploadProgressLogging#logByteTransfer

* [aws-api] TTFI UploadProgressListener

* [aws-lib] TTFI UploaderLogging

* [aws-api] UploadProgressListener refactoring

* [aws-api] UploadProgressListener remove IO/Monad

This class is a wrapper for a callback method used by the AWS SDK.
Unfortunately you can't get the IO() created when that listener is
called by the SDK, so unless we manually run unsafeRunSync, as we have
done previously, it would never be executed.

This removes the IO monad entirely and simply runs the code when the
callback is triggered.

* [aws-lib] S3ClientLogging remove unused method

* [aws-lib] TTFI S3ClientLogging

* [aws-lib] TTFI S3ClientCopier

* [aws-lib] TTFI S3ClientObjectLister

* [aws-lib] TTFI Uploader

* [aws-lib] TTFI S3ClientDeleter

* [aws-api] TTFI S3Client

* [aws-lib] TTFI S3ClientBuilder and ThorpS3Client

* [core] TTFI ActionSubmitter

* [cli] TTFI Logger

* [core] TTFI MD5HashGenerator

* [core] TTFI LocalFileStream

* [core] Sync refactoring

* [core] TTFI Sync

* [aws-lib] S3ObjectsByHashSuite truncate lastmodified to match Date

* [aws-lib] ThorpS3ClientSuite truncate lastmodified to match Date

* [core] MD5HashGeneratorTest switch to Id from IO

* [sbt] restrict cats-effect to cli module, cats-core elsewhere

* [core] MD5HashGenerator collapse lines
2019-06-14 08:51:05 +01:00
5996632c1e
Enable running outside of sbt (#55)
Enable running outside of sbt
2019-06-11 23:36:08 +01:00
Scala Steward
54404776d9 Update aws-java-sdk-s3 to 1.11.569 (#53) 2019-06-11 07:38:16 +01:00
Scala Steward
208b9bdc78 Update scalatest to 3.0.8 (#54) 2019-06-11 07:36:58 +01:00
97efed76b4
Improve upload logging (#44)
* [aws-lib] fold S3ClientUploader trait into it's only implementation

This trait was only implemented by S3ClientTransferManager.

* [core] SyncLogging: more robust matching

No longer cares about parameters to case classes, just their types.

* [cli] Logger uses IO for log methods

* [aws-lib] remove 'transfer-manager'prefix and only show tryCount > 1

* [sbt,cli] remove log4j and scala-logging dependencies

* [domain] move QuoteStripper to Domain

Use it directly in MD5Hash to strip quotes from any input.

* [core] SyncLogging call info in proper context

If the IO.unit returned by the info calls isn't part of the chain that
is returned from the function, then the delayed IO action is never
called.

* [aws-lib] Display size in bytes of file being uploaded

* [core] call info in correct context

* [cli] call info in correct context

* [aws-lib] raise summary fetch message to info 1

* [cli] include correct level in info messages

* [aws-lib] S3ClientLogging adjust logging levels

* [aws-lib] display file sizes in english

* [aws-lib] ObjectLister use IO.bracket properly

* [aws-lib] Copier use IO.bracket properly

* [aws-lib] Deleter refactor

* [aws-lib] TransferManagerLogging remove unused methods

* [aws-lib] TransferManager refactor

* [aws-lib] TransferManager refactor

* [aws-lib] TransferManager displays log messages

Use the UploadProgressListener that was being ignored, and use
unsafeRunSync to execute the suspended effect within the IO[Unit].
Using unsafeRunSync is required to render the effects as the listener
returns Unit, meaning the suspended effects would be discarded.

* [domain] Extract SizeTranslation into module

* [aws-api] report bytes transferred in progress

* [core] fix calls to info

info now returns an IO already, so don't need to wrap it in one.

* [aws-lib] remove unused class

* [aws-lib] UploadProgress displays progress bar while uploading

* [aws-api] UploadProgressLogging optimise imports

* [aws-api] UploadProgressLogging rename variables

* [domain] add Terminal object

* [aws-api] UploadProgressLogging use console width and two lines

- Improved clearing of lines after progress bar
- Use console width for progress bar size

* [aws-lib] S3ClientLogging optimise imports

* [aws-lib] TransferManager clear line before logging

* [aws-lib] rename class as TransferManager

* [aws-lib] rename TransferManger as Uploader to not clash

We are using an AWS SDK class with the same name.
2019-06-10 19:45:36 +01:00
Scala Steward
07f12ac19f Update aws-java-sdk-s3 to 1.11.568 (#42) 2019-06-08 08:21:07 +01:00
Scala Steward
bb4e7cff4f Update scalamock to 4.2.0 (#43) 2019-06-08 08:20:33 +01:00
aa7fb1eb24
Drop AWS SDK V2 client (#41)
* [sbt] add scalamock as a test dependency

* [aws-lib]SyncSuite: minor layout changes

* [aws-lib]SyncSuite: remove test

* [core] move SyncSuite to same module as subject it tests

* [aws-lib]ThorpS3Client: remove commented lines

* [aws-lib] remove PutObject versions of Uploader

* [aws-lib] rename to TransferManager to remove Multi-part from name

* [aws-lib]TransferManager: change logging prefix

* [aws-lib] convert logging classes to objects

* [aws-lib] convert ObjectLister to use V1 SDK

* [aws-lib] convert Copier to use V1 SDK

* [aws-lib] extract S3ObjectsBy{Hash,Key} to objects

* [aws-lib]S3ClientSuite: rewrite test using mocks

* [aws-lib]TransferManager rewrite using for-comprehension

* [aws-lib]Copier: remote bucket name from target remote key

* [aws-lib]TransferManager: refactor logging to use IO themselves

* [aws-lib] Remove test class MyAmazonS3

* [aws-lib]ObjectLister: optimise imports

* [aws-lib] S3ClientSuite remove commented code

* [aws-lib]ThropS3ClientSuite update to V1 api

* [aws-lib]S3ClientSuite: make test as pending

It works okay on its own, but when run as part of a suite it fails.

Will look at this again once all V2 SDK is removed.

* [aws-lib] convert Deleter to use V1 SDK

* [aws-lib] Client Logging remove redundant braces

* [aws-lib] stop injecting the V2 SDK

* [sbt] remove v2 SDK dependencies

* [aws-lib] remove redundant helpers for v2 SDK

* [sbt] upgrade aws jackson dependencies

The jackson libraries used by AWS have security flaws, but are Java 6
compatible, which AWS want to preserve.

* [aws-lib] clean up TransferManager tests
2019-06-07 21:17:14 +01:00
Scala Steward
908ac855ff Update aws-java-sdk-s3 to 1.11.567 (#38) 2019-06-07 21:04:00 +01:00
Scala Steward
e5078fd0b6 Update cats-effect to 1.3.1 (#39) 2019-06-07 21:03:45 +01:00
Scala Steward
96fc2812dd Update aws-java-sdk-s3 to 1.11.566 (#35) 2019-06-06 19:54:17 +01:00
f54c50aaf3
Split into subprojects (#36)
* [sbt] define existing single module project as legacyRoot

* [sbt] add empty cli module depending on legacyRoot

* [cli] move Main to cli module

* [cli] move ParseArgs to cli module

* [sbt] limit scope of scopt dependency to cli module

* [cli] moved logging config to cli module

* [cli] rename module directory

* [aws-api] added empty module

* [sbt] aggregate builds from cli

* [aws-lib] add empty module

* [core] add empty module

* [sbt] add comment graphing module dependencies

* [sbt] adjust module dependencies to reflect plan

Include legacyRoot at the base until it can be redistributed

* [legacy] make some awssdk classes non-private

during this transition, these classes being private would cause problems

* [aws-lib] create S3ClientBuilder

This is copied from the legacy S3Client companion object

* [domain] add empty module

* [domain] move Bucket into module

* [legacy] RemoteKey no longer has dependency on Config

* [domain] move RemoteKey into module

* [domain] move MD5Hash into module

* [legacy] LocalFile no longer had dependency on MD5HashGenerator

* [domain] move LocalFile into module

* [domain] mode LastModified into module

* [domain] move RemoteMetaData into module

* [domain] move S3MetaData into module

* [domain] move Exclude into module

* [domain] move Filter into module

* [domain] move KeyModified into module

* [domain] move HashModified into module

* [domain] RemoteKey.resolve added

* [domain] add dependency on scalatest

* [domain] LocalFile.resolve added

* [legacy] Remove UnitTest

* [legacy] optimise imports

* [domain] move S3ObjectsData moved into module

* [legacy] wrapper for using GeneralProgressListener

* [domain] move Config into module

* [sbt] move aws-api below legacyRoot in dependencies

This will allow use to move S3Client into the aws-api module

* [legacy] rename S3Client companion as S3ClientBuilder

Preparation to move this into its own file.

* Inject Logger via CLI (#34)

* [S3Client] refactor defaultClient()

* [S3Client] transfermanager explicitly uses the same s3client

* [S3ClientPutObjectUploader] refactor putObjectRequest creation

* [cli] copy in Logging trait as Logger class

* [cli] Main uses Logger

* [cli] simplify Logger and pass to Sync.run

* [legacy] SyncLogging converted to companion

* [cli] Logger info can more easily use levels again

* [legacy] LocalFileStream uses injected info

* [legacy] S3MetaDataEnricher remove unused Logging

* [legacy] ActionGenerator remove unused Logging

* [legacy] convert ActionGenerator to an object

* [legacy] import log methods from SyncLogging

* [legacy] move getS3Status from S3Client to S3MetaDataEnricher

* [legact] convert ActionsSubmitter to an object

* [legacy] convert LocalFileStream to an object

* [legacy] move Action case classes inside companion

* [legacy] move UploadEvent case classes inside companion and rename

* [legacy] move S3Action case classes into companion

* [legacy] convert Sync to an object

* [cli] Logger takes verbosity level at construction

No longer needs to be passed the whole Config implicitly for each info
call.

* [legacy] stop passing implicit Config for logging purposes

Pass a more specific implicit info: Int => String => Unit instead

* [legacy] remove DummyS3Client

* [legacy] remove Logging

* [legacy] convert MD5HashGenerator to an object

* [aws-api] move S3Client into module

* [legacy] convert KeyGenerator to an object

* [legacy] don't use IO.unsafeRunSync directly

* [legacy] refactor/rewrite Sync.run

* [legacy] Rewrite sort using a for-comprehension

* [legacy] Sync inline sorting

* [legacy] SyncLogging rename method

* [legacy] repair tests

* [sbt] move core module to a dependency of legacyRoot

* [sbt] add test dependencies to core module

* [core] move classes into module

* [aws-lib] move classes into module

* [sbt] remove legacy root
2019-06-06 19:24:15 +01:00
Scala Steward
b7e79c0b36 Update aws-java-sdk-s3 to 1.11.564 (#33) 2019-06-06 13:11:27 +01:00
Scala Steward
0386fde322 Update aws-java-sdk-s3 to 1.11.563 (#31) 2019-05-31 09:34:44 +01:00
Scala Steward
602c5ef150 Update aws-java-sdk-s3 to 1.11.562 (#30) 2019-05-30 15:12:34 +01:00
Scala Steward
5011779007 Update aws-java-sdk-s3 to 1.11.561 (#26) 2019-05-29 09:42:40 +01:00
082babb94d
Use multi-part upload for large files (i.e. files > 5Mb) (#22)
* [ThorpS3Client] Extract QuoteStripper and S3ClientObjectLister

* [ThorpS3Client] Extract S3ClientUploader

* [ThorpS3Client] Extract S3ClientCopier

* [ThorpS3Client] Extract S3ClientDeleter

* [ThropS3Client] Can select upload strategy based on file size

Currently switches to an alternate that is a clone of the original
method.

* [MD5HashGenerator] Add md5FilePart

Reimplement md5File using md5FilePart

* [MyS3CatsIOClient] extracted

* [S3ClientMultiPartUploader] add tests for accept def

* [S3ClientMultiPartUploader] initiate multi-part upload

* [Md5HashGenerator] add tests reading part of a file = failing test

* [Md5HashGenerator] fix when reading part of a file

* [S3ClientMultiPartUploader] create UploadPartRequests

* [S3ClientMultiPartUploader] uploadPart delegates to an S3Client

* [S3ClientMultiPartUploader] uploadParts uploads each part

* [S3ClientMultiPartUploader] complete upload should completeUpload

* [S3ClientMultiPartUploader] upload file tests when all okay

* [S3ClientMultiPartUploader] Use Recording client in component tests

* [s3ClientMultiPartUploader] remove unused variable

* [S3ClientMultiPartUploader] failing test for init upload error

* [S3ClientMultiPartUploader] Handle errors during multi-part upload

* [S3ClientMultiPartUploader] Retry uploads

* [S3Action] ErroredS4Action now holds the error

* [S3ClientMultiPartUploader] Add logging

* [S3ClientMultiPartUploader] Display warning messages

* [S3ClientMultiPartUploader] test creation of CreateMulitpartUploadRequest

* [S3ClientMultiPartUploader] specify bucket in UploadPartRequest

* [S3ClientMultiPartUploader] verify complete request has upload id

* [S3ClientMultiPartUploader] verify abort request contains upload id

* [S3ClientMultiPartUploader] add logging around retry errors

* [S3ClientMultiPartUploader] verify upload part request had remote key

* [S3ClientMultipartuploaderLogging] refactoring/rewriting strings

* [S3ClientMultiPartUploader] add bucket to abort request

* [S3ClientMultiPartUploader] part numbers must start at 1

* [S3ClientMultiPartUploader] fix capitalisation in comment

* [Config] define maxRetries

* [S3ClientMultiPartUploader] abort request should have the remote key

* [S3ClientMultiPartUploader] display remote key properly

* [S3ClientMultiPartUploader] rename method for plural parts

* [S3ClientMultiPartUploader] log hash and part number

* [MD5HashGenerator] support creating hash from a byte array

* [sbt] add aws-java-sdk-s3 (v1) for multi-part uploads

The reactive-aws-s3-* library is based on the V2 of the Java library,
which doesn't support multi-part uploads.

* [S3ClientMultiPartUploader] use Amazon S3 Client (from v1 sdk)

* [S3ClientMultiPartUploader] include file and offset in upload part request

* {S3ClientMultiPartUploader] Add part etags to complete request

* [S3ClientMultiPartUploader] Use withers to create requests

* [S3ClientMultiPartUploader] don't bounce responses to tags when client accepts then as is

* [MD5HashGenerator] use MD5Hash

* [S3ClientMultiPartUploader] include hash in sending log message

* [S3ClientMultiPartUploader] tests throw correct exception

* [S3ClientMultiPartUploader] Include returned hash in error and log when send is finished

* [S3ClientUploader] Extract as trait, renaming implementations

* [S3Client] upload def now requires tryCount

* [S3ClientUploader] add accepts to trait

* [S3ClientMultiPartUploaderSuite] remove ambiguity over class import

* [S3ClientMultiPartTransferManager] implement and use
2019-05-27 20:37:59 +01:00
a5311fec72 [sbt] remote unused fs2 Stream dependency 2019-05-24 07:49:19 +01:00
Scala steward
f296fc05f2
Update slf4j-log4j12 to 1.7.26 2019-05-14 04:45:44 +02:00
5b05022d92 [sbt] Add scala-logging 3.9.2 / slf4j-log4j12 1.7.25 as dependencies
As a minimum this should clear the no logger/appender errors.
2019-05-11 19:35:26 +01:00
38edf68f43 [sbt] add reactive-aws-s3 1.1.3 as dependency 2019-05-07 08:58:22 +01:00
223d6b9392 [sbt] delegate cats-effect to fs2 transient version
Add comments and adjust order to libraryDependencies
2019-05-06 17:31:48 +01:00
afab304bae Revert "[sbt] Bump cats-effect from 1.2.0 to 1.3.0"
This reverts commit 88d42e2130.
2019-05-06 17:25:29 +01:00
0011028c59 [sbt] add fs2-{core,io} 1.0.4 as dependencies 2019-05-06 17:15:43 +01:00
8a18171d7d [main] initial parsing of command line args 2019-05-06 17:15:43 +01:00