Commit graph

67 commits

Author SHA1 Message Date
Scala Steward
0386fde322 Update aws-java-sdk-s3 to 1.11.563 (#31) 2019-05-31 09:34:44 +01:00
Scala Steward
602c5ef150 Update aws-java-sdk-s3 to 1.11.562 (#30) 2019-05-30 15:12:34 +01:00
Scala Steward
5011779007 Update aws-java-sdk-s3 to 1.11.561 (#26) 2019-05-29 09:42:40 +01:00
082babb94d
Use multi-part upload for large files (i.e. files > 5Mb) (#22)
* [ThorpS3Client] Extract QuoteStripper and S3ClientObjectLister

* [ThorpS3Client] Extract S3ClientUploader

* [ThorpS3Client] Extract S3ClientCopier

* [ThorpS3Client] Extract S3ClientDeleter

* [ThropS3Client] Can select upload strategy based on file size

Currently switches to an alternate that is a clone of the original
method.

* [MD5HashGenerator] Add md5FilePart

Reimplement md5File using md5FilePart

* [MyS3CatsIOClient] extracted

* [S3ClientMultiPartUploader] add tests for accept def

* [S3ClientMultiPartUploader] initiate multi-part upload

* [Md5HashGenerator] add tests reading part of a file = failing test

* [Md5HashGenerator] fix when reading part of a file

* [S3ClientMultiPartUploader] create UploadPartRequests

* [S3ClientMultiPartUploader] uploadPart delegates to an S3Client

* [S3ClientMultiPartUploader] uploadParts uploads each part

* [S3ClientMultiPartUploader] complete upload should completeUpload

* [S3ClientMultiPartUploader] upload file tests when all okay

* [S3ClientMultiPartUploader] Use Recording client in component tests

* [s3ClientMultiPartUploader] remove unused variable

* [S3ClientMultiPartUploader] failing test for init upload error

* [S3ClientMultiPartUploader] Handle errors during multi-part upload

* [S3ClientMultiPartUploader] Retry uploads

* [S3Action] ErroredS4Action now holds the error

* [S3ClientMultiPartUploader] Add logging

* [S3ClientMultiPartUploader] Display warning messages

* [S3ClientMultiPartUploader] test creation of CreateMulitpartUploadRequest

* [S3ClientMultiPartUploader] specify bucket in UploadPartRequest

* [S3ClientMultiPartUploader] verify complete request has upload id

* [S3ClientMultiPartUploader] verify abort request contains upload id

* [S3ClientMultiPartUploader] add logging around retry errors

* [S3ClientMultiPartUploader] verify upload part request had remote key

* [S3ClientMultipartuploaderLogging] refactoring/rewriting strings

* [S3ClientMultiPartUploader] add bucket to abort request

* [S3ClientMultiPartUploader] part numbers must start at 1

* [S3ClientMultiPartUploader] fix capitalisation in comment

* [Config] define maxRetries

* [S3ClientMultiPartUploader] abort request should have the remote key

* [S3ClientMultiPartUploader] display remote key properly

* [S3ClientMultiPartUploader] rename method for plural parts

* [S3ClientMultiPartUploader] log hash and part number

* [MD5HashGenerator] support creating hash from a byte array

* [sbt] add aws-java-sdk-s3 (v1) for multi-part uploads

The reactive-aws-s3-* library is based on the V2 of the Java library,
which doesn't support multi-part uploads.

* [S3ClientMultiPartUploader] use Amazon S3 Client (from v1 sdk)

* [S3ClientMultiPartUploader] include file and offset in upload part request

* {S3ClientMultiPartUploader] Add part etags to complete request

* [S3ClientMultiPartUploader] Use withers to create requests

* [S3ClientMultiPartUploader] don't bounce responses to tags when client accepts then as is

* [MD5HashGenerator] use MD5Hash

* [S3ClientMultiPartUploader] include hash in sending log message

* [S3ClientMultiPartUploader] tests throw correct exception

* [S3ClientMultiPartUploader] Include returned hash in error and log when send is finished

* [S3ClientUploader] Extract as trait, renaming implementations

* [S3Client] upload def now requires tryCount

* [S3ClientUploader] add accepts to trait

* [S3ClientMultiPartUploaderSuite] remove ambiguity over class import

* [S3ClientMultiPartTransferManager] implement and use
2019-05-27 20:37:59 +01:00
a5311fec72 [sbt] remote unused fs2 Stream dependency 2019-05-24 07:49:19 +01:00
Scala steward
f296fc05f2
Update slf4j-log4j12 to 1.7.26 2019-05-14 04:45:44 +02:00
5b05022d92 [sbt] Add scala-logging 3.9.2 / slf4j-log4j12 1.7.25 as dependencies
As a minimum this should clear the no logger/appender errors.
2019-05-11 19:35:26 +01:00
38edf68f43 [sbt] add reactive-aws-s3 1.1.3 as dependency 2019-05-07 08:58:22 +01:00
223d6b9392 [sbt] delegate cats-effect to fs2 transient version
Add comments and adjust order to libraryDependencies
2019-05-06 17:31:48 +01:00
afab304bae Revert "[sbt] Bump cats-effect from 1.2.0 to 1.3.0"
This reverts commit 88d42e2130.
2019-05-06 17:25:29 +01:00
0011028c59 [sbt] add fs2-{core,io} 1.0.4 as dependencies 2019-05-06 17:15:43 +01:00
8a18171d7d [main] initial parsing of command line args 2019-05-06 17:15:43 +01:00
9f7cb86c51 [sbt] Add cats-effect recommended scalaOptions 2019-05-06 17:15:43 +01:00
88d42e2130 [sbt] Bump cats-effect from 1.2.0 to 1.3.0
Added Sources and Javadoc
2019-05-06 17:15:43 +01:00
fb6d8073c4 [sbt] Add cats-effect 1.2.0 as dependency 2019-05-06 17:15:42 +01:00
f949ba0912 [sbt] Add scalatest 3.0.7 as test dependency 2019-05-06 17:15:39 +01:00
16af5cb333 Initial commit 2019-04-29 20:05:41 +01:00