Paul Campbell
082babb94d
* [ThorpS3Client] Extract QuoteStripper and S3ClientObjectLister
* [ThorpS3Client] Extract S3ClientUploader
* [ThorpS3Client] Extract S3ClientCopier
* [ThorpS3Client] Extract S3ClientDeleter
* [ThropS3Client] Can select upload strategy based on file size
Currently switches to an alternate that is a clone of the original
method.
* [MD5HashGenerator] Add md5FilePart
Reimplement md5File using md5FilePart
* [MyS3CatsIOClient] extracted
* [S3ClientMultiPartUploader] add tests for accept def
* [S3ClientMultiPartUploader] initiate multi-part upload
* [Md5HashGenerator] add tests reading part of a file = failing test
* [Md5HashGenerator] fix when reading part of a file
* [S3ClientMultiPartUploader] create UploadPartRequests
* [S3ClientMultiPartUploader] uploadPart delegates to an S3Client
* [S3ClientMultiPartUploader] uploadParts uploads each part
* [S3ClientMultiPartUploader] complete upload should completeUpload
* [S3ClientMultiPartUploader] upload file tests when all okay
* [S3ClientMultiPartUploader] Use Recording client in component tests
* [s3ClientMultiPartUploader] remove unused variable
* [S3ClientMultiPartUploader] failing test for init upload error
* [S3ClientMultiPartUploader] Handle errors during multi-part upload
* [S3ClientMultiPartUploader] Retry uploads
* [S3Action] ErroredS4Action now holds the error
* [S3ClientMultiPartUploader] Add logging
* [S3ClientMultiPartUploader] Display warning messages
* [S3ClientMultiPartUploader] test creation of CreateMulitpartUploadRequest
* [S3ClientMultiPartUploader] specify bucket in UploadPartRequest
* [S3ClientMultiPartUploader] verify complete request has upload id
* [S3ClientMultiPartUploader] verify abort request contains upload id
* [S3ClientMultiPartUploader] add logging around retry errors
* [S3ClientMultiPartUploader] verify upload part request had remote key
* [S3ClientMultipartuploaderLogging] refactoring/rewriting strings
* [S3ClientMultiPartUploader] add bucket to abort request
* [S3ClientMultiPartUploader] part numbers must start at 1
* [S3ClientMultiPartUploader] fix capitalisation in comment
* [Config] define maxRetries
* [S3ClientMultiPartUploader] abort request should have the remote key
* [S3ClientMultiPartUploader] display remote key properly
* [S3ClientMultiPartUploader] rename method for plural parts
* [S3ClientMultiPartUploader] log hash and part number
* [MD5HashGenerator] support creating hash from a byte array
* [sbt] add aws-java-sdk-s3 (v1) for multi-part uploads
The reactive-aws-s3-* library is based on the V2 of the Java library,
which doesn't support multi-part uploads.
* [S3ClientMultiPartUploader] use Amazon S3 Client (from v1 sdk)
* [S3ClientMultiPartUploader] include file and offset in upload part request
* {S3ClientMultiPartUploader] Add part etags to complete request
* [S3ClientMultiPartUploader] Use withers to create requests
* [S3ClientMultiPartUploader] don't bounce responses to tags when client accepts then as is
* [MD5HashGenerator] use MD5Hash
* [S3ClientMultiPartUploader] include hash in sending log message
* [S3ClientMultiPartUploader] tests throw correct exception
* [S3ClientMultiPartUploader] Include returned hash in error and log when send is finished
* [S3ClientUploader] Extract as trait, renaming implementations
* [S3Client] upload def now requires tryCount
* [S3ClientUploader] add accepts to trait
* [S3ClientMultiPartUploaderSuite] remove ambiguity over class import
* [S3ClientMultiPartTransferManager] implement and use
33 lines
960 B
Scala
33 lines
960 B
Scala
name := "s3thorp"
|
|
|
|
version := "0.1"
|
|
|
|
scalaVersion := "2.12.8"
|
|
|
|
// command line arguments parser
|
|
libraryDependencies += "com.github.scopt" %% "scopt" % "4.0.0-RC2"
|
|
|
|
// AWS SDK
|
|
/// wraps the in-preview Java SDK V2 which is incomplete and doesn't support multi-part uploads
|
|
libraryDependencies += "com.github.j5ik2o" %% "reactive-aws-s3-core" % "1.1.3"
|
|
libraryDependencies += "com.github.j5ik2o" %% "reactive-aws-s3-cats" % "1.1.3"
|
|
|
|
// AWS SDK - multi-part upload
|
|
libraryDependencies += "com.amazonaws" % "aws-java-sdk-s3" % "1.11.560"
|
|
|
|
// Logging
|
|
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2"
|
|
libraryDependencies += "org.slf4j" % "slf4j-log4j12" % "1.7.26"
|
|
|
|
// testing
|
|
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.7" % "test"
|
|
|
|
// recommended for cats-effects
|
|
scalacOptions ++= Seq(
|
|
"-feature",
|
|
"-deprecation",
|
|
"-unchecked",
|
|
"-language:postfixOps",
|
|
"-language:higherKinds",
|
|
"-Ypartial-unification")
|
|
|