kemitix/thorp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
thorp/build.sbt

34 lines
960 B
Text
Raw Normal View History

Initial commit
2019-04-29 20:05:41 +01:00
name := "s3thorp"
version := "0.1"
[sbt] Add scalatest 3.0.7 as test dependency
2019-04-29 20:19:04 +01:00
scalaVersion := "2.12.8"
[sbt] delegate cats-effect to fs2 transient version Add comments and adjust order to libraryDependencies
2019-05-06 17:31:48 +01:00
// command line arguments parser
[main] initial parsing of command line args
2019-05-06 10:50:41 +01:00
libraryDependencies += "com.github.scopt" %% "scopt" % "4.0.0-RC2"
[sbt] Add cats-effect 1.2.0 as dependency
2019-04-29 20:19:21 +01:00
[sbt] add reactive-aws-s3 1.1.3 as dependency
2019-05-07 08:24:56 +01:00
// AWS SDK
Use multi-part upload for large files (i.e. files > 5Mb) (#22) * [ThorpS3Client] Extract QuoteStripper and S3ClientObjectLister * [ThorpS3Client] Extract S3ClientUploader * [ThorpS3Client] Extract S3ClientCopier * [ThorpS3Client] Extract S3ClientDeleter * [ThropS3Client] Can select upload strategy based on file size Currently switches to an alternate that is a clone of the original method. * [MD5HashGenerator] Add md5FilePart Reimplement md5File using md5FilePart * [MyS3CatsIOClient] extracted * [S3ClientMultiPartUploader] add tests for accept def * [S3ClientMultiPartUploader] initiate multi-part upload * [Md5HashGenerator] add tests reading part of a file = failing test * [Md5HashGenerator] fix when reading part of a file * [S3ClientMultiPartUploader] create UploadPartRequests * [S3ClientMultiPartUploader] uploadPart delegates to an S3Client * [S3ClientMultiPartUploader] uploadParts uploads each part * [S3ClientMultiPartUploader] complete upload should completeUpload * [S3ClientMultiPartUploader] upload file tests when all okay * [S3ClientMultiPartUploader] Use Recording client in component tests * [s3ClientMultiPartUploader] remove unused variable * [S3ClientMultiPartUploader] failing test for init upload error * [S3ClientMultiPartUploader] Handle errors during multi-part upload * [S3ClientMultiPartUploader] Retry uploads * [S3Action] ErroredS4Action now holds the error * [S3ClientMultiPartUploader] Add logging * [S3ClientMultiPartUploader] Display warning messages * [S3ClientMultiPartUploader] test creation of CreateMulitpartUploadRequest * [S3ClientMultiPartUploader] specify bucket in UploadPartRequest * [S3ClientMultiPartUploader] verify complete request has upload id * [S3ClientMultiPartUploader] verify abort request contains upload id * [S3ClientMultiPartUploader] add logging around retry errors * [S3ClientMultiPartUploader] verify upload part request had remote key * [S3ClientMultipartuploaderLogging] refactoring/rewriting strings * [S3ClientMultiPartUploader] add bucket to abort request * [S3ClientMultiPartUploader] part numbers must start at 1 * [S3ClientMultiPartUploader] fix capitalisation in comment * [Config] define maxRetries * [S3ClientMultiPartUploader] abort request should have the remote key * [S3ClientMultiPartUploader] display remote key properly * [S3ClientMultiPartUploader] rename method for plural parts * [S3ClientMultiPartUploader] log hash and part number * [MD5HashGenerator] support creating hash from a byte array * [sbt] add aws-java-sdk-s3 (v1) for multi-part uploads The reactive-aws-s3-* library is based on the V2 of the Java library, which doesn't support multi-part uploads. * [S3ClientMultiPartUploader] use Amazon S3 Client (from v1 sdk) * [S3ClientMultiPartUploader] include file and offset in upload part request * {S3ClientMultiPartUploader] Add part etags to complete request * [S3ClientMultiPartUploader] Use withers to create requests * [S3ClientMultiPartUploader] don't bounce responses to tags when client accepts then as is * [MD5HashGenerator] use MD5Hash * [S3ClientMultiPartUploader] include hash in sending log message * [S3ClientMultiPartUploader] tests throw correct exception * [S3ClientMultiPartUploader] Include returned hash in error and log when send is finished * [S3ClientUploader] Extract as trait, renaming implementations * [S3Client] upload def now requires tryCount * [S3ClientUploader] add accepts to trait * [S3ClientMultiPartUploaderSuite] remove ambiguity over class import * [S3ClientMultiPartTransferManager] implement and use
2019-05-27 20:37:59 +01:00
/// wraps the in-preview Java SDK V2 which is incomplete and doesn't support multi-part uploads
[sbt] add reactive-aws-s3 1.1.3 as dependency
2019-05-07 08:24:56 +01:00
libraryDependencies += "com.github.j5ik2o" %% "reactive-aws-s3-core" % "1.1.3"
libraryDependencies += "com.github.j5ik2o" %% "reactive-aws-s3-cats" % "1.1.3"
Use multi-part upload for large files (i.e. files > 5Mb) (#22) * [ThorpS3Client] Extract QuoteStripper and S3ClientObjectLister * [ThorpS3Client] Extract S3ClientUploader * [ThorpS3Client] Extract S3ClientCopier * [ThorpS3Client] Extract S3ClientDeleter * [ThropS3Client] Can select upload strategy based on file size Currently switches to an alternate that is a clone of the original method. * [MD5HashGenerator] Add md5FilePart Reimplement md5File using md5FilePart * [MyS3CatsIOClient] extracted * [S3ClientMultiPartUploader] add tests for accept def * [S3ClientMultiPartUploader] initiate multi-part upload * [Md5HashGenerator] add tests reading part of a file = failing test * [Md5HashGenerator] fix when reading part of a file * [S3ClientMultiPartUploader] create UploadPartRequests * [S3ClientMultiPartUploader] uploadPart delegates to an S3Client * [S3ClientMultiPartUploader] uploadParts uploads each part * [S3ClientMultiPartUploader] complete upload should completeUpload * [S3ClientMultiPartUploader] upload file tests when all okay * [S3ClientMultiPartUploader] Use Recording client in component tests * [s3ClientMultiPartUploader] remove unused variable * [S3ClientMultiPartUploader] failing test for init upload error * [S3ClientMultiPartUploader] Handle errors during multi-part upload * [S3ClientMultiPartUploader] Retry uploads * [S3Action] ErroredS4Action now holds the error * [S3ClientMultiPartUploader] Add logging * [S3ClientMultiPartUploader] Display warning messages * [S3ClientMultiPartUploader] test creation of CreateMulitpartUploadRequest * [S3ClientMultiPartUploader] specify bucket in UploadPartRequest * [S3ClientMultiPartUploader] verify complete request has upload id * [S3ClientMultiPartUploader] verify abort request contains upload id * [S3ClientMultiPartUploader] add logging around retry errors * [S3ClientMultiPartUploader] verify upload part request had remote key * [S3ClientMultipartuploaderLogging] refactoring/rewriting strings * [S3ClientMultiPartUploader] add bucket to abort request * [S3ClientMultiPartUploader] part numbers must start at 1 * [S3ClientMultiPartUploader] fix capitalisation in comment * [Config] define maxRetries * [S3ClientMultiPartUploader] abort request should have the remote key * [S3ClientMultiPartUploader] display remote key properly * [S3ClientMultiPartUploader] rename method for plural parts * [S3ClientMultiPartUploader] log hash and part number * [MD5HashGenerator] support creating hash from a byte array * [sbt] add aws-java-sdk-s3 (v1) for multi-part uploads The reactive-aws-s3-* library is based on the V2 of the Java library, which doesn't support multi-part uploads. * [S3ClientMultiPartUploader] use Amazon S3 Client (from v1 sdk) * [S3ClientMultiPartUploader] include file and offset in upload part request * {S3ClientMultiPartUploader] Add part etags to complete request * [S3ClientMultiPartUploader] Use withers to create requests * [S3ClientMultiPartUploader] don't bounce responses to tags when client accepts then as is * [MD5HashGenerator] use MD5Hash * [S3ClientMultiPartUploader] include hash in sending log message * [S3ClientMultiPartUploader] tests throw correct exception * [S3ClientMultiPartUploader] Include returned hash in error and log when send is finished * [S3ClientUploader] Extract as trait, renaming implementations * [S3Client] upload def now requires tryCount * [S3ClientUploader] add accepts to trait * [S3ClientMultiPartUploaderSuite] remove ambiguity over class import * [S3ClientMultiPartTransferManager] implement and use
2019-05-27 20:37:59 +01:00
// AWS SDK - multi-part upload
libraryDependencies += "com.amazonaws" % "aws-java-sdk-s3" % "1.11.560"
[sbt] Add scala-logging 3.9.2 / slf4j-log4j12 1.7.25 as dependencies As a minimum this should clear the no logger/appender errors.
2019-05-11 19:07:30 +01:00
// Logging
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2"
Update slf4j-log4j12 to 1.7.26
2019-05-14 03:45:44 +01:00
libraryDependencies += "org.slf4j" % "slf4j-log4j12" % "1.7.26"
[sbt] Add scala-logging 3.9.2 / slf4j-log4j12 1.7.25 as dependencies As a minimum this should clear the no logger/appender errors.
2019-05-11 19:07:30 +01:00
[sbt] delegate cats-effect to fs2 transient version Add comments and adjust order to libraryDependencies
2019-05-06 17:31:48 +01:00
// testing
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.7" % "test"
// recommended for cats-effects
[sbt] Add cats-effect recommended scalaOptions
2019-05-05 19:28:52 +01:00
scalacOptions ++= Seq(
"-feature",
"-deprecation",
"-unchecked",
"-language:postfixOps",
"-language:higherKinds",
"-Ypartial-unification")
Reference in a new issue Copy permalink