* [changelog] updated
* [readme] updated
* [core] ConfigQuery added sources()
* [cli] ParseArgs allow specifying multiple sources
* [domain,core,cli] Source datatype changed to Path
* [domain] Sources added to hold multiple paths in order
* [domain] Config sources change datatype to Sources
* [core] Scan sources for .thorp.config and include any sources listed
This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.
* [core] ConfigurationBuilderTest add more pending tests
* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures
* [core] ConfigOptionTest use TemporaryFolder
* [core] ConfigOptionTest remove unused fields
* [cli] ParseArgsTest don't use get on an Option
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolder Move import to top of file
* [core] TemporaryFolder use Try over try-finally
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolders.withDirectory propogate errors
* [core] TemporaryFolders add writeFile and createFile
* [core] PlanBuilderTest create a plan with two sources with unique files in both
* [core] ActionGenerator only upload file by name in first source
create a plan
two sources
same filename in both
- only upload file in first source
* [domain] LastModified with no params is now()
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [domain] RemoteKey map to a file when prefix is empty
* [domain] S3ObjectData defaults to empty
* [core] KeyGenerator Avoid delimiter when empty prefix key
* [core] PlanBuilderTest when remote not in sources delete from remote
* [core] PlanBuilderTest extract helper md5Hash()
* [core] PlanBuilderTest one source a file no matching remote key
* [core] PlanBuildingTest file with matching key and hash do nothing
* [core] PlanBuilderTest file w/matching remote key and different hash
* [core] PlanBuilderTest a remote key with and without local file
* [core] DummyStorageService Use wildcards when selecting more than 6 elements
* [changelog] updated
* [core] Wrap Stream[LocalFile] as LocalFiles
* [core] LocalFiles counts files
* [core] LocalFiles sums file lengths
* [core] Restore logFileScan
* [storage-aws] Lister logs when fetching object summaries
* [storage-aws] Extract ListerLogger
* [core] Synchronise use leftMap
* [core] Syncronise extract assemblePlan
* [core] Wrap Stream[Action] in SyncPlan
* [core] Copy the file count and totalSizeBytes across to SyncPlan
* [cli] Program rename actions as syncPlan
* [cli] Program extract thorpArchive def
* [cli] Program extract createPlan def
* [cli] Program refactoring
* [cli] Program remove println showing version
* [cli] Program rename actions parameter as syncPlan
* [core] ThorpArchive add an index to each action
* [cli] Program make SyncPlan available to ThorpArchive
* [core] Pass SyncTotals to Archive
* [domain] Move SyncTotals into module
* [domain] Pass index and SyncTotals to UploadEventListener
* [domain] UploadEventLogger add file count a size progress bars
* [domain] UploadEventLogger better display stability and add file index
* [cli] Index files in correct order
* [cli] Program extends Synchronise
* [core] Rename Synchronise as PlanBuilder
* [cli] Program add test to check actions don't get reordered from plan
* [core] collect file size totals
* [domain] UploadEventLogger include percentage
* [cli] ProgramTest Use wildcards when selecting more than 6 elements
* [changelog] Updated
* [readme] Updated
* [domain] Config Add batch-mode flag
* [core] ConfigOption Add BatchMode option
* [core] ConfigQuery Add batchMode query
Also replaced verbose exists case clauses with a simple contains.
* [core] ConfigOptions added to replace Seq[ConfigOption]
* [core] Syncronise rename method to createPlan
* [cli] Program rename apply as run
* [storage-aws] S3StorageServiceBuilder stop using IO to create object
* [storage-aws] S3StorageServiceBuilder make default service lazy
* [storage-aws] Rename S3ClientCopier => Copier
* [storage-aws] Rename S3ClientDeleter => Deleter
* [storage-aws] Rename S3ClientObjectLister => Lister
* [storage-aws] Only attach upload listener when in batch mode
Only detects batch mode when selected as a command line option
* [core] Synchronise use leftMap rather than swap.map.swap
* [cli] ParseArgs add `-B` and `--batch` options to enable batch mode
* [core] ThorpArchive logs file uploaded when in batch mode
* [sbt] make name and version available within Scala
Available as `thorp.BuildInfo._`
* [core] ConfigOption Add Version option
* [cli] ParseArgs add command-line options `-V` and `--version`
* [core] ConfigQuery add showVersion query
* [cli] Program shows version then exit
* [cli] Main don't show "Done"
* [storage-aws] ETagGenerator add stub
* [core] MD5HashGenerator add hex and digest helpers
* [domain] MD5Hash can always provide base64 and also digest
Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.
MD5Hash is now cleaner now that it no longer has Option parameters.
* [core] MD5HashGenerator add stubs to allow reading file chunks
* [domain] MD5HashData add sub-objects
* [domain] MD5HashData move back into test where it belongs
* [sbt] add sbt-bloop plugin
* [domain] MD5HashData Add hash of big-file
* [domain] MD5HashData Add hash of big-file
* [core] MD5HashGenerator find end of chunk correctly
* [core] MD5HashGenerator offset is a Long
* [core] MD5HashGenerator don't read past the end of the file
* [storage-aws] ETagGenerator can reproduce ETags
* [storage-aws] ETagGeneratorTest added
* [storate-aws] ETagGenerator refactoring
* [storage-aws] ETageGenerator refactoring
* [core] SyncSuite remove redundant braces
* [storage-api] HashService added
* [storage-aws] S3HashService added
* [core] LocalFileStream refactoring
* [core] integrate HashService and ETagGenerator
* Optimise imports
* [domain] HexEncoder added to replace java 8 only DataTypeConverter
* [core] MD5HashGenerator refactoring
* [core] S3MetaDataEnricher refactoring
* [core] S3MetaDataEnricherSuite refactoring
* [storage-aws] ETagGeneratorTest refactoring
* [storage-aws] StorageServiceSuite refactoring
* [core] S3MetaDataEnricher refactoring
* [core] refactoring
* [storage-aws] refactoring
* [core,storage-{api,aws}] Allow Lister errors to be reported to user
* [cli,core,storage-*] Display simple error message when bucket is invalid
* [core] optimise imports
* [storate-aws] optimise imports
* [core] SyncSuite don't use get on an optional type
* [aws-lib] Uploader provide request with the already calculated md5 hash
* [aws-lib] remove unused accepts method
* [aws-lib] Uploader refactoring
* [domain] Config remove unused threshold and max retries items
* [core] Show upload errors in summary
* [domain] LocalFile add helper to explicitly compare by hash value
Looking to add an optional field to MD5Hash but we want to do our
checks here only on the hash value, not whether a digest is available
or not.
* [core] Sync refactoring
* [core] SyncSuite invoke subject inside it method and after declaring expectations
* [core] SyncSuite use the localfile hash rather than something arbitrary
* [cli] Add `--no-global` and `--no-user` options
* [core] LocalFileStream refactoring
* [core] SyncSuite: ignore user and global configuration files
* [domain] MD5Hash now can optionally store the base64 encoded hash
* [core] MD5HashGenerator pass the digest to MD5Hash
* [aws-lib] Uploader use the base64 encoded hash
* [changelog] updated
* [core] rename the config supplied from CLI as such
This distinguishes it as config supplied from the command line.
* [core] add ConfigOption
* [core] ConfigOption can update a Config
* [core] new validator for config
* [domain] Config doesn't validate source any more
* [cli] PrintLogger default to not print debug messages
* [cli] Use ConfigOptions and new ConfigValidator
* [sbt] Use common settings for project root
* [domain] RemoteKey can handle when prefix is empty
* [cli] remove banner
* [domain] Logger can create version of itself with debug flipped
* [core] Build and validate Config within core module
This means that the `thorp-lib` module can validate its input from a
list of `ConfigOption`s.
* [core] refactor ConfigurationBuilder
* [core] refactor ConfigurationBuilder
* [sbt] starting back from tagless-final by using IO where needed
* [core] Add ParseConfigFile
* [sbt] Make cats-effect available from the domain
* Roll back from tagless-final to just use cat-effect's IO
* [core] extract ParseConfigLines
* [core] ConfigurationBuilder rename apply as buildConfig
* [core] ParseConfig[Files,Lines] rename apply methods
* [core] refactor ParseConfigFile and add tests
* [core] Sync fix call to run
* [core] SyncSuite update tests to use ConfigOptions
* [aws-api] TTFI UploadProgressLogging#logTransfer
* [aws-api] TTFI UploadProgressLogging#logRequestCycle
* [aws-api] TTFI UploadProgressLogging#logByteTransfer
* [aws-api] TTFI UploadProgressListener
* [aws-lib] TTFI UploaderLogging
* [aws-api] UploadProgressListener refactoring
* [aws-api] UploadProgressListener remove IO/Monad
This class is a wrapper for a callback method used by the AWS SDK.
Unfortunately you can't get the IO() created when that listener is
called by the SDK, so unless we manually run unsafeRunSync, as we have
done previously, it would never be executed.
This removes the IO monad entirely and simply runs the code when the
callback is triggered.
* [aws-lib] S3ClientLogging remove unused method
* [aws-lib] TTFI S3ClientLogging
* [aws-lib] TTFI S3ClientCopier
* [aws-lib] TTFI S3ClientObjectLister
* [aws-lib] TTFI Uploader
* [aws-lib] TTFI S3ClientDeleter
* [aws-api] TTFI S3Client
* [aws-lib] TTFI S3ClientBuilder and ThorpS3Client
* [core] TTFI ActionSubmitter
* [cli] TTFI Logger
* [core] TTFI MD5HashGenerator
* [core] TTFI LocalFileStream
* [core] Sync refactoring
* [core] TTFI Sync
* [aws-lib] S3ObjectsByHashSuite truncate lastmodified to match Date
* [aws-lib] ThorpS3ClientSuite truncate lastmodified to match Date
* [core] MD5HashGeneratorTest switch to Id from IO
* [sbt] restrict cats-effect to cli module, cats-core elsewhere
* [core] MD5HashGenerator collapse lines
* [domain] SizeTranslation includes decimals for larger sizes
* [core] MD5HashGenerator rewrite for memory efficiency
No longer attempt to create an Array the size of the file to be
parsed.
Now it creates a single small buffer and reads 8kb chunks in at a
time. Only creating an additional smaller buffer to read the tail of
the file.
Remove methods to parsing only part of a file are they were no longer
used, and remove the relevant tests.
* [aws-lib] Uploader refactoring
* [aws-lib] Uploader remove redundant braces
* [aws-lib] Uploader start upload inside IO
The upload method starts the upload to S3, so should be within the IO.
* [core] log count of errors
* [aws-lib] Uploader handle errors and count them
* [aws-lib] fold S3ClientUploader trait into it's only implementation
This trait was only implemented by S3ClientTransferManager.
* [core] SyncLogging: more robust matching
No longer cares about parameters to case classes, just their types.
* [cli] Logger uses IO for log methods
* [aws-lib] remove 'transfer-manager'prefix and only show tryCount > 1
* [sbt,cli] remove log4j and scala-logging dependencies
* [domain] move QuoteStripper to Domain
Use it directly in MD5Hash to strip quotes from any input.
* [core] SyncLogging call info in proper context
If the IO.unit returned by the info calls isn't part of the chain that
is returned from the function, then the delayed IO action is never
called.
* [aws-lib] Display size in bytes of file being uploaded
* [core] call info in correct context
* [cli] call info in correct context
* [aws-lib] raise summary fetch message to info 1
* [cli] include correct level in info messages
* [aws-lib] S3ClientLogging adjust logging levels
* [aws-lib] display file sizes in english
* [aws-lib] ObjectLister use IO.bracket properly
* [aws-lib] Copier use IO.bracket properly
* [aws-lib] Deleter refactor
* [aws-lib] TransferManagerLogging remove unused methods
* [aws-lib] TransferManager refactor
* [aws-lib] TransferManager refactor
* [aws-lib] TransferManager displays log messages
Use the UploadProgressListener that was being ignored, and use
unsafeRunSync to execute the suspended effect within the IO[Unit].
Using unsafeRunSync is required to render the effects as the listener
returns Unit, meaning the suspended effects would be discarded.
* [domain] Extract SizeTranslation into module
* [aws-api] report bytes transferred in progress
* [core] fix calls to info
info now returns an IO already, so don't need to wrap it in one.
* [aws-lib] remove unused class
* [aws-lib] UploadProgress displays progress bar while uploading
* [aws-api] UploadProgressLogging optimise imports
* [aws-api] UploadProgressLogging rename variables
* [domain] add Terminal object
* [aws-api] UploadProgressLogging use console width and two lines
- Improved clearing of lines after progress bar
- Use console width for progress bar size
* [aws-lib] S3ClientLogging optimise imports
* [aws-lib] TransferManager clear line before logging
* [aws-lib] rename class as TransferManager
* [aws-lib] rename TransferManger as Uploader to not clash
We are using an AWS SDK class with the same name.
* [domain] rename Filter as Include
* [cli]ParseArgs allow exclude and include parameters to be repeated
* [core] don't include include/exclude details in logging
* [domain] combine Include and Exclude into Filter
Config now collect includes and Excludes into a single list and passed
each file to the Filter.isIncluded method, with the list of Filters,
to determine if a file should be included.
* [core] MD5HashGenerator uses IO to return where there is file IO
This required that LocalFile in the domain module no longer be
supplied with a function to convert a File into an MD5Hash. Because
such a function requires reading the file it now must use IO, which we
don't allow in the domain module.
Unfortunate ripple effects out to users of MD5HashGenerator and
LocalFile.
* [aws-lib] Add own copy of test class MD5HashData
* [core] close files after calculating their MD5 hash
FileInputStream was never closed, so eventually ran into
ToManyFilesOpen.
Will come back to look at this again with IO.bracket for better
guarantee that FIS is closed.
Signed-off-by: Paul Campbell <pcampbell@kemitix.net>
* [aws-lib] Fetch all MD5 hashes under prefix
Initial request only returns the first 1000.
* [sbt] add scalamock as a test dependency
* [aws-lib]SyncSuite: minor layout changes
* [aws-lib]SyncSuite: remove test
* [core] move SyncSuite to same module as subject it tests
* [aws-lib]ThorpS3Client: remove commented lines
* [aws-lib] remove PutObject versions of Uploader
* [aws-lib] rename to TransferManager to remove Multi-part from name
* [aws-lib]TransferManager: change logging prefix
* [aws-lib] convert logging classes to objects
* [aws-lib] convert ObjectLister to use V1 SDK
* [aws-lib] convert Copier to use V1 SDK
* [aws-lib] extract S3ObjectsBy{Hash,Key} to objects
* [aws-lib]S3ClientSuite: rewrite test using mocks
* [aws-lib]TransferManager rewrite using for-comprehension
* [aws-lib]Copier: remote bucket name from target remote key
* [aws-lib]TransferManager: refactor logging to use IO themselves
* [aws-lib] Remove test class MyAmazonS3
* [aws-lib]ObjectLister: optimise imports
* [aws-lib] S3ClientSuite remove commented code
* [aws-lib]ThropS3ClientSuite update to V1 api
* [aws-lib]S3ClientSuite: make test as pending
It works okay on its own, but when run as part of a suite it fails.
Will look at this again once all V2 SDK is removed.
* [aws-lib] convert Deleter to use V1 SDK
* [aws-lib] Client Logging remove redundant braces
* [aws-lib] stop injecting the V2 SDK
* [sbt] remove v2 SDK dependencies
* [aws-lib] remove redundant helpers for v2 SDK
* [sbt] upgrade aws jackson dependencies
The jackson libraries used by AWS have security flaws, but are Java 6
compatible, which AWS want to preserve.
* [aws-lib] clean up TransferManager tests
* [core] convert QuoteStripper to an object and move to core
* [aws-lib]S3ClientUploader: use case matching instead of else if blocks
* [aws-lib] put imports at top of file
* [domain] remove redundant braces after class definition
* [aws-lib] remove redundant braces after class definition
* [core] avoid using head on a collection
* [sbt] define existing single module project as legacyRoot
* [sbt] add empty cli module depending on legacyRoot
* [cli] move Main to cli module
* [cli] move ParseArgs to cli module
* [sbt] limit scope of scopt dependency to cli module
* [cli] moved logging config to cli module
* [cli] rename module directory
* [aws-api] added empty module
* [sbt] aggregate builds from cli
* [aws-lib] add empty module
* [core] add empty module
* [sbt] add comment graphing module dependencies
* [sbt] adjust module dependencies to reflect plan
Include legacyRoot at the base until it can be redistributed
* [legacy] make some awssdk classes non-private
during this transition, these classes being private would cause problems
* [aws-lib] create S3ClientBuilder
This is copied from the legacy S3Client companion object
* [domain] add empty module
* [domain] move Bucket into module
* [legacy] RemoteKey no longer has dependency on Config
* [domain] move RemoteKey into module
* [domain] move MD5Hash into module
* [legacy] LocalFile no longer had dependency on MD5HashGenerator
* [domain] move LocalFile into module
* [domain] mode LastModified into module
* [domain] move RemoteMetaData into module
* [domain] move S3MetaData into module
* [domain] move Exclude into module
* [domain] move Filter into module
* [domain] move KeyModified into module
* [domain] move HashModified into module
* [domain] RemoteKey.resolve added
* [domain] add dependency on scalatest
* [domain] LocalFile.resolve added
* [legacy] Remove UnitTest
* [legacy] optimise imports
* [domain] move S3ObjectsData moved into module
* [legacy] wrapper for using GeneralProgressListener
* [domain] move Config into module
* [sbt] move aws-api below legacyRoot in dependencies
This will allow use to move S3Client into the aws-api module
* [legacy] rename S3Client companion as S3ClientBuilder
Preparation to move this into its own file.
* Inject Logger via CLI (#34)
* [S3Client] refactor defaultClient()
* [S3Client] transfermanager explicitly uses the same s3client
* [S3ClientPutObjectUploader] refactor putObjectRequest creation
* [cli] copy in Logging trait as Logger class
* [cli] Main uses Logger
* [cli] simplify Logger and pass to Sync.run
* [legacy] SyncLogging converted to companion
* [cli] Logger info can more easily use levels again
* [legacy] LocalFileStream uses injected info
* [legacy] S3MetaDataEnricher remove unused Logging
* [legacy] ActionGenerator remove unused Logging
* [legacy] convert ActionGenerator to an object
* [legacy] import log methods from SyncLogging
* [legacy] move getS3Status from S3Client to S3MetaDataEnricher
* [legact] convert ActionsSubmitter to an object
* [legacy] convert LocalFileStream to an object
* [legacy] move Action case classes inside companion
* [legacy] move UploadEvent case classes inside companion and rename
* [legacy] move S3Action case classes into companion
* [legacy] convert Sync to an object
* [cli] Logger takes verbosity level at construction
No longer needs to be passed the whole Config implicitly for each info
call.
* [legacy] stop passing implicit Config for logging purposes
Pass a more specific implicit info: Int => String => Unit instead
* [legacy] remove DummyS3Client
* [legacy] remove Logging
* [legacy] convert MD5HashGenerator to an object
* [aws-api] move S3Client into module
* [legacy] convert KeyGenerator to an object
* [legacy] don't use IO.unsafeRunSync directly
* [legacy] refactor/rewrite Sync.run
* [legacy] Rewrite sort using a for-comprehension
* [legacy] Sync inline sorting
* [legacy] SyncLogging rename method
* [legacy] repair tests
* [sbt] move core module to a dependency of legacyRoot
* [sbt] add test dependencies to core module
* [core] move classes into module
* [aws-lib] move classes into module
* [sbt] remove legacy root
