* [BROKEN]Remove ZIO and EIP-ZIO
* [BROKEN]uishell.UIShell.receiver: implement
* [BROKEN]domain.Channel: new implementation replacing MessageChannel
* [BROKEN] use domain.Channel
* Shutdown filescanner channel when finished
* start the uiChannel
* domain.Channel: channel completes once shutdown and queue empty
* uishell: down use eraseLineForward in batch mode
* lib: set file runner before adding listener
* uishell: don’t log do nothing events when not in batch mode
* domain.Channel: if exception in child thread then shutdown channel
* uishell: use correct line endings for showing chosen actions
* domain.Channel: don’t wait for shutdown if not running
* domain: remove legacy MessageChannel
* domain.Channel: don’t hold thread array
* lib.LocalFileSystem: restore delete scanner
* lib.LocalFileSystem: shutdown deletetion channel
* domain.Channel: improved shutdown logic
* clean up
* uishell.UploadProgressEvent: convert to Java
* uishell.UIEvent: convert to Java
* uishell.ProgressUI: convert to Java
* uishell.ProgressEvent: remove unused
* lib.MessageChannel: added as replacement for eip-zio version
* domain: move MessageChannel to module
* storage: convert to Java
* Add root and parent pom.xml files
* parent: add scala-maven-plugin to pluginManagement
* fix packaging
* domain: add module
* parent,domain: add dependencies
* maven: add filesystem module
* parent: disable coverage and pitest
* domain: move classes from test to main as they are used in filesystem
* config: add module
* console: add module
* Add reactor-graph
* fix typo
* uishell: add module
* Restore explicit dependencies
Don’t depend upon transitive dependencies if we want to be able to restructure module dependencies.
* storage: add module
* lib: add module
Included adding a Resource case class in filesystem to replace one generated by a scala plugin
* storage-aws: add module
* cli: add module
* app: add module
* app: make jar runnable
* Updated build instructions in README
* Convert readme to markdown
* Add reactor graph to readme
* Switch Guthub Actions to using Maven to build
* [domain] Define Hashes in domain package
* [filesystem] Load and parse any .thorp.cache files found
* [filesystem] Use cached file data when available and up-to-date
* [lib] FileScanner refactoring
* [filesystem] scan sub-dirs first to minimise time cache is on heap
* [filesystem] Write new cache data to temp file
* [lib] replace cache file when finished updating
* [filesystem] AppendLines to correct file with new lines
* [domain] decode HashType from String
* [filesystem] Store last modified time as epoch milliseconds
* [filesystem] parse lastmodified as a long
* [filesystem] use all hash values in cache
* [lib] FileScanner rearrange code
* [lib] Create and use a single cache file per source
* [storage-aws] Use ETag hash from cache when available
* [filesystem] Merge file data together correctly
* [filesystem] Handle exceptions thrown by Files.mode correctly
* [readme] Add section on caching
* [changelog] updated
* [changelog] add pending dependencies notes
* [lib] Filters should not name methods after their defining object
* [lib] Fix up test
* [storage-aws] Refactoring
* [lib] Add test for FileScanner
* [lib] Add LocalFileSystemTest for scanCopyUpload
* [lib] Add LocalFileSystem tests for scanDelete
Also send a UIEvent for ActionChosen(ToDelete)
* [lib] LocalFileSystemTest can handle files in any order
* [lib] LocalFileSystemTest can handle files in any order #2
Don't include accountCounter or byteCounter are the order can change.
* [domain] Remove LocalFile.relativeToSource
* [sbt] Add missing modules to aggregate
* [sbt] Rename storage-api as storage
* [storage] remove dependency upon console
* [storage] remove dependency upon config
* [console] remove dependency upon config
* [sbt] Add app module
Make cli module actually cli, by moving CliArgs parser into it and
Main and Program into app.
* add app that depends on cli and thorp-lib
* move non-cli specific to app
* make cli depend on config
* make cli not depend on thorp-lib
* [sbt] make module dependencies more explicit
* make app depend on storage-aws
* make cli depend on filesystem's tests
* make thorp-lib depend on core
* make thorp-lib not depend on storage-aws
* make storage-aws not depend on core's tests
* make storage-aws depend on storage
* make storage-aws depend on filesystem and its tests
* make storage-aws depend on console
* make storage-aws depend on core
* make core depend on filesystem and its tests
* make filesystem depend on domain and its tests
* [sbt] merge thorp-lib with core as lib
* [sbt] add zio streams
* [lib] Add EIPTest
* [sbt] Allow NonUnitStatements
* [lib] EIPTest Message Channel rewritten using ZIO Stream
* [sbt] Add eip-zip 0.2.0 as dependency in lib
* Remove file counter and total upload size progress
Simplifying UnversionedMirrorArchive so we can create it before we
know what actions are needed.
* Fetch Remote Data before preparing any plans
* [domain] RemoteObjects only holds a single RemoteKey per Hash
Having multiple keys for a hash is redundant. They are only used to
create copy commands, and only one source remote key is needed for
that.
* [lib] Add a State trait
* [lib] Add FileScanner
* Add FileSystem.length(File)
* Add Clock to the Environment
* [domain] Sources update format
* [domain] Asking for a path that isn't in any Sources is fatal
There should never be any situation where are path not within a Source
is supplied. If there is, then something is badly wrong.
* [lib] Add test on use of zio.Ref
* [uishell] Add stub module
* [sbt] Upgrade eip-zio from 0.2.0 to 0.3.0
* [uishell] Add UIEvent stub
* [uishell] Add UIShell stub
* [sbt] Add eip-zio dependencies to app module
* [app] Wrap existing execution in simple point to point channel
* [uishell] Add UIEvent.ShowValidConfig
* [app] Remember to end the channel to allow prog to exit
* [app] purify environment for showValidConfig
* [app] Create type alias for pure effect free channel ref
* [app] Program refactoring
* [uishell] Add UIEvent.RemoteDataFetched
* [domain] Move Counters from lib
* [uishell] Add UIEvent.ShowSummary
* [lib] Add stub for PushLocalChanges
* [lib] Clean up FileScanner Environment types
* [lib] End channel after scanning files
* [lib] PushLocalChanges uses FileScanner
Scans files and sends them to a dummy receiver.
* [uishell] Add UIEvent.FileFound
* [lib] rename PushLocalChanges.apply as LocalFielSystem.scanCopyUpload
* [lib] FileScanner return LocalFile
* [domain] add length to LocalFile
* [domain] Add interogation queries to RemoteObjects
* [domain] Remove RemoteObject.keyForHashes
* [domain] RemoteObjects.remoteHasHash return the key and the hash
* [lib] LocalFileSystem.scanCopyUpload create Actions
* [domain] Move Action from lib
* [uishell] Log actions
* [lib] FileScanner respects Filters
* [lib] Create remoteKey for files correctly
* [lib] LocalFileSystem refactoring
* [lib] ThorpArchive.update doesn't need Console
* [uishell] Don't log choosen Action
* [uishell] Add UIEvent.ActionFinished
* [lib] LocalFileSystem refactoring
* [lib] Switch to using LocalFileSystem to do Copy/Upload
Todo or Broken:
- [ ] Delete actions don't happen
- [ ] Counters in summary are all zeros
* [lib] LocalFileStream display summary counters correctly
* [app] Restore ability to delete remote files
* [lib] LocalFileSystem deletes remote when local does NOT exist
* [filesystem] move hasLocalFile to FileSystem
* [filesystem] fix detection of local files from a RemoteKey
The configured Prefix wasn't being taken into account, meaning that
the expected local file for a RemoteKey was wrong.
* [filesystem] fix broken FileSystem test
* [domain] fix RemoteKey test
* [sbt] Upgrade eip-zio to 0.3.1 for zio-stream 1.0.0-RC12-1 compatibility
* [app] Program refactorting
* [lib] Remove unused class
* [lib] Remove test
* [uishell] Refactor large method
* [domain] rename S3MetaData as MatchedMetadata
* [domain] rename S3ObjectsData as RemoteObjects
* [core] ActionGenerator refactoring
* [core] ActionGenerator.createAction renamed and no longer a stream
* [core] ActionGenerator refactor
* [core] ActionGenerator Usage of head on collections
* [core] Extract Filters from domain.Filter
* [core] extract LocalFileValidator
* [domain] LocalFile remove unused isDirectory
* [domain] LocalFile move/rename relative as relativeToSource on companion
* [domain] LocalFile move and rename matches as matchesHash on companion
* [domain] LocalFile move md5base64 to companion
* [domain] Logger remove
* [domain] MD5Hash move hash to companion
* [domain] MD5Hash move digest to companion
* [domain] MD5Hash move hash64 to companion
* [domain] RemoteKey move class methods to companion
* [domain] Sources move forPath to companion
Led to being able to cleanup LocalFileStream.localFile and adding
LocalFiles.one
* [domain] UploadEventLogger rename method as apply
* [core] Convert HashService into an effect Hasher
* [core] LocalFileStreamSuite fix prep of test env
* [core] LocalFileStreamSuite Refactor
* [core] Add Hasher.hashObjectChunk
* [core] make MD5HashGenerator private
Moved MD5HashGenerator into package hasher and made it private to that
package, together with the Hasher effect which is the only permitted
user of it. Added hex and digest methods to Hasher.
Similar for the ETageGenerator in [storage-aws], moved into a hasher
package and made private. It can only be accessed using the S3Hasher
effect.
* [domain] Add Monoid
* [domain] Sources use Monoid
* [core] ActionGenerator Refactor
* [core] Use >>= and *> operators
* [config] Refactoring
* [storage-aws] Refactoring
* [domain] Refactoring
* [core] Refactoring
* [config] refactoring
* Refactoring
* [core] PlanBuilder Refactoring
* [core] Make PlanBuilder into an object
* [storate-aws] Copier Reduce the number of parameters
* [config] new module
* [config] stub module
* [domain] Rename domain.Config as domain.LegacyConfig
* [config] Move LegacyConfig to config module
* [config] Move config parsing and validation into module
* [config] Complete migration to module for Config
* [config] Config You should not name methods after their defining object
* [config] Rename LegacyConfig as Configuration
Also remove redundant uses
* [core] LocalFileStream Refactoring
* [changelog] update
* [sbt] Add ZIO dependency to storage-api
* Convert to use ZIO
* [sbt] remove cats-effect dependency
* [changelog] updated
* [cli] Program restore actions to correct order (copy, upload, delete)
* [cli] Program You should not name methods after their defining object
* [core] ConfigValidationException Redundant braces after class definition
* [core] LocalFileStreamSuite Usage of get on optional type.
* [core] PlanBuilderTest Usage of get on optional type.
* [domain] Drop use of Lenses macro and only create used lenses
* [core] Drop use of Lenses macro and only create used lenses
* [sbt] drop paradise plugin
* [changelog] updated
* [core] Add test file that we create incorrect hash for
By 'incorrect' we mean "not what AWS S3 think is correct" for this
file.
* [domain] HexEncoder rewritten to correctly decode hex
* [domain] HexEncoder encode to uppercase
* [sbt] Add monocle for support for lenses
* [domain] UploadEventListener make a case class
* [domain] Add @Lenses to all case classes
* [core] Add @Lenses to case classes
* [core] ActionGenerator use lense
* [core] ConfigOption use Lenses
* [core] ConfigurationBuilder remove unused fields
* [core] ConfigurationBuilder refactoring
* [core] SyncLogging use lenses
* [core] syncLogging refactoring
* [scalafmt] Add .scalafmt
* [sbt] Add unused import warning and make them errors
* [core] ConfigurationBuilder remove unused import
* [core] ConfigurationBuilder apply scalafmt
* [domain] reformat
* [storage-api] reformat
* [core] reformat and some refactoring
* [storage-aws] reformat
* [cli] reformat
* [core] post rebase fix up
* [storage-aws] UploaderSuite tidy up
* [domain] MD5Hash tweak
Without doing this we have a file that we don't create a valid md5
hash for. See #103
* [storage-aws] UploaderSuite remove unused import
* [storage-aws] StorageServiceSuite reformatted
* [sbt] consistent settings for all modules
Can't enable as stubbing TransferManager attempts to use the default
constructor for TransferManager.
* [storage-aws] Add AmazonTransferManager
This gives us an interface we can safely stub in unit tests without
the compiler failing the build because TransferManager's default
constructor is deprecated.
* [storage-aws] Add AmazonUpload
Prevents deprecation errors due to deprecated parameters on Transfer.
* [sbt] mode import to top of file
* [domain] Config defaults to an empty Sources list
* [core] ConfigurationBuilterTest add test for parsing .thorp.conf
* [core] ConfigQuery if no Sources given returns current dir
* [core] rewrote config loader
- Only settings from explicit sources are used
* [changelog] updated
* [core] Remove stray println statements
* [core] SyncLogging tidy up multi-source messages
* [changelog] updated
* [readme] updated
* [core] ConfigQuery added sources()
* [cli] ParseArgs allow specifying multiple sources
* [domain,core,cli] Source datatype changed to Path
* [domain] Sources added to hold multiple paths in order
* [domain] Config sources change datatype to Sources
* [core] Scan sources for .thorp.config and include any sources listed
This allows the inclusion of a `.thorp.config` file in a source with a
single line `source = ....` that causes that other source to also be
synched into the same remote prefix as the current source.
* [core] ConfigurationBuilderTest add more pending tests
* [[core] ConfigurationBuilderTest rewrite using loan-pattern for fixtures
* [core] ConfigOptionTest use TemporaryFolder
* [core] ConfigOptionTest remove unused fields
* [cli] ParseArgsTest don't use get on an Option
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolder Move import to top of file
* [core] TemporaryFolder use Try over try-finally
* [core] ConfigurationBuilderTest don't use get on Either
* [core] TemporaryFolders.withDirectory propogate errors
* [core] TemporaryFolders add writeFile and createFile
* [core] PlanBuilderTest create a plan with two sources with unique files in both
* [core] ActionGenerator only upload file by name in first source
create a plan
two sources
same filename in both
- only upload file in first source
* [domain] LastModified with no params is now()
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [core] PlanBuilderTest 2 sources w/remote only in 2nd src do nothing
* [domain] RemoteKey map to a file when prefix is empty
* [domain] S3ObjectData defaults to empty
* [core] KeyGenerator Avoid delimiter when empty prefix key
* [core] PlanBuilderTest when remote not in sources delete from remote
* [core] PlanBuilderTest extract helper md5Hash()
* [core] PlanBuilderTest one source a file no matching remote key
* [core] PlanBuildingTest file with matching key and hash do nothing
* [core] PlanBuilderTest file w/matching remote key and different hash
* [core] PlanBuilderTest a remote key with and without local file
* [core] DummyStorageService Use wildcards when selecting more than 6 elements
* [changelog] updated
* [core] Wrap Stream[LocalFile] as LocalFiles
* [core] LocalFiles counts files
* [core] LocalFiles sums file lengths
* [core] Restore logFileScan
* [storage-aws] Lister logs when fetching object summaries
* [storage-aws] Extract ListerLogger
* [core] Synchronise use leftMap
* [core] Syncronise extract assemblePlan
* [core] Wrap Stream[Action] in SyncPlan
* [core] Copy the file count and totalSizeBytes across to SyncPlan
* [cli] Program rename actions as syncPlan
* [cli] Program extract thorpArchive def
* [cli] Program extract createPlan def
* [cli] Program refactoring
* [cli] Program remove println showing version
* [cli] Program rename actions parameter as syncPlan
* [core] ThorpArchive add an index to each action
* [cli] Program make SyncPlan available to ThorpArchive
* [core] Pass SyncTotals to Archive
* [domain] Move SyncTotals into module
* [domain] Pass index and SyncTotals to UploadEventListener
* [domain] UploadEventLogger add file count a size progress bars
* [domain] UploadEventLogger better display stability and add file index
* [cli] Index files in correct order
* [cli] Program extends Synchronise
* [core] Rename Synchronise as PlanBuilder
* [cli] Program add test to check actions don't get reordered from plan
* [core] collect file size totals
* [domain] UploadEventLogger include percentage
* [cli] ProgramTest Use wildcards when selecting more than 6 elements
* [changelog] Updated
* [readme] Updated
* [domain] Config Add batch-mode flag
* [core] ConfigOption Add BatchMode option
* [core] ConfigQuery Add batchMode query
Also replaced verbose exists case clauses with a simple contains.
* [core] ConfigOptions added to replace Seq[ConfigOption]
* [core] Syncronise rename method to createPlan
* [cli] Program rename apply as run
* [storage-aws] S3StorageServiceBuilder stop using IO to create object
* [storage-aws] S3StorageServiceBuilder make default service lazy
* [storage-aws] Rename S3ClientCopier => Copier
* [storage-aws] Rename S3ClientDeleter => Deleter
* [storage-aws] Rename S3ClientObjectLister => Lister
* [storage-aws] Only attach upload listener when in batch mode
Only detects batch mode when selected as a command line option
* [core] Synchronise use leftMap rather than swap.map.swap
* [cli] ParseArgs add `-B` and `--batch` options to enable batch mode
* [core] ThorpArchive logs file uploaded when in batch mode
* [storage-aws] ETagGenerator add stub
* [core] MD5HashGenerator add hex and digest helpers
* [domain] MD5Hash can always provide base64 and also digest
Rather that store the base 64 digest some of the time, simply decode
it from the hex hash. The same for the binary digest.
MD5Hash is now cleaner now that it no longer has Option parameters.
* [core] MD5HashGenerator add stubs to allow reading file chunks
* [domain] MD5HashData add sub-objects
* [domain] MD5HashData move back into test where it belongs
* [sbt] add sbt-bloop plugin
* [domain] MD5HashData Add hash of big-file
* [domain] MD5HashData Add hash of big-file
* [core] MD5HashGenerator find end of chunk correctly
* [core] MD5HashGenerator offset is a Long
* [core] MD5HashGenerator don't read past the end of the file
* [storage-aws] ETagGenerator can reproduce ETags
* [storage-aws] ETagGeneratorTest added
* [storate-aws] ETagGenerator refactoring
* [storage-aws] ETageGenerator refactoring
* [core] SyncSuite remove redundant braces
* [storage-api] HashService added
* [storage-aws] S3HashService added
* [core] LocalFileStream refactoring
* [core] integrate HashService and ETagGenerator
* Optimise imports
* [domain] HexEncoder added to replace java 8 only DataTypeConverter
* [core] MD5HashGenerator refactoring
* [core] S3MetaDataEnricher refactoring
* [core] S3MetaDataEnricherSuite refactoring
* [storage-aws] ETagGeneratorTest refactoring
* [storage-aws] StorageServiceSuite refactoring
* [core] S3MetaDataEnricher refactoring
* [core] refactoring
* [storage-aws] refactoring
* [aws-lib] Uploader provide request with the already calculated md5 hash
* [aws-lib] remove unused accepts method
* [aws-lib] Uploader refactoring
* [domain] Config remove unused threshold and max retries items
* [core] Show upload errors in summary
* [domain] LocalFile add helper to explicitly compare by hash value
Looking to add an optional field to MD5Hash but we want to do our
checks here only on the hash value, not whether a digest is available
or not.
* [core] Sync refactoring
* [core] SyncSuite invoke subject inside it method and after declaring expectations
* [core] SyncSuite use the localfile hash rather than something arbitrary
* [cli] Add `--no-global` and `--no-user` options
* [core] LocalFileStream refactoring
* [core] SyncSuite: ignore user and global configuration files
* [domain] MD5Hash now can optionally store the base64 encoded hash
* [core] MD5HashGenerator pass the digest to MD5Hash
* [aws-lib] Uploader use the base64 encoded hash
* [changelog] updated
* [domain] rewrite and expand ansi codes available
* [aws-api] refactor progress-bar
* [domain] move progress-bar into Terminal
* [aws-api] remove unused value
* [domain] use unicode characters to get a smooth progress bar
* [core] rename the config supplied from CLI as such
This distinguishes it as config supplied from the command line.
* [core] add ConfigOption
* [core] ConfigOption can update a Config
* [core] new validator for config
* [domain] Config doesn't validate source any more
* [cli] PrintLogger default to not print debug messages
* [cli] Use ConfigOptions and new ConfigValidator
* [sbt] Use common settings for project root
* [domain] RemoteKey can handle when prefix is empty
* [cli] remove banner
* [domain] Logger can create version of itself with debug flipped
* [core] Build and validate Config within core module
This means that the `thorp-lib` module can validate its input from a
list of `ConfigOption`s.
* [core] refactor ConfigurationBuilder
* [core] refactor ConfigurationBuilder
* [sbt] starting back from tagless-final by using IO where needed
* [core] Add ParseConfigFile
* [sbt] Make cats-effect available from the domain
* Roll back from tagless-final to just use cat-effect's IO
* [core] extract ParseConfigLines
* [core] ConfigurationBuilder rename apply as buildConfig
* [core] ParseConfig[Files,Lines] rename apply methods
* [core] refactor ParseConfigFile and add tests
* [core] Sync fix call to run
* [core] SyncSuite update tests to use ConfigOptions
* [domain] SizeTranslation includes decimals for larger sizes
* [core] MD5HashGenerator rewrite for memory efficiency
No longer attempt to create an Array the size of the file to be
parsed.
Now it creates a single small buffer and reads 8kb chunks in at a
time. Only creating an additional smaller buffer to read the tail of
the file.
Remove methods to parsing only part of a file are they were no longer
used, and remove the relevant tests.
* [aws-lib] fold S3ClientUploader trait into it's only implementation
This trait was only implemented by S3ClientTransferManager.
* [core] SyncLogging: more robust matching
No longer cares about parameters to case classes, just their types.
* [cli] Logger uses IO for log methods
* [aws-lib] remove 'transfer-manager'prefix and only show tryCount > 1
* [sbt,cli] remove log4j and scala-logging dependencies
* [domain] move QuoteStripper to Domain
Use it directly in MD5Hash to strip quotes from any input.
* [core] SyncLogging call info in proper context
If the IO.unit returned by the info calls isn't part of the chain that
is returned from the function, then the delayed IO action is never
called.
* [aws-lib] Display size in bytes of file being uploaded
* [core] call info in correct context
* [cli] call info in correct context
* [aws-lib] raise summary fetch message to info 1
* [cli] include correct level in info messages
* [aws-lib] S3ClientLogging adjust logging levels
* [aws-lib] display file sizes in english
* [aws-lib] ObjectLister use IO.bracket properly
* [aws-lib] Copier use IO.bracket properly
* [aws-lib] Deleter refactor
* [aws-lib] TransferManagerLogging remove unused methods
* [aws-lib] TransferManager refactor
* [aws-lib] TransferManager refactor
* [aws-lib] TransferManager displays log messages
Use the UploadProgressListener that was being ignored, and use
unsafeRunSync to execute the suspended effect within the IO[Unit].
Using unsafeRunSync is required to render the effects as the listener
returns Unit, meaning the suspended effects would be discarded.
* [domain] Extract SizeTranslation into module
* [aws-api] report bytes transferred in progress
* [core] fix calls to info
info now returns an IO already, so don't need to wrap it in one.
* [aws-lib] remove unused class
* [aws-lib] UploadProgress displays progress bar while uploading
* [aws-api] UploadProgressLogging optimise imports
* [aws-api] UploadProgressLogging rename variables
* [domain] add Terminal object
* [aws-api] UploadProgressLogging use console width and two lines
- Improved clearing of lines after progress bar
- Use console width for progress bar size
* [aws-lib] S3ClientLogging optimise imports
* [aws-lib] TransferManager clear line before logging
* [aws-lib] rename class as TransferManager
* [aws-lib] rename TransferManger as Uploader to not clash
We are using an AWS SDK class with the same name.
* [domain] rename Filter as Include
* [cli]ParseArgs allow exclude and include parameters to be repeated
* [core] don't include include/exclude details in logging
* [domain] combine Include and Exclude into Filter
Config now collect includes and Excludes into a single list and passed
each file to the Filter.isIncluded method, with the list of Filters,
to determine if a file should be included.
* [core] MD5HashGenerator uses IO to return where there is file IO
This required that LocalFile in the domain module no longer be
supplied with a function to convert a File into an MD5Hash. Because
such a function requires reading the file it now must use IO, which we
don't allow in the domain module.
Unfortunate ripple effects out to users of MD5HashGenerator and
LocalFile.
* [aws-lib] Add own copy of test class MD5HashData
* [core] convert QuoteStripper to an object and move to core
* [aws-lib]S3ClientUploader: use case matching instead of else if blocks
* [aws-lib] put imports at top of file
* [domain] remove redundant braces after class definition
* [aws-lib] remove redundant braces after class definition
* [core] avoid using head on a collection
