kemitix/thorp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
thorp/CHANGELOG.org
Paul Campbell f35ea9795d
Create and use a cache of hashes for local files (#249) 
* [domain] Define Hashes in domain package

* [filesystem] Load and parse any .thorp.cache files found

* [filesystem] Use cached file data when available and up-to-date

* [lib] FileScanner refactoring

* [filesystem] scan sub-dirs first to minimise time cache is on heap

* [filesystem] Write new cache data to temp file

* [lib] replace cache file when finished updating

* [filesystem] AppendLines to correct file with new lines

* [domain] decode HashType from String

* [filesystem] Store last modified time as epoch milliseconds

* [filesystem] parse lastmodified as a long

* [filesystem] use all hash values in cache

* [lib] FileScanner rearrange code

* [lib] Create and use a single cache file per source

* [storage-aws] Use ETag hash from cache when available

* [filesystem] Merge file data together correctly

* [filesystem] Handle exceptions thrown by Files.mode correctly

* [readme] Add section on caching

* [changelog] updated

* [changelog] add pending dependencies notes

* [lib] Filters should not name methods after their defining object

* [lib] Fix up test
2019-10-27 19:53:00 +00:00

6.7 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[1.0.0] - ???

Added

  • Create and use a cache of hashes for local files (#249)

Dependencies

  • Revert "[sbt] Rollback sbt-ci-release to 1.3.2 (#231)"
  • Update sbt to 1.3.3 (#238)

[0.10.0] - 2019-10-08

This is the last v0.x feature release. The next feature release will be v1.x.

Fixed

  • Can't publish new releases to sonatype (#219)

Added

  • Accept parallel in config files (#230)

Changed

  • Reduce Progress widget from 3 lines to 2 (#221)
  • Perform delete operations in parallel (#223)

Dependencies

  • Update sbt-bloop to 1.3.3 (#224)
  • Update zio, zio-streams to 1.0.0-RC14 (#226)
  • Update sbt-bloop to 1.3.4 (#228)
  • Rollback sbt-ci-release to 1.3.2 (#231)

[0.9.1] - 2019-09-28

Fixed

  • Progress UI should be updated when upload fails (#218)

Dependencues

  • Update aws-java-sdk-s3 to 1.11.641 (#217)

[0.9.0] - 2019-09-27

Added

  • Support multiple parallel uploads (#188)

Dependencues

  • Update jackson-dataformat-cbor to 2.10.0 (#216)
  • Update jackson-databind to 2.10.0 (#214)
  • Update sbt-ci-release to 1.4.31 (#211)

[0.8.1] - 2019-09-23

Fixed

  • Not wrapping exceptions thrown in waitForUploadResult (#162)

Dependencies

  • Update jackson-dataformat-cbor to 2.9.10 (#208)
  • Update jackson-databind to 2.9.10 (#206)
  • Update sbt to 1.3.2 (#207)
  • Update zio, zio-streams to 1.0.0-RC13~ (#209)
  • Update aws-java-sdk-s3 to 1.11.635 (#203)
  • Update sbt to 1.3.1 (#204)
  • Update sbt-wartremover to 2.4.3 (#202)
  • Update sbt-ci-release to 1.3.2 (#200)

[0.8.0] - 2019-09-11

Added

  • Log copy and delete operations (#119)
  • Log errors with red label (#119)
  • Ensure file handles are always closed (#122)
  • [build] Install WartRemover plugin (#150)
  • [build] Add GitHub Action
  • [build] Add sbt-jacoco plugin for code coverage (#184)

Changed

  • [internal] Replace cats-effect with zio (#117)(#148)(#149)(#178)
  • [internal] Replace Monocle with local SimpleLens implementation (#121)
  • [internal] Don't use String as key in Map for hashes (#124)
  • [internal] Convert Storage to full ZIO effect module (#133)
  • [internal] Convert Config to full ZIO effect module (#134)
  • [internal] Create FileSystem ZIO effect module (#135)
  • [internal] Merge config package object with Config object (#138)
  • [internal] Merge FS object into FileSystem object (#139)
  • [internal] Assorted refactoring (#129)(#128)(#137)(#141)(#142)(#143)
  • [internal] case classes shouldn't be OO objects (#147)
  • [internal] Restructure using EIP-ZIO channels (#183)

Fixed

  • [progressbar] Reset progress bar for each file (#145)
  • Handle building large changesets better (#163)

Dependencies

  • Update aws-java-sdk-s3 from 1.11.587 to 1.11.610 (#118)(#144)(#158)
  • Update jackson-databind from 2.9.9.1 to 2.9.9.3 (#132)(#152)
  • Update scalamock from 4.3.0 to 4.4.0 (#165)
  • Upgrade to Scala 2.13.0 (#176)
  • Update sbt from 1.2.8 to 1.3.0 (#180)
  • Update sbt-ci-release from 1.2.6 to 1.3.1 (#186)

[0.7.2] - 2019-07-19

Changed

  • Apply scalafmt (#108)
  • Uses Lenses (#113)

Fixed

  • Creates incorrect MD5 hash for some files (#103)
  • NullPointerException creating scaladoc (#115)

[0.7.1] - 2019-07-15

Changed

  • Only settings in .thorp.conf for explicit sources are used (#111)

Fixed

  • Not reading .thorp.conf file (#110)(#111)

[0.7.0] - 2019-07-12

Added

  • Add a version command-line option (#99)
  • Add a batch mode (#85)
  • Display total size and progress for entire run (#94)
  • Sync more than one source directory into a single bucket/prefix (#25)

Dependencies

  • Upgrade jackson-databind from 2.9.9 to 2.9.9.1 (#101)
  • Upgrade aws-java-sdk-s3 from 1.11.580 to 1.11.587 (#104)
  • Upgrade sbt-assembly from 0.14.9 to 0.14.10 (#105)

[0.6.1] - 2019-07-03

Fixed

  • Release to sonatype using correct profile name (#102)

[0.6.0] - 2019-06-30

Added

  • Abstraction layer encapsulating S3 as Storage (#76)
  • Display simple error message when bucket is invalid (#81)
  • Shutdown storage service once completed (#88)
  • Release snapshots to sonatype from TravisCI (#92)
  • Embed launch script in fat-jar (#98)

Fixed

  • Use correct hash locally for comparing multi-part uploaded files (#82)
  • Handle large list of action in a stack-safe manner (#93)

Dependencies

  • Upgrade scalamock from 4.2.0 to 4.3.0 (#84)
  • Upgrade aws-java-sdk-s3 from 1.11.573 to 1.11.580 (#86)

[0.5.0] - 2019-06-21

Added

  • Add thorp-lib module (#66)
  • Enable running outside of sbt (#55)
  • -d, --debug flag for log messages (#60)
  • Read config from .thorp.conf in source directory (#71)
  • Read config from $HOME/.config/thorp.conf and /etc/thorp.conf (#73)
  • Add --no-global and --no-user options (#50)
  • Display any upload errors in summary report (#50)

Changed

  • Rename project as 'thorp' (#75)
  • Suppress Transfer event messages (#64)
  • Better error message when source not found (#51)
  • Reduced logging (#59)
  • Prevent AWS SDK from recalculating MD5 hash (#50)

Fixed

  • Error when calculating md5 hash for large files (#56)

Removed

  • -v verbosity flag (#63)

Dependencies

  • Upgrade aws-java-sdk-s3 from 1.11.569 to 1.11.570 (#57)

[0.4.0] - 2019-06-11

Added

  • Multi-part uploads
  • --include to select files for synchronisation
  • Upload progress bar

Changed

  • --filter renamed to -exclude

Fixed

  • Fetch md5 hashes for all remote objects, not just the first 1000
  • Handle when a file goes away between scanning and uploading

Dependencies

  • Removed reactive-aws-s3-{core,cats} and the AWS SDK v2 that it depended upon in favour of the AWS SDK v1
  • Upgrade aws-java-sdk-s3 from 1.11.560 to 1.11.569
  • Upgrade cats-effect from 1.2.0 to 1.3.1
  • Upgade scalatest from 3.0.7 to 3.0.8

[0.3.0] - 2019-05-23

Added

  • Filter to exclude files

[0.2.0] - 2019-05-22

Added

  • Display count of the number of files uploaded

Changed

  • Improved performance by fetching all MD5 hashes from S3 in single request at startup

[0.1.0] - 2019-05-13

Added

  • Initial Release
  • Synchronise files with an S3 bucket, using an MD5 hash to identify when the file has changed and needs to be uploaded