Paul Campbell
7af4004c75
If the remote file is missing then return None. S3MetaDataEnricher.enrichWithS3MetaData now returns an IO[Either[File, S3MetaData]]. If objectHead returns None, the this returns the file, otherwise, the Some[Hash, LastModified] from objectHead is used to create the S3MetaData as before. |
||
|---|---|---|
| src | ||
| .gitignore | ||
| build.sbt | ||
| README.org | ||
- s3thorp
- How does aws-s3-sync-by-hash do it?
- constructor
- def sync(): Promise[Upload]
- def createUploadPromise(): Promise[Upload]
- def loadS3MetaData: Stream[S3MetaData]
- def filterByHash(p: S3MetaData => Boolean): Stream[S3MetaData]
- def uploadFile(upload: Upload): IO[Unit]
- def createDeletePromise(): Promise[Delete]
- def deleteFile(delete: Delete): IO[Unit]
s3thorp
Synchronisation of files with S3 using the hash of the file contents.
Based on Alex Kudlick's JavaScript implementation aws-s3-sync-by-hash.
The normal aws s3 sync ... command only uses the time stamp of files
to decide what files need to be copied. This utility looks at the md5
hash of the file contents.
How does aws-s3-sync-by-hash do it?
The following is a rough, first draft, pseudo-scala, impression of the process.
constructor
val options = Load command line arguments and AWS security keys.
def sync(): Promise[Upload]
val uploadPromise = createUploadPromise() if options contains delete then createDeletePromise() else return uploadPromise
def createUploadPromise(): Promise[Upload]
readdir(options(root))
loadS3MetaData
filterByHash
uploadFile
callback(file > uploadedFiles + file)
def loadS3MetaData: Stream[S3MetaData]
HEAD(bucket, key) map (metadata => S3MetaData(localFile, bucket, key, metadata.hash, metadata.lastModified))
def filterByHash(p: S3MetaData => Boolean): Stream[S3MetaData]
md5File(localFile)
filter(localHash > options.force || localHash ! metadataHash)
def uploadFile(upload: Upload): IO[Unit]
S3Upload(bucket, key, localFile)
def createDeletePromise(): Promise[Delete]
S3AllKeys(bucket, key) filter(remoteKey => localFileExists(remoteFile).negate)
def deleteFile(delete: Delete): IO[Unit]
S3Delete(bucket, key, remoteKey)