54 lines
1.4 KiB
Org Mode
54 lines
1.4 KiB
Org Mode
* s3thorp
|
|
|
|
Synchronisation of files with S3 using the hash of the file contents.
|
|
|
|
Based on Alex Kudlick's JavaScript implementation [[https://github.com/akud/aws-s3-sync-by-hash][aws-s3-sync-by-hash]].
|
|
|
|
The normal ~aws s3 sync ...~ command only uses the time stamp of files
|
|
to decide what files need to be copied. This utility looks at the md5
|
|
hash of the file contents.
|
|
|
|
* How does aws-s3-sync-by-hash do it?
|
|
|
|
The following is a rough, first draft, pseudo-scala, impression of the process.
|
|
|
|
** constructor
|
|
|
|
val options = Load command line arguments and AWS security keys.
|
|
|
|
** def sync(): Promise[Upload]
|
|
|
|
val uploadPromise = createUploadPromise()
|
|
if options contains delete then createDeletePromise()
|
|
else return uploadPromise
|
|
|
|
** def createUploadPromise(): Promise[Upload]
|
|
|
|
readdir(options(root))
|
|
loadS3MetaData
|
|
filterByHash
|
|
uploadFile
|
|
callback(file => uploadedFiles += file)
|
|
|
|
** def loadS3MetaData: Stream[S3MetaData]
|
|
|
|
HEAD(bucket, key)
|
|
map (metadata => S3MetaData(localFile, bucket, key, metadata.hash, metadata.lastModified))
|
|
|
|
** def filterByHash(p: S3MetaData => Boolean): Stream[S3MetaData]
|
|
|
|
md5File(localFile)
|
|
filter(localHash => options.force || localHash != metadataHash)
|
|
|
|
** def uploadFile(upload: Upload): IO[Unit]
|
|
|
|
S3Upload(bucket, key, localFile)
|
|
|
|
** def createDeletePromise(): Promise[Delete]
|
|
|
|
S3AllKeys(bucket, key)
|
|
filter(remoteKey => localFileExists(remoteFile).negate)
|
|
|
|
** def deleteFile(delete: Delete): IO[Unit]
|
|
|
|
S3Delete(bucket, key, remoteKey)
|