question

yadayada avatar image
yadayada asked

Lost metadata update with concurrent overwrite

Let's say I 1) create an empty file foo.bar.tmp, 2) start to overwrite foo.bar.tmp's content (in a separate thread) 3) rename "foo.bar.tmp" to "foo.bar" 4) wait till the overwrite is finished Judging by the returned node JSON, the rename operation was successful. When the overwrite operation finishes, its returned JSON will show the original file name, foo.bar.tmp. But that's not what I would expect, since an overwrite should only concern the node content, not its metadata.
amazon drive
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jamie Grossman avatar image
Jamie Grossman answered
Hey there, Thanks for posting. I'll pass this information on to the team to see whats going on and will get back to you as soon as I have more information on the matter. Regards, Jamie
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jamie Grossman avatar image
Jamie Grossman answered
Hi there, Thanks for your patience. Is this critical for an application you are developing? Any potential use-cases you have in mind would be a great example to show the need for this type of implementation. Regardless, it would also be very helpful if you could provide some more information including Times, node_ids, request_ids, timing used to repro, etc. Feel free to contact the team here: clouddrive-api-support@amazon.com Regards, Jamie
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

yadayada avatar image
yadayada answered
Thanks for your reply. I would not say this is critical, exactly, but it is confusing. Multi-threading was not expected when designing the service, I guess. Users of acd_cli (including me) may encounter this problem when using the Amazon Cloud Drive as a virtual hard disk. See https://github.com/yadayada/acd_cli/issues/116#issuecomment-134297103 and below. I think it is easily reproducible with any kind of testing environment. To reproduce with acd_cli, one needs a large enough file commands, e.g. dd if=/dev/urandom bs=1M count=500m of=rand500m.tmp Then, create an empty file in the cloud drive echo | acdcli stream rand500m.tmp Initiate the overwrite acdcli ul -of rand500.tmp / While uploading, rename the empty file acdcli rn /rand500m.tmp foobar The rename will be successful and the empty "foobar" file will show up in the web interface. But when the upload is finished, the file name will revert to "rand500m.tmp". The use case is using rsync or an ordinary cp to upload files to the cloud drive via a FUSE mount. On the file system level, it is customary to create an empty file, write into it and rename it. The renaming operation may be initiated while the last portion of the file is still in buffer and about to being written.
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.