Hi Jingqie, Why do you send all your emails twice?
On Oct 21 2016, jingjie li <[email protected]> wrote: > I'm sorry! > My expression is not correct. > This is not a problem with S3QL. > I just can't understand the flow of S3QL put big file(size > max_obj_size) to > backend. > > > Let me describe the flow again as below. > > File is uploaded from client to S3QL. > |---------------------| > | File | > |---------------------| > > S3QL separates the file to 3 blocks as below. > |---------------------| > |A |B | C |The size of the file is A + B + C. > |---------------------| A = B = max_obj_size > C = FileSize % max_obj_size > > File is transferred from client to S3QL > |---------------------------------------> > > S3QL creates object "D" which filled in zeros and have the same size with > object "C", > and PUT Object "D" to backend. > |--------------> > S3QL PUT Object "A" to backend. > |------------> > S3QL PUT Object "B" to backend. > |-----------> > S3QL PUT Object "C" to backend. > |-------> > S3QL DELETE Object "D" > |------| > > |-----------------------------------------------------------------> > |Time Most likely your application is not writing the data into the file as it grows, but it first creates a giant empty file with all zeros, and then it overwrites the zeros with the desired content. S3QL then starts uploading the zeros before your application is done overwriting them (because it has no way of knowing that this is not the actual data). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
