Sure, happy to try to help. What's happening with the hadoop filesystem is that before it writes each key it checks to see if the "parent directory" exists by checking for a key with the prefix up to the last "/", and if that key isn't found it then creates empty marker files to cause of that parent directory to exist. These existence checks are S3 HEAD requests. None of this is helpful in the case of Flink checkpointing.
And yes, while Presto avoids this unnecessary overhead, entropy injection remains an important tool for improving scalability. Where you'll run into quotas and rate limits depends, of course, on factors like the number of stateful tasks in your job(s), the parallelism, and the checkpointing interval. On Thu, May 19, 2022 at 8:04 PM Aeden Jameson <aeden.jame...@gmail.com> wrote: > Thanks for the response David. That's the conclusion I came to as > well. The Hadoop plugin behavior doesn't appear to reflect more > recent changes to S3 like strong read-after-write consistency, > > https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ > . > > Given the improved interaction of the Presto plugin with S3 I would > conclude that this increases the size of the cluster that would > require entropy injection, yes? But that it doesn't necessarily get > rid of the need because one could have a large enough cluster and say > a lifecycle policy that could still end up requiring entropy > injection. > > On Thu, May 19, 2022 at 10:24 AM David Anderson <dander...@apache.org> > wrote: > > > > Aeden, this is probably happening because you are using the Hadoop > implementation of S3. > > > > The Hadoop S3 filesystem tries to imitate a filesystem on top of S3. In > so doing it makes a lot of HEAD requests. These are expensive, and they > violate read-after-create visibility, which is what you seem to be > experiencing. By contrast, the Presto S3 implementation doesn't do the same > (harmful in this case) magic, and simply does PUT/GET operations. Because > that's all Flink needs to checkpointing, this works much better. > > > > Best, > > David > > > > On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <aeden.jame...@gmail.com> > wrote: > >> > >> We're using S3 to store checkpoints. They are taken every minute. I'm > >> seeing a large number of 404 responses from S3 being generated by the > >> job manager. The order of the entries in the debugging log would imply > >> that it's a result of a HEAD request to a key. For example all the > >> incidents look like this, > >> > >> > >> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending > >> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com > >> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers: > >> (amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream, > >> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788 > >> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8 > >> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, ) > >> > >> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received > >> error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not > >> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not > >> Found; ......) > >> > >> The key does in fact exist. How can I go about resolving this? > >> > >> -- > >> Cheers, > >> Aeden > >> > >> GitHub: https://github.com/aedenj > > > > -- > Cheers, > Aeden > > GitHub: https://github.com/aedenj > Linked In: http://www.linkedin.com/in/aedenjameson >
