Hi Do not send to @dev but to @user
Help is volunteer based so there is no guarantee for an answer. If you want guaranteed help, then you can look at commercial support. Thrre is a page on the website with companies. On Fri, Jul 3, 2020 at 11:53 AM Akhilesh Pandey <[email protected]> wrote: > > Hi, > > I am using camel's aws and kafka components. > > from("aws-s3://checkoutk1?autocloseBody=true&deleteAfterRead=false&args=....") > .idempotentConsumer(header("CamelAwsS3ETag"), > > FileIdempotentRepository.fileIdempotentRepository(new > File("target/checkoutk1.data"), 250, 512000)) > .to("kafka:test1?brokers=localhost:9092") > > I have added 1 text file in S3. It works well and adds file content to > kafka only once. > But I see these logs, seems like S3 object is constantly fetched, and only > filtered before adding to kafka. > > Am i correct in understanding this behaviour ? > > If yes, Is there a work around so the file is not fetched. > > Logs > > If you check text file CamelAwsS3ContentLength, its always > CamelAwsS3ContentLength=88. > > 020-07-03 14:41:13,793 [s3.Basic.main()] INFO AppInfoParser > - Kafka version: 2.5.0 > 2020-07-03 14:41:13,793 [s3.Basic.main()] INFO AppInfoParser > - Kafka commitId: 66563e712b0b9f84 > 2020-07-03 14:41:13,793 [s3.Basic.main()] INFO AppInfoParser > - Kafka startTimeMs: 1593767473790 > 2020-07-03 14:41:13,806 [s3.Basic.main()] INFO > InternalRouteStartupManager - Route: route1 started and consuming > from: aws-s3://checkoutk1 > 2020-07-03 14:41:13,806 [s3.Basic.main()] INFO AbstractCamelContext > - Total 1 routes, of which 1 are started > 2020-07-03 14:41:13,807 [s3.Basic.main()] INFO AbstractCamelContext > - Apache Camel 3.5.0-SNAPSHOT (camel-1) started in 1.174 > seconds > 2020-07-03 14:41:14,037 [ad | producer-1] INFO Metadata > - [Producer clientId=producer-1] Cluster ID: > 2NaFZgsTQg-4L3vQzwmlkQ > 2020-07-03 14:41:15,004 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:15,005 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/ > 2020-07-03 14:41:15,012 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=0, > CamelAwsS3ContentMD5=null, > CamelAwsS3ContentType=application/x-directory, > CamelAwsS3ETag=d41d8cd98f00b204e9800998ecf8427e, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=0, Content-Type=application/x-directory, > ETag=d41d8cd98f00b204e9800998ecf8427e, Last-Modified=Thu Jul 02 > 16:30:15 IST 2020}, CamelAwsS3Key=data/, CamelAwsS3LastModified=Thu > Jul 02 16:30:15 IST 2020, CamelAwsS3ReplicationStatus=null, > CamelAwsS3ServerSideEncryption=null, CamelAwsS3StorageClass=null, > CamelAwsS3UserMetadata={}, CamelAwsS3VersionId=null} > 2020-07-03 14:41:15,015 [s3://checkoutk1] INFO route1 > - This is new message > 2020-07-03 14:41:15,050 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:15,050 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/text2.txt > 2020-07-03 14:41:15,051 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=88, > CamelAwsS3ContentMD5=null, CamelAwsS3ContentType=text/plain, > CamelAwsS3ETag=f638e3b03d75d5d1d11984585c23bb8d, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=88, Content-Type=text/plain, > ETag=f638e3b03d75d5d1d11984585c23bb8d, Last-Modified=Fri Jul 03 > 14:28:25 IST 2020}, CamelAwsS3Key=data/text2.txt, > CamelAwsS3LastModified=Fri Jul 03 14:28:25 IST 2020, > CamelAwsS3ReplicationStatus=null, CamelAwsS3ServerSideEncryption=null, > CamelAwsS3StorageClass=null, CamelAwsS3UserMetadata={}, > CamelAwsS3VersionId=null} > 2020-07-03 14:41:15,052 [s3://checkoutk1] INFO route1 > - This is new message > 2020-07-03 14:41:15,068 [Producer[test1]] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=0, > CamelAwsS3ContentMD5=null, > CamelAwsS3ContentType=application/x-directory, > CamelAwsS3ETag=d41d8cd98f00b204e9800998ecf8427e, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=0, Content-Type=application/x-directory, > ETag=d41d8cd98f00b204e9800998ecf8427e, Last-Modified=Thu Jul 02 > 16:30:15 IST 2020}, CamelAwsS3Key=data/, CamelAwsS3LastModified=Thu > Jul 02 16:30:15 IST 2020, CamelAwsS3ReplicationStatus=null, > CamelAwsS3ServerSideEncryption=null, CamelAwsS3StorageClass=null, > CamelAwsS3UserMetadata={}, CamelAwsS3VersionId=null, > org.apache.kafka.clients.producer.RecordMetadata=[test1-0@93]} > 2020-07-03 14:41:15,069 [Producer[test1]] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=88, > CamelAwsS3ContentMD5=null, CamelAwsS3ContentType=text/plain, > CamelAwsS3ETag=f638e3b03d75d5d1d11984585c23bb8d, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=88, Content-Type=text/plain, > ETag=f638e3b03d75d5d1d11984585c23bb8d, Last-Modified=Fri Jul 03 > 14:28:25 IST 2020}, CamelAwsS3Key=data/text2.txt, > CamelAwsS3LastModified=Fri Jul 03 14:28:25 IST 2020, > CamelAwsS3ReplicationStatus=null, CamelAwsS3ServerSideEncryption=null, > CamelAwsS3StorageClass=null, CamelAwsS3UserMetadata={}, > CamelAwsS3VersionId=null, > org.apache.kafka.clients.producer.RecordMetadata=[test1-0@94]} > 2020-07-03 14:41:20,208 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:20,209 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/ > 2020-07-03 14:41:20,209 [s3://checkoutk1] INFO route1 - > {CamelAwsS3BucketName=checkoutk1, CamelAwsS3ContentControl=null, > CamelAwsS3ContentDisposition=null, CamelAwsS3ContentEncoding=null, > CamelAwsS3ContentLength=0, CamelAwsS3ContentMD5=null, > CamelAwsS3ContentType=application/x-directory, > CamelAwsS3ETag=d41d8cd98f00b204e9800998ecf8427e, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=0, Content-Type=application/x-directory, > ETag=d41d8cd98f00b204e9800998ecf8427e, Last-Modified=Thu Jul 02 > 16:30:15 IST 2020}, CamelAwsS3Key=data/, CamelAwsS3LastModified=Thu > Jul 02 16:30:15 IST 2020, CamelAwsS3ReplicationStatus=null, > CamelAwsS3ServerSideEncryption=null, CamelAwsS3StorageClass=null, > CamelAwsS3UserMetadata={}, CamelAwsS3VersionId=null} > 2020-07-03 14:41:20,210 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:20,210 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/text2.txt > 2020-07-03 14:41:20,210 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=88, > CamelAwsS3ContentMD5=null, CamelAwsS3ContentType=text/plain, > CamelAwsS3ETag=f638e3b03d75d5d1d11984585c23bb8d, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=88, Content-Type=text/plain, > ETag=f638e3b03d75d5d1d11984585c23bb8d, Last-Modified=Fri Jul 03 > 14:28:25 IST 2020}, CamelAwsS3Key=data/text2.txt, > CamelAwsS3LastModified=Fri Jul 03 14:28:25 IST 2020, > CamelAwsS3ReplicationStatus=null, CamelAwsS3ServerSideEncryption=null, > CamelAwsS3StorageClass=null, CamelAwsS3UserMetadata={}, > CamelAwsS3VersionId=null} > Jul 03, 2020 2:41:20 PM > com.amazonaws.services.s3.internal.S3AbortableInputStream close > WARNING: Not all bytes were read from the S3ObjectInputStream, > aborting HTTP connection. This is likely an error and may result in > sub-optimal behavior. Request only the bytes you need via a ranged GET > or drain the input stream after use. > 2020-07-03 14:41:25,519 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:25,519 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/ > 2020-07-03 14:41:25,520 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=0, > CamelAwsS3ContentMD5=null, > CamelAwsS3ContentType=application/x-directory, > CamelAwsS3ETag=d41d8cd98f00b204e9800998ecf8427e, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=0, Content-Type=application/x-directory, > ETag=d41d8cd98f00b204e9800998ecf8427e, Last-Modified=Thu Jul 02 > 16:30:15 IST 2020}, CamelAwsS3Key=data/, CamelAwsS3LastModified=Thu > Jul 02 16:30:15 IST 2020, CamelAwsS3ReplicationStatus=null, > CamelAwsS3ServerSideEncryption=null, CamelAwsS3StorageClass=null, > CamelAwsS3UserMetadata={}, CamelAwsS3VersionId=null} > 2020-07-03 14:41:25,520 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:25,520 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/text2.txt > 2020-07-03 14:41:25,521 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=88, > CamelAwsS3ContentMD5=null, CamelAwsS3ContentType=text/plain, > CamelAwsS3ETag=f638e3b03d75d5d1d11984585c23bb8d, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=88, Content-Type=text/plain, > ETag=f638e3b03d75d5d1d11984585c23bb8d, Last-Modified=Fri Jul 03 > 14:28:25 IST 2020}, CamelAwsS3Key=data/text2.txt, > CamelAwsS3LastModified=Fri Jul 03 14:28:25 IST 2020, > CamelAwsS3ReplicationStatus=null, CamelAwsS3ServerSideEncryption=null, > CamelAwsS3StorageClass=null, CamelAwsS3UserMetadata={}, > CamelAwsS3VersionId=null} > Jul 03, 2020 2:41:25 PM > com.amazonaws.services.s3.internal.S3AbortableInputStream close > WARNING: Not all bytes were read from the S3ObjectInputStream, > aborting HTTP connection. This is likely an error and may result in > sub-optimal behavior. Request only the bytes you need via a ranged GET > or drain the input stream after use. > 2020-07-03 14:41:30,770 [s3://checkoutk1] INFO consuming > - Consumer Fired! > 2020-07-03 14:41:30,771 [s3://checkoutk1] INFO route1 > - Replay Message Sent to file:s3out data/ > 2020-07-03 14:41:30,771 [s3://checkoutk1] INFO route1 > - {CamelAwsS3BucketName=checkoutk1, > CamelAwsS3ContentControl=null, CamelAwsS3ContentDisposition=null, > CamelAwsS3ContentEncoding=null, CamelAwsS3ContentLength=0, > CamelAwsS3ContentMD5=null, > CamelAwsS3ContentType=application/x-directory, > CamelAwsS3ETag=d41d8cd98f00b204e9800998ecf8427e, > CamelAwsS3ExpirationTime=null, CamelAwsS3Headers={Accept-Ranges=bytes, > Content-Length=0, Content-Type=application/x-directory, > ETag=d41d8cd98f00b204e9800998ecf8427e, Last-Modified=Thu Jul 02 > 16:30:15 IST 2020}, CamelAwsS3Key=data/, CamelAwsS3LastModified=Thu > Jul 02 16:30:15 IST 2020, CamelAwsS3ReplicationStatus=null, > CamelAwsS3ServerSideEncryption=null, CamelAwsS3StorageClass=null, > CamelAwsS3UserMetadata={}, CamelAwsS3VersionId=null} > > ... > ... > It keeps on going like above > > I didnt sent it to mailing list, since i got no reply last time. -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
