This is the full configuration that I have used. I am been experimenting with
changing the value.converter between StringConverter and S3ObjectConverter -
otherwise everything is same.
DATA=$( cat << EOF
{
"connector.class":
"org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"camel.source.maxPollDuration": "10000",
"topics": "TEST-S3-SOURCE-DZONE-POC",
"camel.source.path.bucketNameOrArn": " push-json-poc",
"camel.component.aws-s3.region": "US_EAST_1",
"tasks.max": "1",
"camel.source.endpoint.useIAMCredentials": "true",
"camel.source.endpoint.autocloseBody": "true"
}
EOF
)
# "value.converter":
"org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
This is official one from the link in the other email.
name=CamelAWSS3SourceConnector
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
camel.source.maxPollDuration=10000
topics=mytopic
camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
camel.component.aws-s3.access-key=xxxx
camel.component.aws-s3.secret-key=yyyy
camel.component.aws-s3.region=EU_WEST_1
I can see two differences -
1. How the AWS credentials are set-up
2. In my configuration
I am using camel.source.path.bucketNameOrArn instead of
camel.source.url
Because I was getting error that bucketNameOrArn must be defined
In my configuration - autocloseBody is set true to whereas in the
default configuration it is set to false
How does autocloseBody work? What impact does it have when setting it
to true vs. false?
Thank you
-----Original Message-----
From: Andrea Cosentino <[email protected]>
Sent: Saturday, December 26, 2020 3:59 AM
To: [email protected]
Subject: Re: Camel S3 Source Connector - value.converter
Can you show your full configuration?
Il sab 26 dic 2020, 07:40 Arundhati Bhende <[email protected]> ha
scritto:
> I am trying to test The Camel S3 Source connector using basic example
> from this link
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0.
> Below is the file I am uploading to S3.
>
> testfile.txt
> add one line
> add another line
> test the connector
>
>
> After the connector picks up and processes the file, If I use this for
> value converter
>
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> ectConverter
>
> When I run the consumer, it shows the message processed but the output
> shows blank.
>
> If I use the String converter as value converter
>
> org.apache.kafka.connect.storage.StringConverter
>
> I get the output in the form of
>
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
>
> From the example in the above link, the author doesn't seem to have to
> do any actions to see the string.
>
> So, why would the S3ObjectConverter show a blank output for me? I can
> understand the String representation for the object really shows the
> memory address.
>
> What do I need to look at to fix the output.
>
> Also, is it possible to change the converter to JSON or Avro converter
> in the configuration file or I need to add either different properties
> to the configuration or do some processing afterwards?
>
> Thank You
>
>
