Re: Camel S3 Source Connector - value.converter
Yes
Il sab 26 dic 2020, 16:25 Arundhati Bhende
ha scritto:
> Thanks again. Will test with these combinations of different value
> converters after I am able to get the avro format files.
>
> In the previous response, you had mentioned it is better to use aws2-s3
> version of connector - So, in order to do so, to use the camel-aws2-s3
> connector, our aws set-up must be at version 2, right? Because, currently
> it is at v1 and do not know, if we have immediate option of moving to v2
>
> -Original Message-
> From: Andrea Cosentino
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false. In that case,
> > will I be able to use any valid converter ? What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter, but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> > must be at version 2, right? Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino" wrote:
> >
> > Yes, the approach without url is correct
> > Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhe...@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used. I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "1",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > #"value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=1
> > >
> > > topics=mytopic
> > >
> > > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=
> > > camel.component.aws-s3.secret-key=
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1. How the AWS credentials are set-up
> > > 2. In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work? What impact does it have when
> > > setting it to true vs. false?
> > >
> > >
> > > Thank you
> > >
> > >
> > > -Origina
RE: Camel S3 Source Connector - value.converter
Thanks again. Will test with these combinations of different value converters
after I am able to get the avro format files.
In the previous response, you had mentioned it is better to use aws2-s3 version
of connector - So, in order to do so, to use the camel-aws2-s3 connector,
our aws set-up must be at version 2, right? Because, currently it is at v1
and do not know, if we have immediate option of moving to v2
-Original Message-
From: Andrea Cosentino
Sent: Saturday, December 26, 2020 10:19 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
You can use any of the dataformat provided by camel or converters from kafka
connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende ha
scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case,
> will I be able to use any valid converter ? What I mean is currently
> I am testing with a simple text file and hence was trying with
> StringConverter, but in reality, those maybe JSON on Avro formatted
> messages in the files, so will I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I
> need to set-up any other property to convert each line to a separate
> message to the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so
> you want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector,
> based on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhe...@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "1",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > #"value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=1
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=
> > camel.component.aws-s3.secret-key=
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -Original Message-
> > From: Andrea Cosentino
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhe...@pruden
Re: Camel S3 Source Connector - value.converter
You can use any of the dataformat provided by camel or converters from
kafka connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende
ha scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case, will I
> be able to use any valid converter ? What I mean is currently I am testing
> with a simple text file and hence was trying with StringConverter, but in
> reality, those maybe JSON on Avro formatted messages in the files, so will
> I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I need
> to set-up any other property to convert each line to a separate message to
> the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so you
> want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector, based
> on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhe...@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "1",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > #"value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=1
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=
> > camel.component.aws-s3.secret-key=
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -Original Message-
> > From: Andrea Cosentino
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > I am trying to test The Camel S3 Source connector using basic
> example
> > > from this link
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7Cf7b74faf6add4a4f39f808d8a9aeed82%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445916065455084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2ISA5ZwtgVUwF%2BvydKewwZbMB5YVr9HNem1f6r4lDB8%3D&reserved=0
> > .
> >
Re: Camel S3 Source Connector - value.converter
Thank you.
I will test with setting the autocloseBody to false. In that case, will I be
able to use any valid converter ? What I mean is currently I am testing with a
simple text file and hence was trying with StringConverter, but in reality,
those maybe JSON on Avro formatted messages in the files, so will I be able to
use those formats as converters?
Other part of the question - after I get a simple example working - a single
file will contain multiple records, one record per line - do I need to set-up
any other property to convert each line to a separate message to the topic?
One doubt / question - to use the camel-aws2-s3 connector, our aws set-up must
be at version 2, right? Because, currently it is at v1 and do not know, if we
have immediate option of moving to v2
On 12/26/20, 10:00 AM, "Andrea Cosentino" wrote:
Yes, the approach without url is correct
Autoclosebody will close s3object after consuming the payload, so you want
to get something it must be equal to false, while using the converter.
My suggestion by the way is using the camel-aws2-s3 connector, based on ask
v2
Il sab 26 dic 2020, 15:53 Arundhati Bhende
ha scritto:
> This is the full configuration that I have used. I am been experimenting
> with changing the value.converter between StringConverter and
> S3ObjectConverter - otherwise everything is same.
>
>
>
> DATA=$( cat << EOF
> {
> "connector.class":
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> "key.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "value.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "camel.source.maxPollDuration": "1",
> "topics": "TEST-S3-SOURCE-DZONE-POC",
> "camel.source.path.bucketNameOrArn": " push-json-poc",
> "camel.component.aws-s3.region": "US_EAST_1",
> "tasks.max": "1",
> "camel.source.endpoint.useIAMCredentials": "true",
> "camel.source.endpoint.autocloseBody": "true"
> }
> EOF
> )
> #"value.converter":
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
>
>
> This is official one from the link in the other email.
> name=CamelAWSS3SourceConnector
>
>
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> key.converter=org.apache.kafka.connect.storage.StringConverter
>
>
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> camel.source.maxPollDuration=1
>
> topics=mytopic
>
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
>
> camel.component.aws-s3.access-key=
> camel.component.aws-s3.secret-key=
> camel.component.aws-s3.region=EU_WEST_1
>
>
>
>
> I can see two differences -
> 1. How the AWS credentials are set-up
> 2. In my configuration
> I am using camel.source.path.bucketNameOrArn instead of
> camel.source.url
> Because I was getting error that bucketNameOrArn must be defined
>
> In my configuration - autocloseBody is set true to whereas in the
> default configuration it is set to false
>
> How does autocloseBody work? What impact does it have when
> setting it to true vs. false?
>
>
> Thank you
>
>
> -Original Message-
> From: Andrea Cosentino
> Sent: Saturday, December 26, 2020 3:59 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > I am trying to test The Camel S3 Source connector using basic example
> > from this link
> >
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7Cf7b74faf6add4a4f39f808d8a9aeed82%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445916065455084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2ISA5ZwtgVUwF%2BvydKewwZbMB5YVr9HNem1f6r4lDB8%3D&reserved=0
> .
> > Below is the file I am uploading to S3.
> >
> > testfile.txt
> > add one line
> > add another line
> > test the connector
> >
> >
> > After the connector picks up and processes the file, If I use this for
> > value converter
> >
> >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > ectConverter
> >
> > When I run the consumer, it shows the message processed but the output
> > shows blank.
> >
> > If I use
Re: Camel S3 Source Connector - value.converter
Yes, the approach without url is correct
Autoclosebody will close s3object after consuming the payload, so you want
to get something it must be equal to false, while using the converter.
My suggestion by the way is using the camel-aws2-s3 connector, based on ask
v2
Il sab 26 dic 2020, 15:53 Arundhati Bhende
ha scritto:
> This is the full configuration that I have used. I am been experimenting
> with changing the value.converter between StringConverter and
> S3ObjectConverter - otherwise everything is same.
>
>
>
> DATA=$( cat << EOF
> {
> "connector.class":
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> "key.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "value.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "camel.source.maxPollDuration": "1",
> "topics": "TEST-S3-SOURCE-DZONE-POC",
> "camel.source.path.bucketNameOrArn": " push-json-poc",
> "camel.component.aws-s3.region": "US_EAST_1",
> "tasks.max": "1",
> "camel.source.endpoint.useIAMCredentials": "true",
> "camel.source.endpoint.autocloseBody": "true"
> }
> EOF
> )
> #"value.converter":
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
>
>
> This is official one from the link in the other email.
> name=CamelAWSS3SourceConnector
>
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> key.converter=org.apache.kafka.connect.storage.StringConverter
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> camel.source.maxPollDuration=1
>
> topics=mytopic
>
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
>
> camel.component.aws-s3.access-key=
> camel.component.aws-s3.secret-key=
> camel.component.aws-s3.region=EU_WEST_1
>
>
>
>
> I can see two differences -
> 1. How the AWS credentials are set-up
> 2. In my configuration
> I am using camel.source.path.bucketNameOrArn instead of
> camel.source.url
> Because I was getting error that bucketNameOrArn must be defined
>
> In my configuration - autocloseBody is set true to whereas in the
> default configuration it is set to false
>
> How does autocloseBody work? What impact does it have when
> setting it to true vs. false?
>
>
> Thank you
>
>
> -Original Message-
> From: Andrea Cosentino
> Sent: Saturday, December 26, 2020 3:59 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > I am trying to test The Camel S3 Source connector using basic example
> > from this link
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0
> .
> > Below is the file I am uploading to S3.
> >
> > testfile.txt
> > add one line
> > add another line
> > test the connector
> >
> >
> > After the connector picks up and processes the file, If I use this for
> > value converter
> >
> >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > ectConverter
> >
> > When I run the consumer, it shows the message processed but the output
> > shows blank.
> >
> > If I use the String converter as value converter
> >
> > org.apache.kafka.connect.storage.StringConverter
> >
> > I get the output in the form of
> >
> > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> >
> > From the example in the above link, the author doesn't seem to have to
> > do any actions to see the string.
> >
> > So, why would the S3ObjectConverter show a blank output for me? I can
> > understand the String representation for the object really shows the
> > memory address.
> >
> > What do I need to look at to fix the output.
> >
> > Also, is it possible to change the converter to JSON or Avro converter
> > in the configuration file or I need to add either different properties
> > to the configuration or do some processing afterwards?
> >
> > Thank You
> >
> >
>
[ANNOUNCEMENT] Apache Camel-kafka-connector 0.7.0
The Camel community announces the immediate availability of Apache Camel-kafka-connector 0.7.0 The artifacts are published and ready for you to download either from the Apache mirrors or from maven central. For more info please visit the project GitHub repository[1]. Many thanks to all who made this release possible. Regards, Andrea. [1] https://github.com/apache/camel-kafka-connector
RE: Camel S3 Source Connector - value.converter
This is the full configuration that I have used. I am been experimenting with
changing the value.converter between StringConverter and S3ObjectConverter -
otherwise everything is same.
DATA=$( cat << EOF
{
"connector.class":
"org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"camel.source.maxPollDuration": "1",
"topics": "TEST-S3-SOURCE-DZONE-POC",
"camel.source.path.bucketNameOrArn": " push-json-poc",
"camel.component.aws-s3.region": "US_EAST_1",
"tasks.max": "1",
"camel.source.endpoint.useIAMCredentials": "true",
"camel.source.endpoint.autocloseBody": "true"
}
EOF
)
#"value.converter":
"org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
This is official one from the link in the other email.
name=CamelAWSS3SourceConnector
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
camel.source.maxPollDuration=1
topics=mytopic
camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
camel.component.aws-s3.access-key=
camel.component.aws-s3.secret-key=
camel.component.aws-s3.region=EU_WEST_1
I can see two differences -
1. How the AWS credentials are set-up
2. In my configuration
I am using camel.source.path.bucketNameOrArn instead of
camel.source.url
Because I was getting error that bucketNameOrArn must be defined
In my configuration - autocloseBody is set true to whereas in the
default configuration it is set to false
How does autocloseBody work? What impact does it have when setting it
to true vs. false?
Thank you
-Original Message-
From: Andrea Cosentino
Sent: Saturday, December 26, 2020 3:59 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
Can you show your full configuration?
Il sab 26 dic 2020, 07:40 Arundhati Bhende ha
scritto:
> I am trying to test The Camel S3 Source connector using basic example
> from this link
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0.
> Below is the file I am uploading to S3.
>
> testfile.txt
> add one line
> add another line
> test the connector
>
>
> After the connector picks up and processes the file, If I use this for
> value converter
>
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> ectConverter
>
> When I run the consumer, it shows the message processed but the output
> shows blank.
>
> If I use the String converter as value converter
>
> org.apache.kafka.connect.storage.StringConverter
>
> I get the output in the form of
>
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
>
> From the example in the above link, the author doesn't seem to have to
> do any actions to see the string.
>
> So, why would the S3ObjectConverter show a blank output for me? I can
> understand the String representation for the object really shows the
> memory address.
>
> What do I need to look at to fix the output.
>
> Also, is it possible to change the converter to JSON or Avro converter
> in the configuration file or I need to add either different properties
> to the configuration or do some processing afterwards?
>
> Thank You
>
>
Re: Help - User mailing list
Here you can find the info you’re looking for: https://camel.apache.org/manual/latest/mailing-lists.html 2020年12月26日(土) 15:39 Arundhati Bhende : > Hi, > > A couple of questions about the userlist. > > Is it possible to access / post to the list from the web browser? > > How do I search the userlist before asking a question? > > Thank you > > > -- Tadayoshi Sato
Re: Camel S3 Source Connector - value.converter
As example use the official one https://github.com/apache/camel-kafka-connector-examples/blob/master/aws-s3-to-jms/config/CamelAWSS3SourceConnector.properties Il sab 26 dic 2020, 09:59 Andrea Cosentino ha scritto: > Can you show your full configuration? > > Il sab 26 dic 2020, 07:40 Arundhati Bhende < > arundhati.bhe...@prudential.com> ha scritto: > >> I am trying to test The Camel S3 Source connector using basic example >> from this link >> https://dzone.com/articles/reading-aws-s3-file-content-to-kafka-topic. >> Below is the file I am uploading to S3. >> >> testfile.txt >> add one line >> add another line >> test the connector >> >> >> After the connector picks up and processes the file, If I use this for >> value converter >> >> >> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter >> >> When I run the consumer, it shows the message processed but the output >> shows blank. >> >> If I use the String converter as value converter >> >> org.apache.kafka.connect.storage.StringConverter >> >> I get the output in the form of >> >> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a >> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103 >> >> From the example in the above link, the author doesn't seem to have to do >> any actions to see the string. >> >> So, why would the S3ObjectConverter show a blank output for me? I can >> understand the String representation for the object really shows the memory >> address. >> >> What do I need to look at to fix the output. >> >> Also, is it possible to change the converter to JSON or Avro converter in >> the configuration file or I need to add either different properties to the >> configuration or do some processing afterwards? >> >> Thank You >> >>
Re: Camel S3 Source Connector - value.converter
Can you show your full configuration? Il sab 26 dic 2020, 07:40 Arundhati Bhende ha scritto: > I am trying to test The Camel S3 Source connector using basic example from > this link > https://dzone.com/articles/reading-aws-s3-file-content-to-kafka-topic. > Below is the file I am uploading to S3. > > testfile.txt > add one line > add another line > test the connector > > > After the connector picks up and processes the file, If I use this for > value converter > > > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter > > When I run the consumer, it shows the message processed but the output > shows blank. > > If I use the String converter as value converter > > org.apache.kafka.connect.storage.StringConverter > > I get the output in the form of > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103 > > From the example in the above link, the author doesn't seem to have to do > any actions to see the string. > > So, why would the S3ObjectConverter show a blank output for me? I can > understand the String representation for the object really shows the memory > address. > > What do I need to look at to fix the output. > > Also, is it possible to change the converter to JSON or Avro converter in > the configuration file or I need to add either different properties to the > configuration or do some processing afterwards? > > Thank You > >
