saranyaeu2987 edited a comment on issue #251:
URL:
https://github.com/apache/camel-kafka-connector/issues/251#issuecomment-643574409
@oscard
Looking for option to reduce number of files generated in s3 like grouping
all messages for a timeframe (say 5 seconds) before writing and generate
one s3 file for that timeframe.
When we add a hive schema on top of s3 location, query performance reduces
with large number is small files.
One more question, is there a way to autocreate a folder in s3 using simple
(yymmdd) given in camel.sink.url?
camel.sink.url:
aws-s3://selumalai-kafka-s3?keyName=${date:now:yyyyMMdd}/${exchangeId}
On Sat, Jun 13, 2020, 12:37 AM Andrea Cosentino <[email protected]>
wrote:
> I don't see how it would be useful. In camel, when you send a message to
> s3 through a producer it will be written as s3 object directly. In S3 there
> is no append operation. So I really don't see why changing the behavior.
> Also batch operation in what sense? Writing multiple lines on a same file?
> Or write multiple file in one shot? There is no batch support in S3 sdk v1
> as far as I know.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/camel-kafka-connector/issues/251#issuecomment-643573596>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AEXOLXRBTDT662O4VHWC5ADRWMGCFANCNFSM4NNIPOPA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
