Hi Sergio, Can you tell a bit more about your use case?
1) How big are the files? Is there a specific threshold when doesn't it work anymore? 2) What kind of error do you get and where do you get it? (On Camel side or Kafka side, is it memory related)? 3) What kind of files are you using (csv, xml, json)? 4) Did you try to use the split pattern ( https://camel.apache.org/components/3.14.x/eips/split-eip.html)? More in general: I know often you have no influence in the files that are at the source, but working message-based (instead of file-based like SFTP) and with smaller messages (so the load is spread evenly over time) and real-time/event-based (send change when it occurs) mostly is the best approach. If not, you mostly end up with such issues when scaling. Kind regards, Raymond On Thu, Feb 3, 2022 at 11:19 AM Sergio Meana <sme...@darksecurities.com> wrote: > Hello > > We are trying to read files using the following connector > > https://github.com/apache/camel-kafka-connector-examples/tree/main/sftp/sftp-source > The connector send the whole file in one record to kafka which it is > failing with big files > > It is possible to override the apply method to convert the whole record > into small batches? > > public R apply(R r) { > > We unsuccessfully tried a couple of thing but not luck > > Thanks in advance > > [Emails signature size.png] > Sergio Meana > Chief Quant Developer| Data Scientist > DARK SECURITIES LTD > Email smeana[@](mailto:mario.lebr...@bcssmz.org)darksecurities.com > Tel +44 7732581034 > > This email and any files transmitted with it are confidential and may be > privileged or otherwise protected from disclosure, and are intended solely > for the use of the individual or entity to whom they are addressed. If you > received this email in error please notify the sender immediately and > permanently delete all copies of this email and its attachments from your > system. Any unauthorised use, copying, distribution, or disclosure of this > email and its attachments is prohibited; to the fullest extent permitted by > law Dark Securities LTD does not accept any liability or responsibility > with respect to any action taken by you in reliance on the contents of this > email and its attachments.
