Hi, Christian Müller:
Thanks you. I know how to split csv now.
First, please forgive my long e-mail , but this question is confusing me
for a long time. I think I should express it clearly at this chance.
I still feel confused about whether we should do "batch" operation in the
route or do "batch" operation in adaptor. (camel use the words "component or
endpoint" instead of adaptor)
Do "batch" operations in route will be more flexible. But do "batch"
operations in adaptors will achieve more performance: reduce memory usage ,
and increasing the processing speed;
The "csv" is not a good example, maybe we could using "SQL"
component to illustrate my confusing question:
First let's assume the inbound and outbound "SQL" component support
"batch" operations.
For inbound adpator, "batch" means: when "SQL" inbound endpoint
select billions of records, it doesn't output all the records at once, but
it will output batch of records, each time only a batch of records is
output, and it will finish output until all record are outputted ;The record
number of a batch can be limited by "batch size". For inbound adaptor's
"batch operation", it looks like "split" operation is hidden in the inbound
endpoint.
Because the records never output at once, so the memory usage will
be reduced;
For outbound adaptor, "batch" means: it can receive batch of
records, and insert a batch of records into database in a single insert
operation. This is to say: it can construct a long SQL to insert multiple
records into database, so the processing speed will be very fast than insert
record one by one;
Actually, "do batch in adaptor " is not as flexible as "do batching
in route". But I haven't think of a good way which can achieve both flexible
, and performance also.
How does you understand this question?
Does anyone have a good idea for this question?
Thanks a lot
-----orginal-----
Sender: Christian Müller [mailto:christian.muel...@gmail.com]
Date: 2010/12/15 5:26
Receiver: users@camel.apache.org
Subject: Re: Splitting big XML files using xpath() and streaming()
Only for clarification, the streaming mode doesn't work with xpath but works
very well with the Scanner, which is used if you split like this:
split(body(String.class).tokenize("\n")).
If you have to split CSV files, you will probably use the Scanner.
You should also have a look at the camel smooks integration, provided by
Smooks out of the box...
Cheers,
Christian
