Thanks. I'll take a look at the custom BeanioIterator implementation. On
the current project it's a nice to have but the next one will likely
require. We read in 10k to 50k records which we can handle in memory. In
the next project that will be more like 500k records (each mutli-line) and
at that point chunking makes a lot more sense. I'd like to chunk it now if
it is easy. I'm wondering if I could read it in with chunks of lines and
then put it through the unmarshaling. I'll give it a test but it sounds
like I'll probably need the iterator.
<route id="batch.request.route">
<from uri="{{batch.request.endpoint}}" />
<split streaming="true">
<tokenize token="\n" group="1000" />
<unmarshal ref="beanioRecordsIn" />
<recipientList>
<simple>seda:${body.class.simpleName}</simple>
</recipientList>
</split>
</route>
On Tue, Apr 5, 2016 at 12:53 AM, mailingl...@j-b-s.de <mailingl...@j-b-s.de>
wrote:
> Hi Brad!
>
> To read and split a file you need a combination of streaming unmarshalling
> and splitter components. Is tmxPaymentTechIn something you wrote?
>
> You can try to read the file by camel as xml, splitting groups by xpath
> and unmarshall just the group from xml to a bean model...
>
> Unfortunately I do not use BeanIO so I do not know how the unmarshalled
> instance model looks like. Is there a List<Group> involved? If yes, it's to
> late for splitting the unmarshalled beans anyway.
>
> You can write your own processor which makes use of template-producer to
> create individual messages per group while you read the file on your own.
>
> Because you need additional header fields set I would try to write a camel
> processor to create the messages.
>
> Jens
>
> Von meinem iPhone gesendet
>
> > Am 05.04.2016 um 00:54 schrieb Brad Johnson <
> brad.john...@mediadriver.com>:
> >
> > I have a beanio mapping that's rather standard and I read it in but it
> > appears that it reads the entire file into memory instead of streaming
> it.
> > Is there a way to change that behavior so that I can take items as they
> are
> > read and put them on a queue?
> >
> > <from uri="file:/dir/inbound" />
> > <unmarshal ref="tmxPaymentTechIn" />
> > <log message="Process and convert records." />
> > <to uri="seda:processRecords" />
> >
> > The file is an ole timey batch record with headers and mutli-line, fixed
> > width fields per record. What I'd like to see is each element header,
> > transaction, etc. read from the file and then put on the queue. What I
> get
> > is an array of the entire file which I can, of course, split but that
> will
> > get memory intensive.
> >
> > BeanIO works very well for this as there are variable row records
> (Address
> > for example) and it snaps that stuff up well. But I really want records
> or
> > groups to come through individually.
> >
> >
> > <stream name="fileIn" format="fixedlength">
> >
> >
> > <record name="header" order="1" minOccurs="1" maxOccurs="1"
> > class="foo.Header">
> > <field name="identifier" rid="true" literal="P"
> > position="0" length="1" />
> > <field name="pid" position="4" length="6" />
> > <field name="pidPassword" position="11" length="8" />
> > <field name="sid" position="24" length="6" />
> > <field name="sidPassword" position="31" length="8" />
> > <field name="directive" position="40" length="6" />
> > </record>
> >
> > <group name="transaction" order="2" minOccurs="1"
> > maxOccurs="unbounded" class="foo.Transaction">
> >
> > <record name="summary" minOccurs="0" maxOccurs="1"
> > class="foo.Summary">
> > <field name="identifier" rid="true" literal="S"
> > position="0" length="1" />
> >
> > <field name="reportGroup" position="1" length="10" />
> > <field name="orderId" position="11" length="8" />
> > <field name="batchId" position="19" length="14" />
> > <field name="actionCode" position="33" length="2" />
> >
> > <segment name="group" class="foo.Group">
> > <field name="my" position="35" length="2" />
> > <field name="group" position="37" length="19" />
> > <field name="info" position="56" length="4" />
> > </segment>
> > <field name="source" position="60" length="12" />
> > <field name="integerOrderSource" position="78"
> > length="1" />
> > <field name="activityBranch" position="112" length="4"
> > />
> > </record>
> >
> > <record name="company" class="foo.Company" minOccurs="0"
> > maxOccurs="1">
> > <field name="companyName" rid="true" literal="AB"
> > position="0" length="2" />
> > <field name="companyName" position="2" length="29" />
> > </record>
> > <record name="addressList" collection="list"
> > class="foo.Addressline" minOccurs="0" maxOccurs="unbounded">
> > <field name="identifier" rid="true" literal="A"
> > position="0" length="1" />
> > <field name="address" position="2" length="29" />
> > </record>
> > </group>
> >
> > <record name="BRec" order="4" class="foo.Rec" minOccurs="0"
> > maxOccurs="1">
> > <field name="identifier" rid="true" literal="B"
> > position="0" length="1" />
> > <field name="record" position="1" length="unbounded" />
> > </record>
> > <record name="TRec" order="5" class="foo.Rec" minOccurs="0"
> > maxOccurs="1">
> > <field name="identifier" rid="true" literal="T"
> > position="0" length="1" />
> > <field name="record" position="1" length="unbounded" />
> > </record>
> > <record name="footer" order="6" class="foo.Footer"
> > minOccurs="0" maxOccurs="1">
> > <field name="identifier" rid="true" literal="PID"
> > position="0" length="3" />
> >
> > </record>
> >
> > </stream>
>
