Hi Charles And the "trailer" is something that flatpack support out-of-the-box? I am by no means an expert on flatpack but are you sure its something it understands?
In the camel-flatpack code there is an option to skip the first record, but none for the last record. So I am just digging here. Feel free to put together a small junit test that we can add to the test in camel-flatpack where the input file has both a header and a trailer row. Then we can use that as a base for a bugfix. You can use some of the existing unit tests as inspiration how to add your unit test. Med venlig hilsen Claus Ibsen ...................................... Silverbullet Skovsgårdsvænget 21 8362 Hørning Tlf. +45 2962 7576 Web: www.silverbullet.dk -----Original Message----- From: cmoulliard [mailto:[EMAIL PROTECTED] Sent: 16. juli 2008 10:21 To: [email protected] Subject: Re: Component to read FixedLength file ? I have a new error when I use the following routes. from("file:///c:/temp/cutas/?delete=false&lock=true&consumer.delay=500&consumer.initialDelay=10") .to("flatpack:fixed:com/company/transformation/in/CUTAS-ST.pzmap.xml"); from("flatpack:fixed:com/company/cutas/transformation/in/CUTAS-ST.pzmap.xml") .convertBodyTo(Map.class) .to("activemq:queue:cutas.in.queue"); and xml file <?xml version='1.0'?> <!DOCTYPE PZMAP SYSTEM "flatpack.dtd" > <PZMAP> <RECORD id="header" startPosition="1" endPosition="3" indicator="HBT"> <COLUMN name="INDICATOR" length="3" /> <COLUMN name="HEADERDATA" length="26" /> </RECORD> <COLUMN name="RECORDTYPE" length="1" /> <COLUMN name="SYSTEMID" length="3" /> <COLUMN name="THIRD_PARTY_REF" length="15" /> <COLUMN name="CUTAS_ACCOUNT_NR" length="15" /> <COLUMN name="FUND_CODE" length="5" /> <COLUMN name="PRODUCT_CODE" length="5" /> <COLUMN name="PRODUCT_TYPE" length="4" /> <COLUMN name="CUTAS_REPLY_DATE" length="8" /> <COLUMN name="CUTAS_REPLY_HOUR" length="4" /> <COLUMN name="TRANSACTION_TYPE" length="2" /> <COLUMN name="FUND_TYPE" length="1" /> <COLUMN name="UNIT_SIGN" length="1" /> <COLUMN name="DATA" length="412" /> <RECORD id="trailer" startPosition="1" endPosition="3" indicator="FBT"> <COLUMN name="INDICATOR" length="3" /> <COLUMN name="TRAILERDATA" length="26" /> </RECORD> </PZMAP> Remark : In my case, we have a header / Trailer in the message. Error reported : 43org.apache.camel.RuntimeCamelException: java.util.NoSuchElementException: Column [recordtype] does not exist, check case/spelling. key:trailer at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:411) at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:42) at org.apache.camel.impl.converter.DefaultTypeConverter.convertTo(DefaultTypeConverter.java:67) at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:59) at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:50) at org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:41) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155) at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91) at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) at org.apache.camel.processor.Pipeline.process(Pipeline.java:85) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:43) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:39) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:66) at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:41) at org.apache.camel.component.flatpack.FixedLengthEndpoint.processDataSet(FixedLengthEndpoint.java:67) at org.apache.camel.component.flatpack.FlatpackProducer.process(FlatpackProducer.java:41) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155) at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:43) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:39) at org.apache.camel.component.file.FileConsumer.pollFile(FileConsumer.java:128) at org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:77) at org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:84) at org.apache.camel.component.file.FileConsumer.poll(FileConsumer.java:57) at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:65) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.util.NoSuchElementException: Column [recordtype] does not exist, check case/spelling. key:trailer at net.sf.flatpack.util.ParserUtils.getColumnIndex(ParserUtils.java:785) at net.sf.flatpack.DefaultDataSet.getStringValue(DefaultDataSet.java:184) at net.sf.flatpack.DefaultDataSet.getString(DefaultDataSet.java:262) at org.apache.camel.component.flatpack.FlatpackConverter.putValues(FlatpackConverter.java:48) at org.apache.camel.component.flatpack.FlatpackConverter.toMap(FlatpackConverter.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:407) ... 55 more Content of the message : 4349422 [Thread: 1 [EMAIL PROTECTED] INFO org.apache.camel.processor.interceptor.TraceInterceptor - ID-dell-charles/1446-1216195452000/2-0 -> to7 To[flatpack:fixed:com/bravura/babel/cutas/transformation/in/CUTAS.xml] InOnly Properties:{CamelCauseException=null, org.apache.camel.file.lock.name=c:\temp\cutas\ST000355.txt.cameLock, org.apache.camel.file.lock=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid], [EMAIL PROTECTED] Headers:{org.apache.camel.file.name=ST000355.txt} BodyType:java.io.File Body:HBT00032700000327190620071714 C00111a30571b6b0008Z001412 FIEU BFM 200805271554 +0000000010000000000009076000000138137A000090760EUR GBPSS808800714636FIEU /0008031800000000000000000000000000000000000000000000000000090760000000000000000000000000000000000138137413092007000000000000000000001522000 00001522000 000000000000000O 0000000000000000 20070907155200 X C00111a30571b170007B000005 APAA 200805271554 +0000000001835910000003941700000039417A000332147USD GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 000000000000000000000000000 00000000000 000000000000000B 0000000000000000 20070619101000 X C00111a30571a600006B000005 APAA 200805271554 +0000000001855550000003941700000039417A000052147GBP GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 000000000000000000000000000 00000000000 000000000000000X 0000000000000000 20070619101000 X C00111a30571c2a0009B000005 APAA 200805271554 +0000000001855550000003941700000039417A000052147GBP GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 000000000000000000000000000 00000000000 000000000000000 0000000000000000 20070619101000 X FBT00032700000001190620071714 Info displayed on the console : 4349609 [Thread: 1 [EMAIL PROTECTED] INFO org.apache.camel.processor.interceptor.TraceInterceptor - ID-dell-charles/1446-1216195452000/2-1 -> interceptor4 Interceptor[Delegate(Pipeline[DeadLetterChannel[Delegate(TraceInterceptor[convertBodyTo[ null]]), sendTo(Endpoint[seda:errors]), RedeliveryPolicy[maximumRedeliveries=1]], DeadLetterChannel[Delegate(TraceInterceptor[To[activemq:queue:cutas.in.queue]]), sendTo(Endpoint[seda:errors]), RedeliveryPolicy[maximumRedeliveries=1]]])] InOnly Properties:{} Headers:{camelFlatpackCounter=0} BodyType:net.sf.flatpack.DefaultDataSet Body:Errors:0 Rows:6 Position:0 Conversion Props:{java.lang.Double=net.sf.flatpack.converter.ConvertDouble, java.math.BigDecimal=net.sf.flatpack.converter.ConvertBigDecimal, java.lang.Integer=net.sf.flatpack.converter.ConvertInteger} MetaData:Col Names:[Name:RECORDTYPE Length:1 Start:0 End:0 , Name:SYSTEMID Length:3 Start:0 End:0 , Name:THIRD_PARTY_REF Length:15 Start:0 End:0 , Name:CUTAS_ACCOUNT_NR Length:15 Start:0 End:0 , Name:FUND_CODE Length:5 Start:0 End:0 , Name:PRODUCT_CODE Length:5 Start:0 End:0 , Name:PRODUCT_TYPE Length:4 Start:0 End:0 , Name:CUTAS_REPLY_DATE Length:8 Start:0 End:0 , Name:CUTAS_REPLY_HOUR Length:4 Start:0 End:0 , Name:TRANSACTION_TYPE Length:2 Start:0 End:0 , Name:FUND_TYPE Length:1 Start:0 End:0 , Name:UNIT_SIGN Length:1 Start:0 End:0 , Name:DATA Length:412 Start:0 End:0 ] Col Index Map:{unit_sign=11, transaction_type=9, fund_type=10, systemid=1, cutas_account_nr=3, product_code=5, product_type=6, recordtype=0, data=12, cutas_reply_date=7, third_party_ref=2, fund_code=4, cutas_reply_hour=8} XML Record Elements:[EMAIL PROTECTED], [EMAIL PROTECTED] Regards, Charles James.Strachan wrote: > > FWIW I've just checked in a FlatpackDataFormat; which is a tad easier > to use. The downside is the data format kinda has to turn the DataSet > into a List of Maps to make it easier to work with; but at least the > routes are a bit simpler. e.g. this is from the > DelimitedWithUnmarshalTest (the convertBodyTo is not required > currently) > > <camelContext xmlns="http://activemq.apache.org/camel/schema/spring";> > <route> > <from uri="file://src/test/data/delim?noop=true"/> > <unmarshal ref="delimitedFormat"/> > <splitter> > <simple>in.body</simple> > <convertBodyTo type="java.util.Map"/> > <to uri="mock:results"/> > </splitter> > </route> > </camelContext> > > <bean id="delimitedFormat" > class="org.apache.camel.component.flatpack.FlatpackDataFormat"> > <property name="definition" value="INVENTORY-Delimited.pzmap.xml"/> > </bean> > > 2008/7/15 James Strachan <[EMAIL PROTECTED]>: >> Yeah - I did ponder if it should be a dataformat or a component; or >> maybe even both. The bit I wasn't sure about was flatpack really just >> provides an iterator over rows of the file; I wasn't sure if this >> properly matched the dataformat model of marshalling or unmarshalling >> a payload (which currently assumes that unmarshalling generates a >> single result). >> >> Maybe we need to enhance the dataformat concept to work with this use >> case too? >> >> So maybe if we want to process each row of a file we do something like... >> >> from("file://whatnot"). >> >> unmarshall().flatpack().fixed("locationOftheXMLConfigurationfile.pzmap.xml"). >> split().body(). >> to("whatever"); >> >> where the split().body() would do the iteration over each DataSet? >> >> >> 2008/7/15 Claus Ibsen <[EMAIL PROTECTED]>: >>> Hi >>> >>> Yes I thought also at first it was a dataformat ;) But how James created >>> it it works as if it is a dataformat. You just have to do it in to..from >>> from..to steps. >>> >>> >>> from("file://c:/test").to("flatpack:foo: >>> locationOftheXMLConfigurationfile.pzmap.xml"); >>> >>> from("flatpack:foo").to("whatever you want"); >>> >>> As a component it gets more attention, as people can find it on the >>> component list. The dataformats are kinda hidden in the wiki >>> documentation. >>> Maybe they need a "dataformat" top menu as well ;) >>> >>> Med venlig hilsen >>> >>> Claus Ibsen >>> ...................................... >>> Silverbullet >>> Skovsgårdsvænget 21 >>> 8362 Hørning >>> Tlf. +45 2962 7576 >>> Web: www.silverbullet.dk >>> >>> -----Original Message----- >>> From: cmoulliard [mailto:[EMAIL PROTECTED] >>> Sent: 15. juli 2008 09:32 >>> To: [email protected] >>> Subject: Re: Component to read FixedLength file ? >>> >>> >>> Many thanks. >>> >>> I will test the integration of this component within Camel. >>> >>> Remark : I'm a little confused about how Flatpack has been integrated. >>> The >>> project has been integrated like a component but from my point of view, >>> it >>> should be integrated (for consistency reason) like a Dataformat. Because >>> like the other framework/technology implemented in Camel (JAXB, Artix, >>> XStream, ....), they transform the content from a format (XML, CSV, ...) >>> into another format (Collection of Java objects) >>> >>> E.g. >>> >>> DataFormat flatpack = new >>> DataFormat("locationOftheXMLConfigurationfile.pzmap.xml.xml") >>> >>> from("file:///c:/test") >>> .unmarshall(flatpack) >>> .to("whatever you want") >>> >>> Regards, >>> >>> Charles >>> >>> >>> >>> James.Strachan wrote: >>>> >>>> BTW trunk has an early spike of a flatpack component. Needs more work, >>>> but there's a basic fixed-width reader example there for the curious >>>> >>>> 2008/7/14 James Strachan <[EMAIL PROTECTED]>: >>>>> I've raised a JIRA to track this... >>>>> https://issues.apache.org/activemq/browse/CAMEL-717 >>>>> >>>>> 2008/7/14 James Strachan <[EMAIL PROTECTED]>: >>>>>> 2008/7/14 cmoulliard <[EMAIL PROTECTED]>: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Is there a component expected to read fixedlength files in Camel >>>>>>> like >>>>>>> flatpack (http://flatpack.sourceforge.net/) allow to do it ? >>>>>> >>>>>> It should be pretty easy to write a flatpack component. >>>>>> >>>>>> there's a commercial tool called Artix Data Services which comes with >>>>>> a nice design UI that does this with Camel... >>>>>> http://activemq.apache.org/camel/artix-data-services.html >>>>>> >>>>>> but it'd be nice to have an OSS option :) >>>>>> >>>>>> -- >>>>>> James >>>>>> ------- >>>>>> http://macstrac.blogspot.com/ >>>>>> >>>>>> Open Source Integration >>>>>> http://open.iona.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> James >>>>> ------- >>>>> http://macstrac.blogspot.com/ >>>>> >>>>> Open Source Integration >>>>> http://open.iona.com >>>>> >>>> >>>> >>>> >>>> -- >>>> James >>>> ------- >>>> http://macstrac.blogspot.com/ >>>> >>>> Open Source Integration >>>> http://open.iona.com >>>> >>>> >>> >>> >>> ----- >>> Enterprise Architect >>> >>> Xpectis >>> 12, route d'Esch >>> L-1470 Luxembourg >>> >>> Phone +352 25 10 70 470 >>> Mobile +352 621 45 36 22 >>> >>> e-mail : [EMAIL PROTECTED] >>> web site : www.xpectis.com >>> -- >>> View this message in context: >>> http://www.nabble.com/Component-to-read-FixedLength-file---tp18444614s22882p18459545.html >>> Sent from the Camel - Users mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> James >> ------- >> http://macstrac.blogspot.com/ >> >> Open Source Integration >> http://open.iona.com >> > > > > -- > James > ------- > http://macstrac.blogspot.com/ > > Open Source Integration > http://open.iona.com > > ----- Enterprise Architect Xpectis 12, route d'Esch L-1470 Luxembourg Phone +352 25 10 70 470 Mobile +352 621 45 36 22 e-mail : [EMAIL PROTECTED] web site : www.xpectis.com -- View this message in context: http://www.nabble.com/Component-to-read-FixedLength-file---tp18444614s22882p18482019.html Sent from the Camel - Users mailing list archive at Nabble.com.
