Hi Charles A workaround for your current problem is to make a mistake in the header/trailer definition and flatpack will skip it, at least in my current unit test.
Med venlig hilsen Claus Ibsen ...................................... Silverbullet Skovsgårdsvænget 21 8362 Hørning Tlf. +45 2962 7576 Web: www.silverbullet.dk -----Original Message----- From: cmoulliard [mailto:[EMAIL PROTECTED] Sent: 16. juli 2008 11:11 To: [email protected] Subject: Re: Component to read FixedLength file ? The error comes from the fact that the convertBody(Map.class) does not take into account that the first record and the last one are the header/trailer of the message. In this case, three different map have to be created : - Header, - Body - Trailer The property of the message and attribute of the DSL/URI must be adapted to use a new keywork -> flatpack:fixedHeader instead of flatpack:fixed to inform the Body converter How to transform the net.sf.flatpack.DefaultDataSet into their three Maps Regards, Charles cmoulliard wrote: > > I have a new error when I use the following routes. > > from("file:///c:/temp/cutas/?delete=false&lock=true&consumer.delay=500&consumer.initialDelay=10") > > .to("flatpack:fixed:com/company/transformation/in/CUTAS-ST.pzmap.xml"); > > > > from("flatpack:fixed:com/company/cutas/transformation/in/CUTAS-ST.pzmap.xml") > .convertBodyTo(Map.class) > .to("activemq:queue:cutas.in.queue"); > > and xml file > > <?xml version='1.0'?> > <!DOCTYPE PZMAP SYSTEM "flatpack.dtd" > > > <PZMAP> > <RECORD id="header" startPosition="1" endPosition="3" indicator="HBT"> > <COLUMN name="INDICATOR" length="3" /> > <COLUMN name="HEADERDATA" length="26" /> > </RECORD> > <COLUMN name="RECORDTYPE" length="1" /> > <COLUMN name="SYSTEMID" length="3" /> > <COLUMN name="THIRD_PARTY_REF" length="15" /> > <COLUMN name="CUTAS_ACCOUNT_NR" length="15" /> > <COLUMN name="FUND_CODE" length="5" /> > <COLUMN name="PRODUCT_CODE" length="5" /> > <COLUMN name="PRODUCT_TYPE" length="4" /> > <COLUMN name="CUTAS_REPLY_DATE" length="8" /> > <COLUMN name="CUTAS_REPLY_HOUR" length="4" /> > <COLUMN name="TRANSACTION_TYPE" length="2" /> > <COLUMN name="FUND_TYPE" length="1" /> > <COLUMN name="UNIT_SIGN" length="1" /> > <COLUMN name="DATA" length="412" /> > > <RECORD id="trailer" startPosition="1" endPosition="3" indicator="FBT"> > <COLUMN name="INDICATOR" length="3" /> > <COLUMN name="TRAILERDATA" length="26" /> > </RECORD> > > </PZMAP> > > Remark : In my case, we have a header / Trailer in the message. > > Error reported : > > > 43org.apache.camel.RuntimeCamelException: > java.util.NoSuchElementException: Column [recordtype] does not exist, > check case/spelling. key:trailer > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:411) > at > org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:42) > at > org.apache.camel.impl.converter.DefaultTypeConverter.convertTo(DefaultTypeConverter.java:67) > at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:59) > at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:50) > at > org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:41) > at > org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) > at > org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) > at > org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155) > at > org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:85) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) > at > org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) > at > org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:43) > at > org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:39) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) > at > org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:66) > at > org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:41) > at > org.apache.camel.component.flatpack.FixedLengthEndpoint.processDataSet(FixedLengthEndpoint.java:67) > at > org.apache.camel.component.flatpack.FlatpackProducer.process(FlatpackProducer.java:41) > at > org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61) > at > org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) > at > org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) > at > org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155) > at > org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:50) > at > org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:79) > at > org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:57) > at > org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:43) > at > org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:39) > at > org.apache.camel.component.file.FileConsumer.pollFile(FileConsumer.java:128) > at > org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:77) > at > org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:84) > at > org.apache.camel.component.file.FileConsumer.poll(FileConsumer.java:57) > at > org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:65) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source) > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.util.NoSuchElementException: Column [recordtype] does not > exist, check case/spelling. key:trailer > at net.sf.flatpack.util.ParserUtils.getColumnIndex(ParserUtils.java:785) > at > net.sf.flatpack.DefaultDataSet.getStringValue(DefaultDataSet.java:184) > at net.sf.flatpack.DefaultDataSet.getString(DefaultDataSet.java:262) > at > org.apache.camel.component.flatpack.FlatpackConverter.putValues(FlatpackConverter.java:48) > at > org.apache.camel.component.flatpack.FlatpackConverter.toMap(FlatpackConverter.java:35) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:407) > ... 55 more > > Content of the message : > > 4349422 [Thread: 1 [EMAIL PROTECTED] > INFO org.apache.camel.processor.interceptor.TraceInterceptor - > ID-dell-charles/1446-1216195452000/2-0 -> to7 > To[flatpack:fixed:com/bravura/babel/cutas/transformation/in/CUTAS.xml] > InOnly Properties:{CamelCauseException=null, > org.apache.camel.file.lock.name=c:\temp\cutas\ST000355.txt.cameLock, > org.apache.camel.file.lock=sun.nio.ch.FileLockImpl[0:9223372036854775807 > exclusive valid], > [EMAIL PROTECTED] > Headers:{org.apache.camel.file.name=ST000355.txt} BodyType:java.io.File > Body:HBT00032700000327190620071714 > C00111a30571b6b0008Z001412 FIEU BFM 200805271554 > +0000000010000000000009076000000138137A000090760EUR > GBPSS808800714636FIEU > /0008031800000000000000000000000000000000000000000000000000090760000000000000000000000000000000000138137413092007000000000000000000001522000 > 00001522000 000000000000000O 0000000000000000 > > 20070907155200 > > X > C00111a30571b170007B000005 APAA 200805271554 > +0000000001835910000003941700000039417A000332147USD > GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 > > 000000000000000000000000000 00000000000 000000000000000B > 0000000000000000 20070619101000 > > X > C00111a30571a600006B000005 APAA 200805271554 > +0000000001855550000003941700000039417A000052147GBP > GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 > > 000000000000000000000000000 00000000000 000000000000000X > 0000000000000000 20070619101000 > > X > C00111a30571c2a0009B000005 APAA 200805271554 > +0000000001855550000003941700000039417A000052147GBP > GBPAGENT188888888BDBTA/0000292200000000000000000000000000000000000000000000000039417GBP000000000000000000000000000000000039417 > > 000000000000000000000000000 00000000000 000000000000000 > 0000000000000000 20070619101000 > > X > FBT00032700000001190620071714 > > Info displayed on the console : > > 4349609 [Thread: 1 [EMAIL PROTECTED] > INFO org.apache.camel.processor.interceptor.TraceInterceptor - > ID-dell-charles/1446-1216195452000/2-1 -> interceptor4 > Interceptor[Delegate(Pipeline[DeadLetterChannel[Delegate(TraceInterceptor[convertBodyTo[ > null]]), sendTo(Endpoint[seda:errors]), > RedeliveryPolicy[maximumRedeliveries=1]], > DeadLetterChannel[Delegate(TraceInterceptor[To[activemq:queue:cutas.in.queue]]), > sendTo(Endpoint[seda:errors]), RedeliveryPolicy[maximumRedeliveries=1]]])] > InOnly Properties:{} Headers:{camelFlatpackCounter=0} > BodyType:net.sf.flatpack.DefaultDataSet Body:Errors:0 > Rows:6 > Position:0 > Conversion > Props:{java.lang.Double=net.sf.flatpack.converter.ConvertDouble, > java.math.BigDecimal=net.sf.flatpack.converter.ConvertBigDecimal, > java.lang.Integer=net.sf.flatpack.converter.ConvertInteger} > MetaData:Col Names:[Name:RECORDTYPE Length:1 Start:0 End:0 > , Name:SYSTEMID Length:3 Start:0 End:0 > , Name:THIRD_PARTY_REF Length:15 Start:0 End:0 > , Name:CUTAS_ACCOUNT_NR Length:15 Start:0 End:0 > , Name:FUND_CODE Length:5 Start:0 End:0 > , Name:PRODUCT_CODE Length:5 Start:0 End:0 > , Name:PRODUCT_TYPE Length:4 Start:0 End:0 > , Name:CUTAS_REPLY_DATE Length:8 Start:0 End:0 > , Name:CUTAS_REPLY_HOUR Length:4 Start:0 End:0 > , Name:TRANSACTION_TYPE Length:2 Start:0 End:0 > , Name:FUND_TYPE Length:1 Start:0 End:0 > , Name:UNIT_SIGN Length:1 Start:0 End:0 > , Name:DATA Length:412 Start:0 End:0 > ] > Col Index Map:{unit_sign=11, transaction_type=9, fund_type=10, systemid=1, > cutas_account_nr=3, product_code=5, product_type=6, recordtype=0, data=12, > cutas_reply_date=7, third_party_ref=2, fund_code=4, cutas_reply_hour=8} > XML Record Elements:[EMAIL PROTECTED], > [EMAIL PROTECTED] > > Regards, > > Charles > > > > James.Strachan wrote: >> >> FWIW I've just checked in a FlatpackDataFormat; which is a tad easier >> to use. The downside is the data format kinda has to turn the DataSet >> into a List of Maps to make it easier to work with; but at least the >> routes are a bit simpler. e.g. this is from the >> DelimitedWithUnmarshalTest (the convertBodyTo is not required >> currently) >> >> <camelContext xmlns="http://activemq.apache.org/camel/schema/spring";> >> <route> >> <from uri="file://src/test/data/delim?noop=true"/> >> <unmarshal ref="delimitedFormat"/> >> <splitter> >> <simple>in.body</simple> >> <convertBodyTo type="java.util.Map"/> >> <to uri="mock:results"/> >> </splitter> >> </route> >> </camelContext> >> >> <bean id="delimitedFormat" >> class="org.apache.camel.component.flatpack.FlatpackDataFormat"> >> <property name="definition" value="INVENTORY-Delimited.pzmap.xml"/> >> </bean> >> >> 2008/7/15 James Strachan <[EMAIL PROTECTED]>: >>> Yeah - I did ponder if it should be a dataformat or a component; or >>> maybe even both. The bit I wasn't sure about was flatpack really just >>> provides an iterator over rows of the file; I wasn't sure if this >>> properly matched the dataformat model of marshalling or unmarshalling >>> a payload (which currently assumes that unmarshalling generates a >>> single result). >>> >>> Maybe we need to enhance the dataformat concept to work with this use >>> case too? >>> >>> So maybe if we want to process each row of a file we do something >>> like... >>> >>> from("file://whatnot"). >>> >>> unmarshall().flatpack().fixed("locationOftheXMLConfigurationfile.pzmap.xml"). >>> split().body(). >>> to("whatever"); >>> >>> where the split().body() would do the iteration over each DataSet? >>> >>> >>> 2008/7/15 Claus Ibsen <[EMAIL PROTECTED]>: >>>> Hi >>>> >>>> Yes I thought also at first it was a dataformat ;) But how James >>>> created it it works as if it is a dataformat. You just have to do it in >>>> to..from from..to steps. >>>> >>>> >>>> from("file://c:/test").to("flatpack:foo: >>>> locationOftheXMLConfigurationfile.pzmap.xml"); >>>> >>>> from("flatpack:foo").to("whatever you want"); >>>> >>>> As a component it gets more attention, as people can find it on the >>>> component list. The dataformats are kinda hidden in the wiki >>>> documentation. >>>> Maybe they need a "dataformat" top menu as well ;) >>>> >>>> Med venlig hilsen >>>> >>>> Claus Ibsen >>>> ...................................... >>>> Silverbullet >>>> Skovsgårdsvænget 21 >>>> 8362 Hørning >>>> Tlf. +45 2962 7576 >>>> Web: www.silverbullet.dk >>>> >>>> -----Original Message----- >>>> From: cmoulliard [mailto:[EMAIL PROTECTED] >>>> Sent: 15. juli 2008 09:32 >>>> To: [email protected] >>>> Subject: Re: Component to read FixedLength file ? >>>> >>>> >>>> Many thanks. >>>> >>>> I will test the integration of this component within Camel. >>>> >>>> Remark : I'm a little confused about how Flatpack has been integrated. >>>> The >>>> project has been integrated like a component but from my point of view, >>>> it >>>> should be integrated (for consistency reason) like a Dataformat. >>>> Because >>>> like the other framework/technology implemented in Camel (JAXB, Artix, >>>> XStream, ....), they transform the content from a format (XML, CSV, >>>> ...) >>>> into another format (Collection of Java objects) >>>> >>>> E.g. >>>> >>>> DataFormat flatpack = new >>>> DataFormat("locationOftheXMLConfigurationfile.pzmap.xml.xml") >>>> >>>> from("file:///c:/test") >>>> .unmarshall(flatpack) >>>> .to("whatever you want") >>>> >>>> Regards, >>>> >>>> Charles >>>> >>>> >>>> >>>> James.Strachan wrote: >>>>> >>>>> BTW trunk has an early spike of a flatpack component. Needs more work, >>>>> but there's a basic fixed-width reader example there for the curious >>>>> >>>>> 2008/7/14 James Strachan <[EMAIL PROTECTED]>: >>>>>> I've raised a JIRA to track this... >>>>>> https://issues.apache.org/activemq/browse/CAMEL-717 >>>>>> >>>>>> 2008/7/14 James Strachan <[EMAIL PROTECTED]>: >>>>>>> 2008/7/14 cmoulliard <[EMAIL PROTECTED]>: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Is there a component expected to read fixedlength files in Camel >>>>>>>> like >>>>>>>> flatpack (http://flatpack.sourceforge.net/) allow to do it ? >>>>>>> >>>>>>> It should be pretty easy to write a flatpack component. >>>>>>> >>>>>>> there's a commercial tool called Artix Data Services which comes >>>>>>> with >>>>>>> a nice design UI that does this with Camel... >>>>>>> http://activemq.apache.org/camel/artix-data-services.html >>>>>>> >>>>>>> but it'd be nice to have an OSS option :) >>>>>>> >>>>>>> -- >>>>>>> James >>>>>>> ------- >>>>>>> http://macstrac.blogspot.com/ >>>>>>> >>>>>>> Open Source Integration >>>>>>> http://open.iona.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> James >>>>>> ------- >>>>>> http://macstrac.blogspot.com/ >>>>>> >>>>>> Open Source Integration >>>>>> http://open.iona.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> James >>>>> ------- >>>>> http://macstrac.blogspot.com/ >>>>> >>>>> Open Source Integration >>>>> http://open.iona.com >>>>> >>>>> >>>> >>>> >>>> ----- >>>> Enterprise Architect >>>> >>>> Xpectis >>>> 12, route d'Esch >>>> L-1470 Luxembourg >>>> >>>> Phone +352 25 10 70 470 >>>> Mobile +352 621 45 36 22 >>>> >>>> e-mail : [EMAIL PROTECTED] >>>> web site : www.xpectis.com >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Component-to-read-FixedLength-file---tp18444614s22882p18459545.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> James >>> ------- >>> http://macstrac.blogspot.com/ >>> >>> Open Source Integration >>> http://open.iona.com >>> >> >> >> >> -- >> James >> ------- >> http://macstrac.blogspot.com/ >> >> Open Source Integration >> http://open.iona.com >> >> > > ----- Enterprise Architect Xpectis 12, route d'Esch L-1470 Luxembourg Phone +352 25 10 70 470 Mobile +352 621 45 36 22 e-mail : [EMAIL PROTECTED] web site : www.xpectis.com -- View this message in context: http://www.nabble.com/Component-to-read-FixedLength-file---tp18444614s22882p18483512.html Sent from the Camel - Users mailing list archive at Nabble.com.
