Hello,
We have now managed to verify a limited time and memory consumption in
a solution that is implemented using two nested Split EIPs and passing
the result to a SQL Producer endpoint.
I.e.
int batchSize = 1000;
from("file:test?readLock=changed")
.transacted("txRequired")
.setProperty("HeaderRows", new ArrayList<HeaderBindy>())
.doTry()
.split(body().tokenize("\n", batchSize, false)).streaming()
.setProperty("BodyRows", new ArrayList<BodyBindy>())
.split(body().tokenize("\n")).streaming()
.choice()
.when(body().startsWith("001"))
.unmarshal(new BindyFixedLengthDataFormat(HeaderBindy.class)
.process(exchange -> {
@SuppressWarnings("unchecked")
List<HeaderBindy> headerRows =
exchange.getProperty("HeaderRows", List.class);
HeaderBindy headerRow =
exchange.getMessage().getBody(HeaderBindy.class);
if (null != headerRow) {
headerRows.add(headerRow);
}
})
.when(body().startsWith("010"))
.unmarshal(new BindyFixedLengthDataFormat(BodyBindy.class)
.process(exchange -> {
@SuppressWarnings("unchecked")
List<BodyBindy> bodyRows =
exchange.getProperty("BodyRows", List.class);
BodyBindy bodyRow =
exchange.getMessage().getBody(BodyBindy.class);
if (null != bodyRow) {
bodyRows.add(bodyRow);
}
})
.otherwise()
.throwException(RuntimeException.class, "Unknown
record: ${body}")
.end() // .choice
.end() // .split rows
.setBody(exchangeProperty("BodyRows"))
.to("sql:...${body.xxx}?batch=true")
.end() // .split batch
.endDoTry()
.doCatch(Throwable.class)
.to("mock:exception")
.process(exchange -> {
Throwable t =
exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Throwable.class);
throw new RollbackExchangeException(exchange, t);
})
.end()
Two things have been noted on finding this solution:
- Loop EIP can't be used since it doesn't propagate the transacted
statement from the route
- Don't use PreparedStatement to an embedded derby DataSource when
measuring memory consumption
Mvh / Best regards,
Håkan Lantz
[email protected]
www.replyto.se
On Fri, 11 Jul 2025 at 20:19, Claus Ibsen <[email protected]> wrote:
>
> Hi
>
> Its easier to help if you put a sample project somewhere on github and also
> make sure it can be build and run easily, and that you try with latest
> releases also.
>
> On Fri, Jul 11, 2025 at 9:34 AM Håkan Lantz <[email protected]> wrote:
>
> > Hello,
> >
> > We have a large file that we would like to transactionally read row by
> > row into a database using Bindy objects without blowing up the Java
> > heap.
> >
> > Several attempts has been made using nested Split EIP and SQL producer
> > with batch constructs. One of them we used a combination Split & Loop
> > EIP with temp files.
> >
> > In that process we discovered that the Loop EIP doesn't copy the
> > "Transacted" state from the exchange in the route and then SQL
> > producer will update the database with AutoCommit.
> >
> > First question is that intended and if so is it documented or should
> > that be reported as a bug?
> >
> > The second question is if anyone else has solved this problem and are
> > willing to give us som tips?
> >
> > Mvh / Best regards,
> > Håkan Lantz
> >
>
>
> --
> Claus Ibsen
