Re: JPA Component Behavior
Hi Claus,I'm very eager to get that feature, too. My requirements for a new project are: run on every 1st of a month (or some other Quartz clause) and then select all objects with status = 0 have a user interface for an administrator in which he/she can customize some search arguments such as a time range in which a JPA object was created, different values for the status field, and so on. I'm surely fit enough to deliver such a UI, e.g. as JEE2. From that UI, I can produce a route with customized query criteria. But the point is: how do I make the JPA consumer run on a button click? In my opinion, scheduled poll consuming is not the only sceanrio in which database or JPA retrieval is required. Do you see a way with which I can achieve that? Or do I have to wait? -- View this message in context: http://camel.465427.n5.nabble.com/JPA-Component-Behavior-tp5725781p5731377.html Sent from the Camel - Users mailing list archive at Nabble.com.
how to read a large amount of data in a 2nd step
Hi there, maybe I am running totally in the wrong direction because I'm a Camel rookie. But what I try to do is the following: I have database table of orders and a 2nd db table of order lines. A single order may have 10 order lines. Each order with all its order lines has to be stored in single csv files. My idea to achieve that was to use the Camel JPA component to read the orders. This is very suitable because the orders have a "status" field which can be updated by the JPA component. I've managed to build the individual csv files and to fill them with the order informations. But now I'm running into trouble. The table with the order lines can't be read by the JPA component because this component is only running in a sort of polling mode - it would be easier for me if this component could be started just like the jdcb or the sql component. So, I tried to implement a jdbc or a sql component. But now I don't know how to set up their connection parameters from the configuration of the JPA component. Is there a way to get the datasource of the JPA component and to stuck it into the SQL or der JDBC component? Or did I oversee something, and my problem could be solved in a completely different way? Many thanks in advance for any help! Christian -- View this message in context: http://camel.465427.n5.nabble.com/how-to-read-a-large-amount-of-data-in-a-2nd-step-tp5731752.html Sent from the Camel - Users mailing list archive at Nabble.com.
share database connection properties between JPA and JDBC consumer
Hi there,maybe I'm running in the totally wrong direction. My problem is this: I have a database table "order" and a 2nd database table "orderlines". My task is to produce a csv file for each "order" entry with a value of 0 in a column "status". This csv file should contain some informations of the "order" table and all entries of the "orderlines" table that belong to the specific order. There can be millions of orderline entries belonging to one order entry. My first step was to use a JPA component to read the order entries and produce the first couple of lines in the csv files. I used the annotation @Consumed to change the value of the column "status". Then I planned to use the JDBC or the SQL component in order to read the orderlines (of course with streaming) and to append them at the csv file. Where I now get stucked is, how to set properties for the DB connection of the JDBC or SQL component with the DB connection settings that I had used in the JPA component. I tried very hard to get a hand on the EntityManager, but I tried in vain. Has anybody an idea how to get that? Or a hint showing me how I can solve my problem in a complete different way? Many Thanks in advance! Christian -- View this message in context: http://camel.465427.n5.nabble.com/share-database-connection-properties-between-JPA-and-JDBC-consumer-tp5731923.html Sent from the Camel - Users mailing list archive at Nabble.com.
NullpointerException in ClassPathXmlApplicationContext
Hi there, I'm quite new to Camel and face some problems when building unit tests which extend CamelSpringTestSupport. In the constructor of ClassPathXmlApplicationContext, I oftenly (*but not always!*) get a NullpointerException. This is the stack trace: Can anybody help me? Many thanks in advance! -- View this message in context: http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: NullpointerException in ClassPathXmlApplicationContext
Hi Claus, I'm using JDK 1.7.0, Camel 2.10.3, and Spring 3.2.0.RELEASE, and I'm running my tests in Eclipse Juno. Thanks for your help. Do you need more informations? Regards, Christian -Ursprüngliche Mitteilung- Von: Claus Ibsen-2 [via Camel] An: Christian Jacob Verschickt: Do, 24 Jan 2013 9:32 pm Betreff: Re: NullpointerException in ClassPathXmlApplicationContext On Thu, Jan 24, 2013 at 9:29 PM, Christian Jacob <[hidden email]> wrote: > Hi there, > I'm quite new to Camel and face some problems when building unit tests which > extend CamelSpringTestSupport. In the constructor of > ClassPathXmlApplicationContext, I oftenly (*but not always!*) get a > NullpointerException. This is the stack trace: > > > Can anybody help me? Many thanks in advance! > What JDK, Camel and Spring version do you use? > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen - Red Hat, Inc. FuseSource is now part of Red Hat Email: [hidden email] Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen If you reply to this email, your message will be added to the discussion below: http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726190.html To unsubscribe from NullpointerException in ClassPathXmlApplicationContext, click here. NAML -- View this message in context: http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726202.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: NullpointerException in ClassPathXmlApplicationContext
Hi Claus and Christian, thanks for your support. Using JAXB 2.2.6 helped it out. But I will also check for a newer JDK version. Kind regards, Christian -Ursprüngliche Mitteilung- Von: Claus Ibsen-2 [via Camel] An: Christian Jacob Verschickt: Fr, 25 Jan 2013 7:42 am Betreff: Re: NullpointerException in ClassPathXmlApplicationContext On Fri, Jan 25, 2013 at 12:06 AM, Christian Müller <[hidden email]> wrote: > Camel 2.10.3 only supports Spring 3.0.x and 3.1.x. > Spring 3.2.x support is added in the upcoming Camel 2.11.0 release. > Which concrete JDK 1.7.0 version do you use? Some older ones had issues > with parsing XML documents. > Yeah the earlier version of JDK 1.7 was buggy. Also you may want to add JAXB 2.2.6 on the classpath which is newer and more reliable than what comes out of JDK. If you use Maven then camel-core has dependencies for that out of the box. > Sent from a mobile device > Am 24.01.2013 22:26 schrieb "Christian Jacob" <[hidden email]>: > >> >> Hi Claus, >> >> I'm using JDK 1.7.0, Camel 2.10.3, and Spring 3.2.0.RELEASE, and I'm >> running my tests in Eclipse Juno. >> >> Thanks for your help. Do you need more informations? >> >> Regards, >> Christian >> >> >> -Ursprüngliche Mitteilung- >> Von: Claus Ibsen-2 [via Camel] <[hidden email]> >> An: Christian Jacob <[hidden email]> >> Verschickt: Do, 24 Jan 2013 9:32 pm >> Betreff: Re: NullpointerException in ClassPathXmlApplicationContext >> >> >> On Thu, Jan 24, 2013 at 9:29 PM, Christian Jacob <[hidden email]> >> wrote: >> > Hi there, >> > I'm quite new to Camel and face some problems when building unit tests >> which >> > extend CamelSpringTestSupport. In the constructor of >> > ClassPathXmlApplicationContext, I oftenly (*but not always!*) >> get a >> > NullpointerException. This is the stack trace: >> > >> > >> > Can anybody help me? Many thanks in advance! >> > >> >> What JDK, Camel and Spring version do you use? >> >> >> >> > >> > >> > -- >> > View this message in context: >> http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189.html >> > Sent from the Camel - Users mailing list archive at Nabble.com. >> >> >> >> -- >> Claus Ibsen >> - >> Red Hat, Inc. >> FuseSource is now part of Red Hat >> Email: [hidden email] >> Web: http://fusesource.com >> Twitter: davsclaus >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> >> >> >> >> >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726190.html >> >> To unsubscribe from NullpointerException >> in ClassPathXmlApplicationContext, click here. >> NAML >> >> >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726202.html >> Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen - Red Hat, Inc. FuseSource is now part of Red Hat Email: [hidden email] Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen If you reply to this email, your message will be added to the discussion below: http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726211.html To unsubscribe from NullpointerException in ClassPathXmlApplicationContext, click here. NAML -- View this message in context: http://camel.465427.n5.nabble.com/NullpointerException-in-ClassPathXmlApplicationContext-tp5726189p5726293.html Sent from the Camel - Users mailing list archive at Nabble.com.
Processing VERY large result sets
Hi there,my task is to execute a JDBC query against a Hive database and
produce rows in csv files. The clue is, that depending on the query
criteria, the number of range from some dozens to some millions. My first
solution was something like this:
from ("...").to ("sql:...") // produces a List>.split(body()).process(myProcessor) // produces a single row for the
csv file.to("file:destination?fileExists=Append");
This was awful slow because the file producer opens the file, appends one
single row, and closes it again.I found some posts how to use an Aggregator
before sending the content to the file producer. This really was the desired
solution, and the performance was satisfying. In this solution, the
aggregator holds the total content of the csv file to be produced.
Unfortunately, the files can be so large that I get stuck in "java gc
overhead limit exceeded" exceptions. No matter how high I set the heap
space, I have no chance to avoid this.Now I'm looking for a way how to get
out of this, and I don't know how. My ideas are:
Use a splitter that produces a sublist - I don't know how I could do it
Use an aggregator that does not produce the total content for the files to
be created, but only for example 1000 lines and then collects the next block
- I don't know it here either
Or maybe someone has a better idea...Kind regards,Christian
--
View this message in context:
http://camel.465427.n5.nabble.com/Processing-VERY-large-result-sets-tp5790018.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Processing VERY large result sets
Meanwhile, I found a way how to solve it. It may not be the best one, but it works. I programmed a custom processor. In the process method of this processor, I navigated from the exchange to the camel context, took its registry, and from that the sql datasource. The sql datasource has a java.sql.Connection. With that connection, I issued the select statement and iterated through the result set. Like in Camel, I mapped each row of the result set to a Map. This map I added to a List. When the size of the list exceeds a configurable limit, I put the List into the body of the current exchange and sent it to an appropriate endpoint, where it is further processed, e.g. produce a Camel file. Clear the list and continue to iterate through the resultset until it is finished. Don't forget to send the list once more to the endpoint after the iteration loop if the list is not empty. Maybe someone who has more knowledge about programming endpoints can take these hints to implement this solution as an endpoint instead of a custom processor... -- View this message in context: http://camel.465427.n5.nabble.com/Processing-VERY-large-result-sets-tp5790018p5790107.html Sent from the Camel - Users mailing list archive at Nabble.com.
