Re: Strange issue with JDBC add Postgres

[patelp7]

Hi Tim,
Can you possible provide me a snippet of what your route actually looks like
with the workaround? I ran into same problem where my fetchsize is not
working and I want to have a work around while still using camel.

Thanks



--
View this message in context: 
http://camel.465427.n5.nabble.com/Strange-issue-with-JDBC-add-Postgres-tp5758548p5782983.html
Sent from the Camel - Users mailing list archive at Nabble.com.

camel-jdbc fetchSize out of memory error

[patelp7]

I am trying to ingest data from postgres to another DB and I am using
camel-jdbc component to do it. I have a large table so I want to read few
rows at a time instead of the whole table altogether. so my route looks like
below (only for testing purpose)
from(fromUri).setBody("select * from 
 limit
10").to("jdbc://myDataSource?resetAutoCommit=false=2").split(body()).streaming().process(test)

As shown above, I am only getting 10 rows at a time for testing purpose and
I have set fetchSize to 2 to only receive 2 rows at a time. However, I am
still receiving all 10 rows altogether. When I remove the "limit 10" from
the query I get Out of Memory error just before the split command which
tells me that its trying to load the entire result set in memory.

What am I missing here or what am I doing wrong? 

Thanks for help.



--
View this message in context: 
http://camel.465427.n5.nabble.com/camel-jdbc-fetchSize-out-of-memory-error-tp5782974.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

I figured the problem is converting from one charset to another. I realized
that the file I am reading is in charset "US-ASCII" and then I am trying to
convert it into "UTF-8" at which point the unicode character is lost. Any
suggestions on how to fix it?



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781188.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

I tried and it just doesnt work for me. As soon as I modify the body, either
try to replace all /r/n with /n or convert the body to string or use the
transform tag, the character is corrupted. I have attached the file if
that's helpful.

  sample_data_export.csv
  



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781109.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

Not true exactly. It has to do with some camel component. I tried few things.

1) Route: file(from).split(body()).streaming().to(file) /This scenario
preserves the character./
2) Route:
file(from).convertBodyTo(String.class,"UTF-8").split(body()).streaming().to(file)
/This scenario converts the character to ?/
3) Route: file(from).split(body().tokenize("\r\n").streaming().to(file)
/This route won't even run. Gives me error
org.apache.camel.RuntimeCamelException: Scanner aborted because of an
IOException!/
4) Route: file(from).split(body().tokenize("\n").streaming().to(file) /This
scenario preserves the character/

The problem is my file has EOL "\r\n" so I have to use #3 for routing and
because its giving me exception I am using solution#2 and #3 together which
changes my character.

Is there a work around?



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781038.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

I changed my route to
from(file).to("file:/?fileName=out.csv=Append")

It preserved the character.



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781036.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

I guess my assumption was wrong. I removed tokenizing and its still not
working :(



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781032.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Unicode characters not recognizing when reading file

[patelp7]

Hi,
I tried both solutions. My JVM default encoding is already set to UTF-8. I
confirmed by printing System.getProperty("file.encoding").
I also changed my route to: 
from(file).marshal().string("UTF-8").split(body().tokenize("\r\n|\n|\r")).streaming()


It's still converting the character to "?". I feel its reading the right
character but its getting corrupted when tokenizing/streaming?



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028p5781030.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Unicode characters not recognizing when reading file

[patelp7]

Hi,
I have defined my file URI as file:///?charset=UTF-8 and my route looks
like below:

from(file)
 .convertBodyTo(String.class,"UTF-8")
 .split(body().tokenize("\r\n|\n|\r")).streaming()
.process(test)
   
.end()

If my file has unicode character such as § , it converts into ? I am trying
to preserve § but am not able to. Also, I have realized that without
converting my body to string, the route fails to process when the EOL
character is "/r/n" but passes when the EOF character is "/n"

What am I missing? 



--
View this message in context: 
http://camel.465427.n5.nabble.com/Unicode-characters-not-recognizing-when-reading-file-tp5781028.html
Sent from the Camel - Users mailing list archive at Nabble.com.