Hello,
I have to consume a csv (big file 14MB) and start to read it from a given
line number. The line number is stored in a file stored in a ftp, at the end
of the process I have to update the line number with the last line I read.
So my approach was unmarshalling the file with .unmarshal(csv) and create a
processor that:
1- Starts a route to download the file locally
2- Read the file and get the linenumber
3- loop on the object (List<List<String>>) exchange.getIn().getBody();
and append the data to a string buffer to create the output (updating the
linecount)
4- update the file line number in the local file
5- start a route to upload the line number file.
Translated in camel routes:
from("direct:" + ROUTE_INCREMENTAL)
.id("direct_"+ROUTE_INCREMENTAL)
.unmarshal(csv)
.process(new Processor() {
@Override
public void process(Exchange exchange)
throws Exception {
getContext().startRoute(ROUTE_GET_CUTTENT_POSITION);
File file = new
File(evaluatePathWithProperties(
localFilePath,
localFileName));
if (file.exists()) {
DataInputStream in =
new DataInputStream(
new
FileInputStream(file));
BufferedReader br = new
BufferedReader(
new
InputStreamReader(in));
String firstLine =
br.readLine();
int currentValue = 1;
if (firstLine != null) {
try {
currentValue = Integer.parseInt(firstLine);
} catch
(Exception e) {
System.out
.println("Error reading position file check if firstline is just
a number");
}
}
in.close();
List<List<String>>
data = (List<List<String>>) exchange
.getIn().getBody();
StringBuffer s = new
StringBuffer();
s.append("<Message><ImageMessage><assetIds>");
int counter = 0;
for (int i =
currentValue; i < data.size(); i++) {
currentValue++;
counter++;
s.append("\n\t<assetId>" + data.get(i).get(0)
+ "</assetId>");
if (counter ==
20) {
s.append("\n</assetIds></ImageMessage>");
s.append("\n<ImageMessage><assetIds>");
counter
= 0;
}
}
s.append("\n</assetIds></ImageMessage></Message>");
exchange.getIn().setBody(s.toString());
FileOutputStream fos =
new FileOutputStream(file);
DataOutputStream dos =
new DataOutputStream(fos);
Integer integer = new
Integer(currentValue);
dos.write(integer.toString().getBytes());
dos.close();
getContext().startRoute(ROUTE_SET_CURRENT_POSITION);
}
}
})
//
.to("file:/opt/servicemix/data/smx-data/outbox/adobes7");
.split(xpath(splitXml))
.to("jms:" + jmsincrementalQueue +
"?jmsMessageType=Text");
from(
ftpDataType + "://" + ftpDataUsername + "@" +
ftpDataUrl
+ ftpDataPath + "?password=" +
ftpDataPassword
+ "&fileName=" + ftpDataFilename
+
"&noop=true&idempotent=false").noAutoStartup()
.id(ROUTE_GET_CUTTENT_POSITION)
.to("file:" + localFilePath + "?fileName=" +
localFileName)
.process(new Processor() {
@Override
public void process(Exchange arg0)
throws Exception {
getContext().getShutdownStrategy().setTimeout(ROUTE_STOP_TIMEOUT);
getContext().stopRoute(ROUTE_GET_CUTTENT_POSITION);
}
});
from("file:" + localFilePath + "?fileName=" + localFileName)
.noAutoStartup()
.id(ROUTE_SET_CURRENT_POSITION)
.to(ftpDataType + "://" + ftpDataUsername + "@"
+ ftpDataUrl
+ ftpDataPath + "?password=" +
ftpDataPassword
+ "&fileName=" + ftpDataFilename
+ "&noop=true&idempotent=false")
.process(new Processor() {
@Override
public void process(Exchange arg0)
throws Exception {
getContext().getShutdownStrategy().setTimeout(ROUTE_STOP_TIMEOUT);
getContext().stopRoute(ROUTE_SET_CURRENT_POSITION);
}
});
but in this case i get an exception:
org.apache.camel.CamelExecutionException: Exception occurred during
execution on the exchange: Exchange[Message: [[Id, Path, Resolution,
Modifier, PostModifier, MaskPath, Expiration, ThumbType, ThumbRes, UserData,
Anchor, ImageSet, Map, Targets, LastModified, TimeStamp, Size, DigimarcInfo,
AssetType], [product-6-model-big,
_optimized_/9d2/9d217286-ded8-4148-b13e-a2b0907c4e75.tif, 72.0, , , , , , ,
, 304,783, , , , Wed Aug 29 12:34:53 IST 2012, 1346240093911, 609,1566, ,
IMAGE], [247593-011_psfront1,
_optimized_/059/059b0e91-7074-46c1-a7fb-2b5f99f7040d.tif, 72.0, , , , , , ,
, 69,117, , , , Thu Aug 30 12:05:09 IST 2012, 1346324709095, 138,235, ,
IMAGE], [247593-011_psback2,
_optimized_/8f3/8f3f7b61-c970-4864-80e9-a7547e1a7abc.tif, , , , , , , , ,
70,86, , , , Thu Aug 30 12:05:24 IST 2012, 1346324724979, 140,172, , IMAGE],
[247593-011_psdet, _optimized_/70f/70fc518f-87fb-4167-8d2e-fb71ca7f92ae.tif,
72.0, , , , , , , , 57,71, , , , Thu Aug 30 12:05:27 IST 2012,
1346324727056, 115,142, , IMAGE], [247593-011_mdfront3,
_optimized_/668/668994ff-0def-4ad6-b361-2658d486376b.tif, 72.0, , , , , , ,
, 57,69,... [Body clipped after 1000 chars, total length is 15819508]]
Caused by: java.lang.OutOfMemoryError: Java heap space
I set the memory to 1024MB, but I still get this error...
Any Idea on how to solve it? and is that the correct approach to do what I
want?
--
View this message in context:
http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-csv-from-a-given-line-tp5720112.html
Sent from the Camel - Users mailing list archive at Nabble.com.
