Re: About HostName display in SparkUI
In the conf/slaves file, are you having the ip addresses? or the hostnames? Thanks Best Regards On Sat, Jun 13, 2015 at 9:51 PM, Sea 261810...@qq.com wrote: In spark 1.4.0, I find that the Address is ip (it was hostname in v1.3.0), why? who did it?
Using queueStream
JavaDStreamString inputStream = ssc.queueStream(rddQueue); Can this rddQueue be of dynamic type in nature .If yes then how to make it run untill rddQueue is not finished . Any other way to get rddQueue from a dynamically updatable Normal Queue . -- Thanks Regards, SERC-IISC Anshu Shukla
Random Forest driver memory
hello, i have noticed that the random forest implementation crashes when to many trees/ to big maxDepth is used. im guessing that this is something to do with the amount of nodes that need to be kept in driver's memory during the run. but when i examined the nodes structure is seems rather small does anyone now where does the memory issue come from? thanks, Isca
Re?? About HostName display in SparkUI
In the conf/slaves file, I have hostnames. Before 1.4.0, it is okay. I view the code in class org.apache.spark.util.Utils, I alter function localHostName and localHostNameForURI, and it turns back to hostnames again. I just don't know why to change these basic functions. Hostname is nice. -- -- ??: Akhil Das;ak...@sigmoidanalytics.com; : 2015??6??15??(??) 5:36 ??: Sea261810...@qq.com; : devdev@spark.apache.org; : Re: About HostName display in SparkUI In the conf/slaves file, are you having the ip addresses? or the hostnames? ThanksBest Regards On Sat, Jun 13, 2015 at 9:51 PM, Sea 261810...@qq.com wrote: In spark 1.4.0, I find that the Address is ip (it was hostname in v1.3.0), why? who did it? 7FAB9BA9@AFBE9573.34FC7E55 Description: Binary data
Problem: Custom Receiver for getting events from a Dynamic Queue
I have written a custom receiver for converting the tuples in the Dynamic Queue/EventGen to the Dstream.But i dont know why It is only processing data for some time (3-4 sec.) only and then shows Queue as Empty .ANy suggestions please .. --code // public class JavaCustomReceiver extends ReceiverString implements ISyntheticEventGen { EventGen eventGen; BlockingQueueListString eventQueue; String csvFileName; String outSpoutCSVLogFileName; double scalingFactor; public JavaCustomReceiver(String csvFileName, String outSpoutCSVLogFileName, double scalingFactor) { super(StorageLevel.MEMORY_AND_DISK()); this.csvFileName = csvFileName; this.outSpoutCSVLogFileName = outSpoutCSVLogFileName; this.scalingFactor = scalingFactor; this.eventGen = new EventGen(this,this.scalingFactor); this.eventGen.launch(this.csvFileName, this.outSpoutCSVLogFileName); //Launch threads this.eventQueue = new LinkedBlockingQueueListString(); System.out.println(for watching queue); } public void onStart() { // Start the thread that receives data over a connection new Thread() { @Override public void run() { receive(); } }.start(); } public void onStop() { // There is nothing much to do as the thread calling receive() // is designed to stop by itself isStopped() returns false } /** Create a socket connection and receive data until receiver is stopped */ private void receive() { try { // connect to the server //socket = new Socket(host, port); //BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); // Until stopped or connection broken continue reading while (!isStopped() ) { ListString entry = this.eventQueue.take(); String str=; for(String s:entry) str+=s+,; System.out.println(Received data ' + str + '); store(str); } // Restart in an attempt to connect again when server is active again restart(Trying to connect again); } catch(Throwable t) { // restart if there is any other error restart(Error receiving data, t); } } @Override public StorageLevel storageLevel() { return StorageLevel.MEMORY_AND_DISK(); } @Override public void receive(ListString event) { // TODO Auto-generated method stub //System.out.println(Called IN SPOUT### ); try { this.eventQueue.put(event); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } -- Thanks Regards, Anshu Shukla
[SparkStreaming] NPE in DStreamCheckPointData.scala:125
I use the attached program to test checkpoint. It's quite simple. When I run the program second time, it will load checkpoint data, that's expected, however I see NPE in driver log. Do you have any idea about the issue? I'm on Spark 1.4.0, thank you very much! == logs == 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream: Restoring checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435313 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435314 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435315 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435316 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435317 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435318 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435319 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO MappedDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO MappedDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO ForEachDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO DStreamGraph: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] ERROR StreamingContext: Error starting the context, marking it as stopped java.io.IOException: java.lang.NullPointerException at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1242) at org.apache.spark.streaming.dstream.DStreamCheckpointData.writeObject(DSt reamCheckpointData.scala:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:44 0) at org.apache.spark.streaming.dstream.DStream$$anonfun$writeObject$1.apply$ mcV$sp(DStream.scala:498) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239) at org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:493 ) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:44 0) at org.apache.spark.streaming.DStreamGraph$$anonfun$writeObject$1.apply$mcV $sp(DStreamGraph.scala:181) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239) at org.apache.spark.streaming.DStreamGraph.writeObject(DStreamGraph.scala:1 76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at