Hey Akhil,
Thanks for your response! No, I am not expecting to receive the values
themselves. I am just trying to receive the RDD object on my second Spark
application. However, I get a NPE when I try to use the object within my
second program. Would you know how I can properly send the RDD object to my
second program?
Thanks,
Raghav
On Mon, Apr 6, 2015 at 3:08 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Are you expecting to receive 1 to 100 values in your second program?
>
> RDD is just an abstraction, you would need to do like:
>
> num.foreach(x => send(x))
>
>
> Thanks
> Best Regards
>
> On Mon, Apr 6, 2015 at 1:56 AM, raggy <raghav0110...@gmail.com> wrote:
>
>> For a class project, I am trying to utilize 2 spark Applications
>> communicate
>> with each other by passing an RDD object that was created from one
>> application to another Spark application. The first application is
>> developed
>> in Scala and creates an RDD and sends it to the 2nd application over the
>> network as follows:
>>
>> val logFile = "../../spark-1.3.0/README.md" // Should be some file on
>> your system
>> val conf = new SparkConf();
>> conf.setAppName("Simple Application").setMaster("local[2]")
>> val sc = new SparkContext(conf)
>> val nums = sc.parallelize(1 to 100, 2).toJavaRDD();
>> val s = new Socket("127.0.0.1", 8000);
>> val objectOutput = new ObjectOutputStream(s.getOutputStream());
>> objectOutput.writeObject(nums);
>> s.close();
>> The second Spark application is a Java application, which tries to receive
>> the RDD object and then perform some operations on it. At the moment, I am
>> trying to see if I have properly obtained the object.
>>
>> ServerSocket listener = null;
>> Socket client;
>>
>> try{
>> listener = new ServerSocket(8000);
>> }catch(Exception e){
>> e.printStackTrace();
>> }
>> System.out.println("Listening");
>> try{
>> client = listener.accept();
>> ObjectInputStream objectInput = new
>> ObjectInputStream(client.getInputStream());
>> Object object =(JavaRDD) objectInput.readObject();
>> JavaRDD tmp = (JavaRDD) object;
>>
>> if(tmp != null){
>> System.out.println(tmp.getStorageLevel().toString());
>> List<Partition> p = tmp.partitions();
>> }
>> else{
>> System.out.println("variable is null");
>> }
>>
>> }catch(Exception e){
>> e.printStackTrace();
>> }
>> The output I get is:
>>
>> StorageLevel(false, false, false, false, 1)
>> java.lang.NullPointerException
>> at
>>
>> org.apache.spark.rdd.ParallelCollectionRDD$.slice(ParallelCollectionRDD.scala:154)
>> at
>>
>> org.apache.spark.rdd.ParallelCollectionRDD.getPartitions(ParallelCollectionRDD.scala:97)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
>> at
>>
>> org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:56)
>> at org.apache.spark.api.java.JavaRDD.partitions(JavaRDD.scala:32)
>> at SimpleApp.main(SimpleApp.java:35)
>> So, System.out.println(tmp.getStorageLevel().toString()); prints out
>> properly. But, List<Partition> p = tmp.partitions(); throws the
>> NullPointerException. I can't seem to figure out why this is happening.
>>
>> In a nutshell, I am basically trying to create an RDD object in one Spark
>> application and then send the object to another application. After
>> receiving
>> the object I try to make sure I received it properly by accessing its
>> methods. Invoking the partitions() method in the original Spark
>> application
>> does not throw any errors either. I would greatly appreciate any
>> suggestion
>> on how I can solve my problem, or an alternative solution for what I am
>> trying to accomplish.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Sending-RDD-object-over-the-network-tp22382.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>
