Re: save as text file throwing null pointer error.

2015-04-14 Thread Akhil Das 
Where exactly is it throwing null pointer exception? Are you starting your
program from another program or something? looks like you are invoking
ProcessingBuilder etc.

Thanks
Best Regards

On Thu, Apr 9, 2015 at 6:46 PM, Somnath Pandeya somnath_pand...@infosys.com
 wrote:



 JavaRDDString lineswithoutStopWords = nonEmptylines

.map(*new* FunctionString, String() {



   /**

   *

*/

   *private* *static* *final* *long*
 *serialVersionUID* = 1L;



   @Override

   *public* String call(String line)
 *throws* Exception {

  // *TODO* Auto-generated method
 stub

  *return* *removeStopWords*(line,
 stopwords);

   }



});



   lineswithoutStopWords.saveAsTextFile(output/testop.txt);







 Exception in task 0.0 in stage 1.0 (TID 1)

 *java.lang.NullPointerException*

at java.lang.ProcessBuilder.start(*ProcessBuilder.java:1012*)

at org.apache.hadoop.util.Shell.runCommand(*Shell.java:404*)

at org.apache.hadoop.util.Shell.run(*Shell.java:379*)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
 *Shell.java:589*)

at org.apache.hadoop.util.Shell.execCommand(*Shell.java:678*)

at org.apache.hadoop.util.Shell.execCommand(*Shell.java:661*)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
 *RawLocalFileSystem.java:639*)

at org.apache.hadoop.fs.FilterFileSystem.setPermission(
 *FilterFileSystem.java:468*)

at org.apache.hadoop.fs.ChecksumFileSystem.create(
 *ChecksumFileSystem.java:456*)

at org.apache.hadoop.fs.ChecksumFileSystem.create(
 *ChecksumFileSystem.java:424*)

at org.apache.hadoop.fs.FileSystem.create(*FileSystem.java:905*)

at org.apache.hadoop.fs.FileSystem.create(*FileSystem.java:798*)

at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(
 *TextOutputFormat.java:123*)

at
 org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)

at
 org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)

at
 org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)

at
 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)

at org.apache.spark.scheduler.Task.run(Task.scala:64)

at
 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)

at java.util.concurrent.ThreadPoolExecutor.runWorker(
 *ThreadPoolExecutor.java:1142*)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 *ThreadPoolExecutor.java:617*)

at java.lang.Thread.run(*Thread.java:745*)

 15/04/09 18:44:36 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1,
 localhost): *java.lang.NullPointerException*

at java.lang.ProcessBuilder.start(*ProcessBuilder.java:1012*)

at org.apache.hadoop.util.Shell.runCommand(*Shell.java:404*)

at org.apache.hadoop.util.Shell.run(*Shell.java:379*)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
 *Shell.java:589*)

at org.apache.hadoop.util.Shell.execCommand(*Shell.java:678*)

at org.apache.hadoop.util.Shell.execCommand(*Shell.java:661*)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
 *RawLocalFileSystem.java:639*)

at org.apache.hadoop.fs.FilterFileSystem.setPermission(
 *FilterFileSystem.java:468*)

at org.apache.hadoop.fs.ChecksumFileSystem.create(
 *ChecksumFileSystem.java:456*)

at org.apache.hadoop.fs.ChecksumFileSystem.create(
 *ChecksumFileSystem.java:424*)

at org.apache.hadoop.fs.FileSystem.create(*FileSystem.java:905*)

at org.apache.hadoop.fs.FileSystem.create(*FileSystem.java:798*)

at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(
 *TextOutputFormat.java:123*)

at
 org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)

at
 org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)

at
 org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)

at
 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)

at org.apache.spark.scheduler.Task.run(Task.scala:64)

at
 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)

at java.util.concurrent.ThreadPoolExecutor.runWorker(
 *ThreadPoolExecutor.java:1142*)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 *ThreadPoolExecutor.java:617*)

at java.lang.Thread.run(*Thread.java:745*)



 15/04/09 18:44:36 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1
 times; aborting job

RE: save as text file throwing null pointer error.

2015-04-14 Thread Somnath Pandeya 
Hi Akhil,

I am running my program standalone, I am getting null pointer exception when I 
running spark program locally and when I  am trying to save my RDD as a text 
file.

From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: Tuesday, April 14, 2015 12:41 PM
To: Somnath Pandeya
Cc: user@spark.apache.org
Subject: Re: save as text file throwing null pointer error.

Where exactly is it throwing null pointer exception? Are you starting your 
program from another program or something? looks like you are invoking 
ProcessingBuilder etc.

Thanks
Best Regards

On Thu, Apr 9, 2015 at 6:46 PM, Somnath Pandeya 
somnath_pand...@infosys.commailto:somnath_pand...@infosys.com wrote:

JavaRDDString lineswithoutStopWords = nonEmptylines
   .map(new FunctionString, String() {

  /**
  *
   */
  private static final long serialVersionUID = 
1L;

  @Override
  public String call(String line) throws 
Exception {
 // TODO Auto-generated method stub
 return removeStopWords(line, 
stopwords);
  }

   });

  lineswithoutStopWords.saveAsTextFile(output/testop.txt);



Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.NullPointerException
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
   at org.apache.hadoop.util.Shell.run(Shell.java:379)
   at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
   at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
   at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
   at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
   at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
   at org.apache.spark.scheduler.Task.run(Task.scala:64)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
15/04/09 18:44:36 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, 
localhost): java.lang.NullPointerException
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
   at org.apache.hadoop.util.Shell.run(Shell.java:379)
   at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
   at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
   at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
   at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
   at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
   at org.apache.spark.scheduler.Task.run(Task.scala:64)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run

save as text file throwing null pointer error.

2015-04-09 Thread Somnath Pandeya 

JavaRDDString lineswithoutStopWords = nonEmptylines
   .map(new FunctionString, String() {

  /**
  *
   */
  private static final long serialVersionUID = 
1L;

  @Override
  public String call(String line) throws 
Exception {
 // TODO Auto-generated method stub
 return removeStopWords(line, 
stopwords);
  }

   });

  lineswithoutStopWords.saveAsTextFile(output/testop.txt);



Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.NullPointerException
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
   at org.apache.hadoop.util.Shell.run(Shell.java:379)
   at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
   at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
   at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
   at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
   at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
   at org.apache.spark.scheduler.Task.run(Task.scala:64)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
15/04/09 18:44:36 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, 
localhost): java.lang.NullPointerException
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
   at org.apache.hadoop.util.Shell.run(Shell.java:379)
   at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
   at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
   at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
   at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
   at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)
   at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
   at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)
   at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
   at org.apache.spark.scheduler.Task.run(Task.scala:64)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)

15/04/09 18:44:36 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; 
aborting job
15/04/09 18:44:36 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have 
all completed, from pool
15/04/09 18:44:36 INFO TaskSchedulerImpl: Cancelling stage 1
15/04/09 18:44:36 INFO DAGScheduler: Job 1 failed: saveAsTextFile at 
TextPreProcessing.java:49, took 0.172959 s
Exception in thread main org.apache.spark.SparkException: Job aborted due to 
stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost 
task 0.0 in stage 1.0 (TID 1, localhost): java.lang.NullPointerException
   at