[jira] (SPARK-19401) Transformation of one RDD inside of other transformations

[Sean Owen (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Sean Owen resolved as Invalid 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
Please read http://spark.apache.org/contributing.html 
Questions go to u...@spark.apache.org 
 
 
 
 
 
 
 
 
 
 Spark /  SPARK-19401 
 
 
 
  Transformation of one RDD inside of other transformations  
 
 
 
 
 
 
 
 
 

Change By:
 
 Sean Owen 
 
 
 

Resolution:
 
 Invalid 
 
 
 

Status:
 
 Open Resolved 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   

[jira] (SPARK-19401) Transformation of one RDD inside of other transformations

[Chetan Panchal (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Chetan Panchal created an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Spark /  SPARK-19401 
 
 
 
  Transformation of one RDD inside of other transformations  
 
 
 
 
 
 
 
 
 

Issue Type:
 
  Question 
 
 
 

Assignee:
 

 Unassigned 
 
 
 

Created:
 

 30/Jan/17 10:20 
 
 
 

Priority:
 
  Major 
 
 
 

Reporter:
 
 Chetan Panchal 
 
 
 
 
 
 
 
 
 
 
Hi, 
I am new in spark. I want to update one RDD Map key by matching in another RDD. When i am using .take(100) or collect(), this is working fine but it is taking so much time. But when i am using .map(), it is giving me error. 
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 381.0 failed 4 times, most recent failure: Lost task 0.3 in stage 381.0 (TID 707, 192.168.200.82): org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see 

SPARK-5063
. 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment