[ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493075#comment-15493075 ]
Srinivas Rishindra Pothireddi commented on SPARK-17538: ------------------------------------------------------- Hi [~srowen], I updated the description of the jira as per your suggestion. Can you please let me know if you need anymore information > sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0 > ----------------------------------------------------------------------------- > > Key: SPARK-17538 > URL: https://issues.apache.org/jira/browse/SPARK-17538 > Project: Spark > Issue Type: Bug > Affects Versions: 2.0.0 > Environment: os - linux > cluster -> yarn and local > Reporter: Srinivas Rishindra Pothireddi > > I have a production job in spark 1.6.2 that registers several dataframes as > tables. > After testing the job in spark 2.0.0, I found that one of the dataframes is > not getting registered as a table. > Line 353 of my code --> self.sqlContext.registerDataFrameAsTable(anonymousDF, > "anonymousTable") > line 354 of my code --> df = self.sqlContext.sql("select AnonymousFiled1, > AnonymousUDF( AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > my stacktrace > File "anonymousFile.py", line 354, in anonymousMethod > df = self.sqlContext.sql("select AnonymousFiled1, AnonymousUDF( > AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/context.py", > line 350, in sql > return self.sparkSession.sql(sqlQuery) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/session.py", > line 541, in sql > return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", > line 933, in __call__ > answer, self.gateway_client, self.target_id, self.name) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/utils.py", > line 69, in deco > raise AnalysisException(s.split(': ', 1)[1], stackTrace) > AnalysisException: u'Table or view not found: anonymousTable; line 1 pos 61' > The same code is working perfectly fine in spark-1.6.2 > -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org