Antony Mayi created SPARK-11498:
-----------------------------------
Summary: TreeNodeException under very special condition
Key: SPARK-11498
URL: https://issues.apache.org/jira/browse/SPARK-11498
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.1
Environment: spark 1.5.1 (official binary distribution) running on
hadoop yarn 2.6 with parquet 1.5.0 (both from cdh5.4.8)
Reporter: Antony Mayi
{code}
from pyspark import SparkContext
from pyspark.sql import SQLContext, Row
sc = SparkContext()
sqlc = SQLContext(sc)
R = Row('id', 'foo')
r = sqlc.createDataFrame(sc.parallelize([R('abc', 'foo')]))
q = sqlc.createDataFrame(sc.parallelize([R('aaaaaaaaaaaaaaaaaaaaaaaa', 'bar')]))
q.write.parquet('/tmp/1.parq')
q = sqlc.read.parquet('/tmp/1.parq')
j = r.join(q, r.id == q.id)
print j.count()
{code}
{noformat}
[user@sandbox test]$ spark-submit --executor-memory=32g /tmp/bug.py
[user@sandbox test]$ hadoop fs -rmr /tmp/1.parq
{noformat}
{noformat}
15/11/04 04:28:38 INFO codegen.GenerateUnsafeProjection: Code generated in
119.90324 ms
Traceback (most recent call last):
File "/tmp/bug.py", line 13, in <module>
print j.count()
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line
268, in count
File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 36,
in deco
File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line
300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o148.count.
: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
TungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)],
output=[count#10L])
TungstenExchange SinglePartition
TungstenAggregate(key=[],
functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#13L])
TungstenProject
BroadcastHashJoin [id#0], [id#8], BuildRight
TungstenProject [id#0]
Scan PhysicalRDD[id#0,foo#1]
ConvertToUnsafe
Scan ParquetRelation[hdfs:///tmp/1.parq][id#8]
at
org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
at
org.apache.spark.sql.execution.aggregate.TungstenAggregate.doExecute(TungstenAggregate.scala:69)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)
...
{noformat}
Note this happens only under following condition:
# executor memory >= 32GB (doesn't fail with up to 31 GB)
# the ID in the q dataframe has exactly 24 chars (doesn't fail with less or
more then 24 chars)
# q is read from parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
