[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306244#comment-16306244 ] Kazuaki Ishizaki commented on SPARK-22500: -- Thank you for your report. I think that this is the same as SPARK-22869. While this code throws this exception, the code finally shows the result. In near future, this program will not throw an exception. > 64KB JVM bytecode limit problem with cast > - > > Key: SPARK-22500 > URL: https://issues.apache.org/jira/browse/SPARK-22500 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kazuaki Ishizaki >Assignee: Kazuaki Ishizaki > Fix For: 2.2.1, 2.3.0 > > > {{Cast}} can throw an exception due to the 64KB JVM bytecode limit when they > use with a lot of structure fields -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306197#comment-16306197 ] Łukasz Żukowski commented on SPARK-22500: - Hmm Below still make 64k exception. Maybe it can be done otherwise but I don't know how to apply cast to root. val spark = SparkSession.builder().appName("name").master("local[*]") .getOrCreate() import org.apache.spark.sql.functions._ val names = (1 to 100).map(i => "f" + i) val largeStruct = StructType(names.map(f => StructField(f, IntegerType, true))) implicit def rowEnc = RowEncoder(largeStruct) val values = Seq(Row.fromSeq(1 to 100)) val rdd = spark.sparkContext.parallelize(values) val df = spark.createDataFrame(rdd, largeStruct) val casted = df.select(struct("*").cast(largeStruct).as("_toUnpack")).select("_toUnpack.*") casted.show > 64KB JVM bytecode limit problem with cast > - > > Key: SPARK-22500 > URL: https://issues.apache.org/jira/browse/SPARK-22500 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kazuaki Ishizaki >Assignee: Kazuaki Ishizaki > Fix For: 2.2.1, 2.3.0 > > > {{Cast}} can throw an exception due to the 64KB JVM bytecode limit when they > use with a lot of structure fields -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278212#comment-16278212 ] Apache Spark commented on SPARK-22500: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/19891 > 64KB JVM bytecode limit problem with cast > - > > Key: SPARK-22500 > URL: https://issues.apache.org/jira/browse/SPARK-22500 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kazuaki Ishizaki >Assignee: Kazuaki Ishizaki > Fix For: 2.2.1, 2.3.0 > > > {{Cast}} can throw an exception due to the 64KB JVM bytecode limit when they > use with a lot of structure fields -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264726#comment-16264726 ] Apache Spark commented on SPARK-22500: -- User 'kiszk' has created a pull request for this issue: https://github.com/apache/spark/pull/19806 > 64KB JVM bytecode limit problem with cast > - > > Key: SPARK-22500 > URL: https://issues.apache.org/jira/browse/SPARK-22500 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kazuaki Ishizaki >Assignee: Kazuaki Ishizaki > Fix For: 2.2.1, 2.3.0 > > > {{Cast}} can throw an exception due to the 64KB JVM bytecode limit when they > use with a lot of structure fields -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248961#comment-16248961 ] Apache Spark commented on SPARK-22500: -- User 'kiszk' has created a pull request for this issue: https://github.com/apache/spark/pull/19730 > 64KB JVM bytecode limit problem with cast > - > > Key: SPARK-22500 > URL: https://issues.apache.org/jira/browse/SPARK-22500 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Kazuaki Ishizaki > > {{Cast}} can throw an exception due to the 64KB JVM bytecode limit when they > use with a lot of structure fields -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org