[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183219#comment-17183219 ] Apache Spark commented on SPARK-32550: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/29529 > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan >Priority: Major > Fix For: 3.1.0 > > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183216#comment-17183216 ] Apache Spark commented on SPARK-32550: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/29529 > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan >Priority: Major > Fix For: 3.1.0 > > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171870#comment-17171870 ] Takeshi Yamamuro commented on SPARK-32550: -- Looks nice. Thanks for the info. > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Priority: Major > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171868#comment-17171868 ] Muhammad Samir Khan commented on SPARK-32550: - [~maropu] added some benchmark results in [https://github.com/apache/spark/pull/29366]. For Avro, with a couple of benchmarks I am proposing to add in SPARK-32531 (PR: [https://github.com/apache/spark/pull/29352]) I get the following on master branch: *Before:* Nested Struct 75s average Array of Structs 34s average *After:* Nested Struct 49s average Array of Structs 19s average > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Priority: Major > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171832#comment-17171832 ] Apache Spark commented on SPARK-32550: -- User 'msamirkhan' has created a pull request for this issue: https://github.com/apache/spark/pull/29366 > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Priority: Major > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171831#comment-17171831 ] Apache Spark commented on SPARK-32550: -- User 'msamirkhan' has created a pull request for this issue: https://github.com/apache/spark/pull/29366 > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Priority: Major > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32550) Make SpecificInternalRow constructors faster by using while loops instead of maps
[ https://issues.apache.org/jira/browse/SPARK-32550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171828#comment-17171828 ] Takeshi Yamamuro commented on SPARK-32550: -- The proposed improvement can affect end-2-end performance? If so, could you show us a query here? > Make SpecificInternalRow constructors faster by using while loops instead of > maps > - > > Key: SPARK-32550 > URL: https://issues.apache.org/jira/browse/SPARK-32550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Muhammad Samir Khan >Priority: Major > > Two constructors in SpecificInternalRow can be made faster by using while > loops instead of maps. This was originally noticed while working on > SPARK-32532 and SPARK-32533 and will have impacts on performance of reading > ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 > for nested and array'd structs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
