[jira] [Commented] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931116#comment-16931116 ] angerszhu commented on SPARK-29022: --- Since after spark-2.0, when we call method

[jira] [Updated] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29022: -- Attachment: image-2019-09-17-13-54-50-896.png > SparkSQLCLI can not use 'ADD JAR' 's jar as Serder

[jira] [Commented] (SPARK-28467) Tests failed if there are not enough executors up before running

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1693#comment-1693 ] Dongjoon Hyun commented on SPARK-28467: --- Since this is not `Closed` status, you can reopen this

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2019-09-16 Thread Rajiv Bandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931110#comment-16931110 ] Rajiv Bandi commented on SPARK-6305: Hi All,           I (at my work) am facing an issues with logs

[jira] [Commented] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931094#comment-16931094 ] Yuming Wang commented on SPARK-29022: - It works before Spark 2.0. > SparkSQLCLI can not use 'ADD

[jira] [Commented] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931093#comment-16931093 ] Sandeep Katta commented on SPARK-29110: --- [~DylanGuedes] I see in the sql file some of the commands

[jira] [Updated] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29022: Affects Version/s: 2.0.2 2.1.3 2.2.3

[jira] [Updated] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29022: Parent: (was: SPARK-23710) Issue Type: Bug (was: Sub-task) > SparkSQLCLI can not use

[jira] [Issue Comment Deleted] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-29110: -- Comment: was deleted (was: [~DylanGuedes] thank you for this, soon I will raise the PR for

[jira] [Commented] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931082#comment-16931082 ] Sandeep Katta commented on SPARK-29110: --- [~DylanGuedes] thank you for this, soon I will raise the

[jira] [Commented] (SPARK-29109) Add window.sql - Part 3

2019-09-16 Thread Udbhav Agrawal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931073#comment-16931073 ] Udbhav Agrawal commented on SPARK-29109: i will work on this > Add window.sql - Part 3 >

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset size is big

2019-09-16 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-29114: Priority: Major (was: Blocker) > Dataset.coalesce(10) throw ChunkFetchFailureException when

[jira] [Commented] (SPARK-29108) Add window.sql - Part 2

2019-09-16 Thread pavithra ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931071#comment-16931071 ] pavithra ramachandran commented on SPARK-29108: --- i shall work on this > Add window.sql -

[jira] [Created] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset size is big

2019-09-16 Thread ZhanxiongWang (Jira)
ZhanxiongWang created SPARK-29114: - Summary: Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset size is big Key: SPARK-29114 URL: https://issues.apache.org/jira/browse/SPARK-29114

[jira] [Updated] (SPARK-29104) Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29104: -- Description: -

[jira] [Commented] (SPARK-29107) Add window.sql - Part 1

2019-09-16 Thread Sharanabasappa G Keriwaddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931064#comment-16931064 ] Sharanabasappa G Keriwaddi commented on SPARK-29107: I will take up this Jira. >

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931057#comment-16931057 ] Adrian Wang commented on SPARK-13446: - or you can just apply the patch from SPARK-27349 and

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931055#comment-16931055 ] Adrian Wang commented on SPARK-13446: - [~jpbordi][~headcra6] I am using mysql as hive metastore

[jira] [Issue Comment Deleted] (SPARK-29112) Expose more details when ApplicationMaster reporter faces a fatal exception

2019-09-16 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29112: --- Comment: was deleted (was: https://github.corp.ebay.com/carmel/spark/pull/551) > Expose more

[jira] [Commented] (SPARK-29112) Expose more details when ApplicationMaster reporter faces a fatal exception

2019-09-16 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931049#comment-16931049 ] Lantao Jin commented on SPARK-29112: https://github.corp.ebay.com/carmel/spark/pull/551 > Expose

[jira] [Updated] (SPARK-29113) Some annotation errors in ApplicationCache.scala

2019-09-16 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29113: Issue Type: Documentation (was: Bug) > Some annotation errors in ApplicationCache.scala >

[jira] [Created] (SPARK-29113) Some annotation errors in ApplicationCache.scala

2019-09-16 Thread feiwang (Jira)
feiwang created SPARK-29113: --- Summary: Some annotation errors in ApplicationCache.scala Key: SPARK-29113 URL: https://issues.apache.org/jira/browse/SPARK-29113 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29112) Expose more details when ApplicationMaster reporter faces a fatal exception

2019-09-16 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29112: -- Summary: Expose more details when ApplicationMaster reporter faces a fatal exception Key: SPARK-29112 URL: https://issues.apache.org/jira/browse/SPARK-29112 Project:

[jira] [Resolved] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29100. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25806

[jira] [Updated] (SPARK-28870) Snapshot old event log files to support compaction

2019-09-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on compacting old event log files into snapshot and

[jira] [Updated] (SPARK-28870) Snapshot old event log files to support compaction

2019-09-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-28870: - Description: This issue tracks the effort on compacting old event log files into snapshot and

[jira] [Created] (SPARK-29111) Support snapshot/restore of KVStore

2019-09-16 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29111: Summary: Support snapshot/restore of KVStore Key: SPARK-29111 URL: https://issues.apache.org/jira/browse/SPARK-29111 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-28467) Tests failed if there are not enough executors up before running

2019-09-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931024#comment-16931024 ] Jungtaek Lim commented on SPARK-28467: -- Looks like it is flaky in Amplab PR builder.

[jira] [Commented] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931017#comment-16931017 ] Xianyin Xin commented on SPARK-29049: - [~hyukjin.kwon], I updated the description. PR

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in

[jira] [Resolved] (SPARK-29008) Define an individual method for each common subexpression in HashAggregateExec

2019-09-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29008. -- Fix Version/s: 3.0.0 Assignee: Takeshi Yamamuro Resolution: Fixed

[jira] [Commented] (SPARK-29047) use spark-submit Not a file: hdfs://

2019-09-16 Thread bailin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931013#comment-16931013 ] bailin commented on SPARK-29047: i solved, it's code problem.   Generate and read parquet,use

[jira] [Updated] (SPARK-29108) Add window.sql - Part 2

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29108: - Description: In this ticket, we plan to add the regression test cases of 

[jira] [Updated] (SPARK-29109) Add window.sql - Part 3

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29109: - Description: In this ticket, we plan to add the regression test cases of 

[jira] [Created] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29110: Summary: Add window.sql - Part 4 Key: SPARK-29110 URL: https://issues.apache.org/jira/browse/SPARK-29110 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29110: - Description: In this ticket, we plan to add the regression test cases of 

[jira] [Created] (SPARK-29109) Add window.sql - Part 3

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29109: Summary: Add window.sql - Part 3 Key: SPARK-29109 URL: https://issues.apache.org/jira/browse/SPARK-29109 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29108) Add window.sql - Part 2

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29108: Summary: Add window.sql - Part 2 Key: SPARK-29108 URL: https://issues.apache.org/jira/browse/SPARK-29108 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29107) Add window.sql - Part 1

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29107: Summary: Add window.sql - Part 1 Key: SPARK-29107 URL: https://issues.apache.org/jira/browse/SPARK-29107 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29106) Add jenkins arm test for spark

2019-09-16 Thread huangtianhua (Jira)
huangtianhua created SPARK-29106: Summary: Add jenkins arm test for spark Key: SPARK-29106 URL: https://issues.apache.org/jira/browse/SPARK-29106 Project: Spark Issue Type: Test

[jira] [Resolved] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

2019-09-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25216. -- Resolution: Duplicate > Provide better error message when a column contains dot and needs

[jira] [Resolved] (SPARK-26781) Additional exchange gets added

2019-09-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26781. -- Resolution: Duplicate > Additional exchange gets added > --- > >

[jira] [Created] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-16 Thread Marcelo Vanzin (Jira)
Marcelo Vanzin created SPARK-29105: -- Summary: SHS may delete driver log file of in progress application Key: SPARK-29105 URL: https://issues.apache.org/jira/browse/SPARK-29105 Project: Spark

[jira] [Created] (SPARK-29104) Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished

2019-09-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29104: - Summary: Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished Key: SPARK-29104 URL: https://issues.apache.org/jira/browse/SPARK-29104

[jira] [Resolved] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26565. --- Resolution: Not A Problem > modify dev/create-release/release-build.sh to let jenkins build

[jira] [Created] (SPARK-29103) CheckAnalysis for data source V2 ALTER TABLE ignores case sensitivity

2019-09-16 Thread Jose Torres (Jira)
Jose Torres created SPARK-29103: --- Summary: CheckAnalysis for data source V2 ALTER TABLE ignores case sensitivity Key: SPARK-29103 URL: https://issues.apache.org/jira/browse/SPARK-29103 Project: Spark

[jira] [Commented] (SPARK-25153) Improve error messages for columns with dots/periods

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930896#comment-16930896 ] Jeff Evans commented on SPARK-25153: Opened a pull request for this (see link added by the bot).

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Priority: Major (was: Minor) > Codegen with switch in InSet expression causes compilation

[jira] [Issue Comment Deleted] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Comment: was deleted (was: Since this is a prevention of the potential bug situation, I lower

[jira] [Commented] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930871#comment-16930871 ] Dongjoon Hyun commented on SPARK-29100: --- Since this is a prevention of the potential bug

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Priority: Minor (was: Major) > Codegen with switch in InSet expression causes compilation

[jira] [Resolved] (SPARK-22381) Add StringParam that supports valid options

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22381. --- Resolution: Won't Fix > Add StringParam that supports valid options >

[jira] [Resolved] (SPARK-10408) Autoencoder

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10408. --- Resolution: Won't Fix > Autoencoder > --- > > Key: SPARK-10408 >

[jira] [Resolved] (SPARK-22111) OnlineLDAOptimizer should filter out empty documents beforehand

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22111. --- Resolution: Won't Fix > OnlineLDAOptimizer should filter out empty documents beforehand >

[jira] [Resolved] (SPARK-24806) Brush up generated code so that JDK Java compilers can handle it

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24806. --- Resolution: Won't Fix > Brush up generated code so that JDK Java compilers can handle it >

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930835#comment-16930835 ] Nicholas Chammas commented on SPARK-29102: -- cc [~cloud_fan] and [~hyukjin.kwon]: I noticed your

[jira] [Created] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-16 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-29102: Summary: Read gzipped file into multiple partitions without full gzip expansion on a single-node Key: SPARK-29102 URL: https://issues.apache.org/jira/browse/SPARK-29102

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930827#comment-16930827 ] Liang-Chi Hsieh commented on SPARK-28927: - Regarding to AUC unstable issue, the nondeterministic

[jira] [Created] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-16 Thread Stuart White (Jira)
Stuart White created SPARK-29101: Summary: CSV datasource returns incorrect .count() from file with malformed records Key: SPARK-29101 URL: https://issues.apache.org/jira/browse/SPARK-29101 Project:

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930801#comment-16930801 ] Karthikeyan Ravi commented on SPARK-25721: -- Attaching screenshot  !Screen Shot 2019-09-16 at

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930803#comment-16930803 ] Karthikeyan Ravi commented on SPARK-25721: -- This problem very similar to

[jira] [Updated] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthikeyan Ravi updated SPARK-25721: - Attachment: Screen Shot 2019-09-16 at 12.27.25 PM.png > maxRate configuration not being

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930796#comment-16930796 ] Karthikeyan Ravi commented on SPARK-25721: -- Hi Team, any updates on this, we are seeing a very

[jira] [Commented] (SPARK-26205) Optimize InSet expression for bytes, shorts, ints, dates

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930787#comment-16930787 ] Liang-Chi Hsieh commented on SPARK-26205: - [~cloud_fan]. I see now. Created SPARK-29100. >

[jira] [Assigned] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reassigned SPARK-29100: --- Assignee: Liang-Chi Hsieh > Codegen with switch in InSet expression causes

[jira] [Closed] (SPARK-23714) Add metrics for cached KafkaConsumer

2019-09-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-23714. - > Add metrics for cached KafkaConsumer > > >

[jira] [Resolved] (SPARK-23714) Add metrics for cached KafkaConsumer

2019-09-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-23714. --- Resolution: Duplicate Apache commons pool added which provides jmx metrics. > Add metrics

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-29100: Description: SPARK-26205 adds an optimization to InSet that generates Java switch

[jira] [Created] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
Liang-Chi Hsieh created SPARK-29100: --- Summary: Codegen with switch in InSet expression causes compilation error Key: SPARK-29100 URL: https://issues.apache.org/jira/browse/SPARK-29100 Project:

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930763#comment-16930763 ] Harichandan Pulagam edited comment on SPARK-27648 at 9/16/19 6:40 PM:

[jira] [Resolved] (SPARK-23694) The staging directory should under hive.exec.stagingdir if we set hive.exec.stagingdir but not under the table directory

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23694. --- Resolution: Won't Fix > The staging directory should under hive.exec.stagingdir if we set >

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930763#comment-16930763 ] Harichandan Pulagam edited comment on SPARK-27648 at 9/16/19 6:38 PM:

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930763#comment-16930763 ] Harichandan Pulagam commented on SPARK-27648: - Here's a code example that reproduces the

[jira] [Resolved] (SPARK-24671) DataFrame length using a dunder/magic method in PySpark

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24671. --- Resolution: Won't Fix > DataFrame length using a dunder/magic method in PySpark >

[jira] [Resolved] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19184. --- Resolution: Won't Fix > Improve numerical stability for method tallSkinnyQR. >

[jira] [Resolved] (SPARK-26524) If the application directory fails to be created on the SPARK_WORKER_DIR on some woker nodes (for example, bad disk or disk has no capacity), the application executor w

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26524. --- Resolution: Won't Fix > If the application directory fails to be created on the SPARK_WORKER_DIR on

[jira] [Assigned] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26929: -- Assignee: hong dongdong > table owner should use user instead of principal when use

[jira] [Resolved] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26929. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 23952

[jira] [Created] (SPARK-29099) org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set

2019-09-16 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-29099: Summary: org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set Key: SPARK-29099 URL: https://issues.apache.org/jira/browse/SPARK-29099

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:42 PM: --- thanks for

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:43 PM: --- thanks for

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:43 PM: --- thanks for

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:41 PM: --- for test, 

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930685#comment-16930685 ] JP Bordenave commented on SPARK-13446: -- i remove hive-exec from spark/jars and  hdfs  /spark-jars/

[jira] [Created] (SPARK-29098) Test both ANSI mode and Spark mode

2019-09-16 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-29098: -- Summary: Test both ANSI mode and Spark mode Key: SPARK-29098 URL: https://issues.apache.org/jira/browse/SPARK-29098 Project: Spark Issue Type: Sub-task

[jira] [Comment Edited] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930658#comment-16930658 ] Liang-Chi Hsieh edited comment on SPARK-28927 at 9/16/19 3:36 PM: --

[jira] [Comment Edited] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930658#comment-16930658 ] Liang-Chi Hsieh edited comment on SPARK-28927 at 9/16/19 3:35 PM: --

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930658#comment-16930658 ] Liang-Chi Hsieh commented on SPARK-28927: - Because you are using 2.2.1,

[jira] [Updated] (SPARK-29070) Make SparkLauncher log full spark-submit command line

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Evans updated SPARK-29070: --- Description: {{org.apache.spark.launcher.SparkLauncher}} wraps a {{ProcessBuilder}}, and builds up

[jira] [Updated] (SPARK-29070) Make SparkLauncher log full spark-submit command line

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Evans updated SPARK-29070: --- Summary: Make SparkLauncher log full spark-submit command line (was: Allow SparkLauncher to return

[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930616#comment-16930616 ] Imran Rashid commented on SPARK-28917: -- I finally got some more info about this case -- they are

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-16 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930600#comment-16930600 ] Thomas Graves commented on SPARK-27495: --- The SPIP vote passed - 

[jira] [Resolved] (SPARK-29072) Properly track shuffle write time with refactor

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29072. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25780

[jira] [Assigned] (SPARK-29072) Properly track shuffle write time with refactor

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29072: Assignee: Matt Cheah > Properly track shuffle write time with refactor >

[jira] [Assigned] (SPARK-29061) Prints bytecode statistics in debugCodegen

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29061: --- Assignee: Takeshi Yamamuro > Prints bytecode statistics in debugCodegen >

[jira] [Resolved] (SPARK-29061) Prints bytecode statistics in debugCodegen

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29061. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25766

  1   2   >