[jira] [Resolved] (SPARK-25133) Documentaion: AVRO data source guide

2018-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25133. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22121

[jira] [Assigned] (SPARK-25133) Documentaion: AVRO data source guide

2018-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25133: --- Assignee: Gengliang Wang > Documentaion: AVRO data source guide >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-08-22 Thread Leo Gallucci (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589721#comment-16589721 ] Leo Gallucci commented on SPARK-18112: -- Same issue. It only gets resolved if I remove

[jira] [Commented] (SPARK-25121) Support multi-part column name for hint resolution

2018-08-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589662#comment-16589662 ] Takeshi Yamamuro commented on SPARK-25121: -- Anybody starts working on this? If no, I could take

[jira] [Resolved] (SPARK-25197) Read from java.nio.Path

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25197. -- Resolution: Won't Fix I would just simply convert the path to string. Wouldn't necessarily

[jira] [Updated] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25176: - Priority: Major (was: Critical) > Kryo fails to serialize a parametrised type hierarchy >

[jira] [Commented] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589649#comment-16589649 ] Hyukjin Kwon commented on SPARK-25176: -- (please avoid to set Critical+ which is usually reserved

[jira] [Updated] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25132: - Summary: Case-insensitive field resolution when reading from Parquet (was: Case-insensitive

[jira] [Updated] (SPARK-25132) Case-insensitive field resolution when reading from Parquet/ORC

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25132: - Summary: Case-insensitive field resolution when reading from Parquet/ORC (was:

[jira] [Updated] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25132: - Summary: Case-insensitive field resolution when reading from Parquet (was: Case-insensitive

[jira] [Resolved] (SPARK-25192) Remove SupportsPushdownCatalystFilter

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25192. -- Resolution: Duplicate > Remove SupportsPushdownCatalystFilter >

[jira] [Assigned] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25205: Assignee: (was: Apache Spark) > typo in spark.network.crypto.keyFactoryIteration >

[jira] [Commented] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589636#comment-16589636 ] Apache Spark commented on SPARK-25205: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25205: Assignee: Apache Spark > typo in spark.network.crypto.keyFactoryIteration >

[jira] [Created] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

2018-08-22 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-25205: Summary: typo in spark.network.crypto.keyFactoryIteration Key: SPARK-25205 URL: https://issues.apache.org/jira/browse/SPARK-25205 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-25167) Minor fixes for R sql tests (tests that fail in development environment)

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25167. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22161

[jira] [Assigned] (SPARK-25167) Minor fixes for R sql tests (tests that fail in development environment)

2018-08-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25167: Assignee: Dilip Biswal > Minor fixes for R sql tests (tests that fail in development

[jira] [Comment Edited] (SPARK-25196) Analyze column statistics in cached query

2018-08-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589604#comment-16589604 ] Takeshi Yamamuro edited comment on SPARK-25196 at 8/23/18 2:28 AM: ---

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

2018-08-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589604#comment-16589604 ] Takeshi Yamamuro commented on SPARK-25196: -- yea, sure. So, I'll make a pr after branch-2.4 cut.

[jira] [Commented] (SPARK-25202) SQL Function Split Should Respect Limit Argument

2018-08-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589601#comment-16589601 ] Liang-Chi Hsieh commented on SPARK-25202: - Let me see if I have time to do this today later. >

[jira] [Commented] (SPARK-25202) SQL Function Split Should Respect Limit Argument

2018-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589596#comment-16589596 ] Wenchen Fan commented on SPARK-25202: - SGTM > SQL Function Split Should Respect Limit Argument >

[jira] [Commented] (SPARK-23932) High-order function: zip_with(array, array, function) → array

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589588#comment-16589588 ] Apache Spark commented on SPARK-23932: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-25202) SQL Function Split Should Respect Limit Argument

2018-08-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589558#comment-16589558 ] Liang-Chi Hsieh commented on SPARK-25202: - I saw Presto has this support. Is it worth adding

[jira] [Updated] (SPARK-25178) Directly ship the StructType objects of the keySchema / valueSchema for xxxHashMapGenerator

2018-08-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-25178: - Summary: Directly ship the StructType objects of the keySchema / valueSchema for

[jira] [Commented] (SPARK-25198) org.apache.spark.sql.catalyst.parser.ParseException: DataType json is not supported.

2018-08-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589547#comment-16589547 ] Liang-Chi Hsieh commented on SPARK-25198: - I think the {{customSchema}} here refers to Spark's

[jira] [Resolved] (SPARK-25127) DataSourceV2: Remove SupportsPushDownCatalystFilters

2018-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25127. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22185

[jira] [Assigned] (SPARK-25186) Stabilize Data Source V2 API

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25186: Assignee: Apache Spark > Stabilize Data Source V2 API > - >

[jira] [Assigned] (SPARK-25186) Stabilize Data Source V2 API

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25186: Assignee: (was: Apache Spark) > Stabilize Data Source V2 API >

[jira] [Commented] (SPARK-25186) Stabilize Data Source V2 API

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589489#comment-16589489 ] Apache Spark commented on SPARK-25186: -- User 'rdblue' has created a pull request for this issue:

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589483#comment-16589483 ] Apache Spark commented on SPARK-24918: -- User 'NiharS' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24785) Making sure REPL prints Spark UI info and then Welcome message

2018-08-22 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24785: --- Assignee: DB Tsai > Making sure REPL prints Spark UI info and then Welcome message >

[jira] [Resolved] (SPARK-24785) Making sure REPL prints Spark UI info and then Welcome message

2018-08-22 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24785. - Resolution: Fixed > Making sure REPL prints Spark UI info and then Welcome message >

[jira] [Commented] (SPARK-25204) rate source test is flaky

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589404#comment-16589404 ] Apache Spark commented on SPARK-25204: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-25204) rate source test is flaky

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25204: Assignee: (was: Apache Spark) > rate source test is flaky >

[jira] [Assigned] (SPARK-25204) rate source test is flaky

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25204: Assignee: Apache Spark > rate source test is flaky > - > >

[jira] [Created] (SPARK-25204) rate source test is flaky

2018-08-22 Thread Jose Torres (JIRA)
Jose Torres created SPARK-25204: --- Summary: rate source test is flaky Key: SPARK-25204 URL: https://issues.apache.org/jira/browse/SPARK-25204 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-25188) Add WriteConfig

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589367#comment-16589367 ] Apache Spark commented on SPARK-25188: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25188) Add WriteConfig

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25188: Assignee: (was: Apache Spark) > Add WriteConfig > --- > >

[jira] [Assigned] (SPARK-25188) Add WriteConfig

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25188: Assignee: Apache Spark > Add WriteConfig > --- > > Key:

[jira] [Resolved] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

2018-08-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25163. -- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0 > Flaky

[jira] [Commented] (SPARK-25203) spark sql, union all does not propagate child partitioning (when possible)

2018-08-22 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589356#comment-16589356 ] Eyal Farago commented on SPARK-25203: - seems I was wrong regarding the resulting distribution,

[jira] [Commented] (SPARK-25119) stages in wrong order within job page DAG chart

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589352#comment-16589352 ] Apache Spark commented on SPARK-25119: -- User 'yunjzhang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25119) stages in wrong order within job page DAG chart

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25119: Assignee: (was: Apache Spark) > stages in wrong order within job page DAG chart >

[jira] [Assigned] (SPARK-25119) stages in wrong order within job page DAG chart

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25119: Assignee: Apache Spark > stages in wrong order within job page DAG chart >

[jira] [Commented] (SPARK-25195) Extending from_json function

2018-08-22 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589335#comment-16589335 ] Maxim Gekk commented on SPARK-25195: > Problem number 1: The from_json function accepts as a schema

[jira] [Commented] (SPARK-25203) spark sql, union all does not propagate child partitioning (when possible)

2018-08-22 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589332#comment-16589332 ] Eyal Farago commented on SPARK-25203: - CC: [~hvanhovell], [~cloud_fan] > spark sql, union all does

[jira] [Created] (SPARK-25203) spark sql, union all does not propagate child partitioning (when possible)

2018-08-22 Thread Eyal Farago (JIRA)
Eyal Farago created SPARK-25203: --- Summary: spark sql, union all does not propagate child partitioning (when possible) Key: SPARK-25203 URL: https://issues.apache.org/jira/browse/SPARK-25203 Project:

[jira] [Created] (SPARK-25202) SQL Function Split Should Respect Limit Argument

2018-08-22 Thread Parker Hegstrom (JIRA)
Parker Hegstrom created SPARK-25202: --- Summary: SQL Function Split Should Respect Limit Argument Key: SPARK-25202 URL: https://issues.apache.org/jira/browse/SPARK-25202 Project: Spark Issue

[jira] [Commented] (SPARK-25126) avoid creating OrcFile.Reader for all orc files

2018-08-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589326#comment-16589326 ] Steve Loughran commented on SPARK-25126: + [~dongjoon] > avoid creating OrcFile.Reader for all

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Matt Sicker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589325#comment-16589325 ] Matt Sicker commented on SPARK-6305: Could be possible that nobody is swapping it out for JUL since

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589311#comment-16589311 ] Chris Martin commented on SPARK-6305: - I don't think that's such a big deal so long as Spark can have

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Matt Sicker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589307#comment-16589307 ] Matt Sicker commented on SPARK-6305: Right, both slf4j-api and log4j-api (the API half of Log4j2)

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589303#comment-16589303 ] Sean Owen commented on SPARK-6305: -- Ah, Java 9 is a good point. That may force the issue. Yes you can

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589293#comment-16589293 ] Chris Martin commented on SPARK-6305: - Thanks [~srowen] and [~ste...@apache.org] for the feedback.

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2018-08-22 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589288#comment-16589288 ] Wes McKinney commented on SPARK-21375: -- Seems there might be some requirements that need to be

[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host

2018-08-22 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589286#comment-16589286 ] Yinan Li commented on SPARK-25162: -- > Where the driver is running _outside-cluster client_ mode,  would

[jira] [Commented] (SPARK-25194) Kubernetes - Define cpu and memory limit to init container

2018-08-22 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589283#comment-16589283 ] Yinan Li commented on SPARK-25194: -- The upcoming Spark 2.4 gets rid of the init-container and switch to

[jira] [Resolved] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

2018-08-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25184. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22182

[jira] [Assigned] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

2018-08-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25184: - Assignee: Tathagata Das > Flaky test: FlatMapGroupsWithState "streaming with

[jira] [Created] (SPARK-25201) Synchronization performed on AtomicReference in LevelDB class

2018-08-22 Thread Ted Yu (JIRA)
Ted Yu created SPARK-25201: -- Summary: Synchronization performed on AtomicReference in LevelDB class Key: SPARK-25201 URL: https://issues.apache.org/jira/browse/SPARK-25201 Project: Spark Issue

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2018-08-22 Thread Matt Sicker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589255#comment-16589255 ] Matt Sicker commented on SPARK-6305: Unless you're willing to patch Log4j 1.x and maintain a fork, it

[jira] [Assigned] (SPARK-25199) InferSchema "all Strings" if one of many CSVs is empty

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25199: Assignee: Apache Spark > InferSchema "all Strings" if one of many CSVs is empty >

[jira] [Assigned] (SPARK-25199) InferSchema "all Strings" if one of many CSVs is empty

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25199: Assignee: (was: Apache Spark) > InferSchema "all Strings" if one of many CSVs is

[jira] [Commented] (SPARK-25199) InferSchema "all Strings" if one of many CSVs is empty

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589252#comment-16589252 ] Apache Spark commented on SPARK-25199: -- User 'yunjzhang' has created a pull request for this issue:

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589249#comment-16589249 ] Marcelo Vanzin commented on SPARK-24918: Unless he gives you push access to his repo, that's

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-22 Thread Nihar Sheth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589248#comment-16589248 ] Nihar Sheth commented on SPARK-24918: - [~irashid] has asked me to add testing to his PR. I'm not

[jira] [Created] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

2018-08-22 Thread Adam Balogh (JIRA)
Adam Balogh created SPARK-25200: --- Summary: Allow setting HADOOP_CONF_DIR as a spark property Key: SPARK-25200 URL: https://issues.apache.org/jira/browse/SPARK-25200 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25164) Parquet reader builds entire list of columns once for each column

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25164: Assignee: Apache Spark > Parquet reader builds entire list of columns once for each

[jira] [Assigned] (SPARK-25164) Parquet reader builds entire list of columns once for each column

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25164: Assignee: (was: Apache Spark) > Parquet reader builds entire list of columns once

[jira] [Commented] (SPARK-25164) Parquet reader builds entire list of columns once for each column

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589228#comment-16589228 ] Apache Spark commented on SPARK-25164: -- User 'bersprockets' has created a pull request for this

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589220#comment-16589220 ] Apache Spark commented on SPARK-25178: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25178: Assignee: (was: Apache Spark) > Use dummy name for xxxHashMapGenerator key/value

[jira] [Assigned] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25178: Assignee: Apache Spark > Use dummy name for xxxHashMapGenerator key/value schema field >

[jira] [Created] (SPARK-25199) InferSchema "all Strings" if one of many CSVs is empty

2018-08-22 Thread Neil McGuigan (JIRA)
Neil McGuigan created SPARK-25199: - Summary: InferSchema "all Strings" if one of many CSVs is empty Key: SPARK-25199 URL: https://issues.apache.org/jira/browse/SPARK-25199 Project: Spark

[jira] [Updated] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the characters per row before truncation when a user runs.show()

2018-08-22 Thread Andrew K Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew K Long updated SPARK-24442: -- Summary: Add configuration parameter to adjust the numbers of records and the characters per

[jira] [Commented] (SPARK-25147) GroupedData.apply pandas_udf crashing

2018-08-22 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589177#comment-16589177 ] Bryan Cutler commented on SPARK-25147: -- Works for me on linux with: Python 3.6.6 pyarrow 0.10.0

[jira] [Resolved] (SPARK-25181) Block Manager master and slave thread pools are unbounded

2018-08-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25181. -- Resolution: Fixed Assignee: Mukul Murthy Fix Version/s: 2.4.0 > Block Manager

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2018-08-22 Thread Alexander (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589151#comment-16589151 ] Alexander commented on SPARK-7768: -- Spark Maintainers, [~viirya], [~jodersky], what do you think? >

[jira] [Assigned] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25183: Assignee: (was: Apache Spark) > Spark HiveServer2 registers shutdown hook with JVM,

[jira] [Commented] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589149#comment-16589149 ] Apache Spark commented on SPARK-25183: -- User 'steveloughran' has created a pull request for this

[jira] [Assigned] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25183: Assignee: Apache Spark > Spark HiveServer2 registers shutdown hook with JVM, not

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2018-08-22 Thread Alexander (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589146#comment-16589146 ] Alexander commented on SPARK-7768: -- Thanks [~eje] and [~metasim] for the awesome examples! I haven't

[jira] [Comment Edited] (SPARK-7768) Make user-defined type (UDT) API public

2018-08-22 Thread Alexander (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589146#comment-16589146 ] Alexander edited comment on SPARK-7768 at 8/22/18 5:24 PM: --- Thanks [~eje] and

[jira] [Commented] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-08-22 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589143#comment-16589143 ] Bryan Cutler commented on SPARK-23698: -- Followup resolved by pull request 20838

[jira] [Resolved] (SPARK-25105) Importing all of pyspark.sql.functions should bring PandasUDFType in as well

2018-08-22 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-25105. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22100

[jira] [Assigned] (SPARK-25105) Importing all of pyspark.sql.functions should bring PandasUDFType in as well

2018-08-22 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-25105: Assignee: kevin yu > Importing all of pyspark.sql.functions should bring PandasUDFType

[jira] [Created] (SPARK-25198) org.apache.spark.sql.catalyst.parser.ParseException: DataType json is not supported.

2018-08-22 Thread antonkulaga (JIRA)
antonkulaga created SPARK-25198: --- Summary: org.apache.spark.sql.catalyst.parser.ParseException: DataType json is not supported. Key: SPARK-25198 URL: https://issues.apache.org/jira/browse/SPARK-25198

[jira] [Commented] (SPARK-25188) Add WriteConfig

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589088#comment-16589088 ] Ryan Blue commented on SPARK-25188: --- One update to that proposal: {{BatchOverwriteSupport}} should be

[jira] [Commented] (SPARK-25188) Add WriteConfig

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589086#comment-16589086 ] Ryan Blue commented on SPARK-25188: --- Here's the original proposal for adding a write config: The read

[jira] [Commented] (SPARK-25127) DataSourceV2: Remove SupportsPushDownCatalystFilters

2018-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589084#comment-16589084 ] Apache Spark commented on SPARK-25127: -- User 'rxin' has created a pull request for this issue:

[jira] [Commented] (SPARK-25190) Better operator pushdown API

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589070#comment-16589070 ] Ryan Blue commented on SPARK-25190: --- The main problem I have with the current pushdown API is that

[jira] [Comment Edited] (SPARK-25187) Revisit the life cycle of ReadSupport instances.

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589060#comment-16589060 ] Ryan Blue edited comment on SPARK-25187 at 8/22/18 3:58 PM: The need for

[jira] [Comment Edited] (SPARK-25187) Revisit the life cycle of ReadSupport instances.

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589060#comment-16589060 ] Ryan Blue edited comment on SPARK-25187 at 8/22/18 3:57 PM: The need for

[jira] [Commented] (SPARK-25187) Revisit the life cycle of ReadSupport instances.

2018-08-22 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589060#comment-16589060 ] Ryan Blue commented on SPARK-25187: --- The need for {{newScanConfigBuilder}} to take key-value options

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

2018-08-22 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589050#comment-16589050 ] Reynold Xin commented on SPARK-25196: - Can we rework the interface so the two are not separate code

[jira] [Commented] (SPARK-25147) GroupedData.apply pandas_udf crashing

2018-08-22 Thread Mike Sukmanowsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589027#comment-16589027 ] Mike Sukmanowsky commented on SPARK-25147: -- [~hyukjin.kwon] should I take any other action

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-08-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589018#comment-16589018 ] Xiao Li commented on SPARK-23874: - [~bryanc] Could you list some examples that can affect our Spark

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

2018-08-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589015#comment-16589015 ] Xiao Li commented on SPARK-25178: - Thanks! > Use dummy name for xxxHashMapGenerator key/value schema

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

2018-08-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589013#comment-16589013 ] Xiao Li commented on SPARK-25196: - This sounds reasonable to me. > Analyze column statistics in cached

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

2018-08-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589014#comment-16589014 ] Xiao Li commented on SPARK-25196: - cc [~rxin] > Analyze column statistics in cached query >

  1   2   >