[jira] [Commented] (SPARK-23711) Add fallback to interpreted execution logic

2018-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443588#comment-16443588 ] Takeshi Yamamuro commented on SPARK-23711: -- +1, I also think so. > Add fallback to interpreted

[jira] [Commented] (SPARK-23711) Add fallback to interpreted execution logic

2018-04-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443573#comment-16443573 ] Liang-Chi Hsieh commented on SPARK-23711: - About this, I suppose that in

[jira] [Updated] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Prabhu Joseph (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-24008: -- Description: SQL / Hive Context fails with NullPointerException while getting configuration

[jira] [Commented] (SPARK-24015) Does method SchedulerBackend.isReady is really well implemented?

2018-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443509#comment-16443509 ] Hyukjin Kwon commented on SPARK-24015: -- [~chenk], is this a question? I think we would have a better

[jira] [Resolved] (SPARK-24015) Does method SchedulerBackend.isReady is really well implemented?

2018-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24015. -- Resolution: Invalid Let me leave this resolved but please let me know if I am mistaken. >

[jira] [Resolved] (SPARK-23919) High-order function: array_position(x, element) → bigint

2018-04-18 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23919. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21037

[jira] [Assigned] (SPARK-23919) High-order function: array_position(x, element) → bigint

2018-04-18 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23919: - Assignee: Kazuaki Ishizaki > High-order function: array_position(x, element) → bigint >

[jira] [Updated] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-18 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-24021: - Description: There's a miswrite in BlacklistTracker's updateBlacklistForFetchFailure:   {code:java} val

[jira] [Assigned] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24021: Assignee: Apache Spark > Fix bug in BlacklistTracker's updateBlacklistForFetchFailure >

[jira] [Assigned] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24021: Assignee: (was: Apache Spark) > Fix bug in BlacklistTracker's

[jira] [Commented] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443496#comment-16443496 ] Apache Spark commented on SPARK-24021: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Created] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-18 Thread wuyi (JIRA)
wuyi created SPARK-24021: Summary: Fix bug in BlacklistTracker's updateBlacklistForFetchFailure Key: SPARK-24021 URL: https://issues.apache.org/jira/browse/SPARK-24021 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443495#comment-16443495 ] Saisai Shao commented on SPARK-24008: - The case you provided seems not so valid. You're trying to

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Prabhu Joseph (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443484#comment-16443484 ] Prabhu Joseph commented on SPARK-24008: --- Yes it's driver specific. But better to handle this case

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443475#comment-16443475 ] Saisai Shao commented on SPARK-24008: - Why do you need to broadcast {{SQLContext}} or

[jira] [Resolved] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24014. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24014: --- Assignee: Liang-Chi Hsieh > Add onStreamingStarted method to StreamingListener >

[jira] [Commented] (SPARK-17901) NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException

2018-04-18 Thread zhangzhaolong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443449#comment-16443449 ] zhangzhaolong commented on SPARK-17901: --- hi [~srowen], did it reopen on master ? we encountered a

[jira] [Commented] (SPARK-24020) Sort-merge join inner range optimization

2018-04-18 Thread Henry Robinson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443392#comment-16443392 ] Henry Robinson commented on SPARK-24020: This sounds like a 'band join' (e.g.

[jira] [Created] (SPARK-24020) Sort-merge join inner range optimization

2018-04-18 Thread Petar Zecevic (JIRA)
Petar Zecevic created SPARK-24020: - Summary: Sort-merge join inner range optimization Key: SPARK-24020 URL: https://issues.apache.org/jira/browse/SPARK-24020 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-23775) Flaky test: DataFrameRangeSuite

2018-04-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23775. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Assigned] (SPARK-23775) Flaky test: DataFrameRangeSuite

2018-04-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23775: -- Assignee: Gabor Somogyi > Flaky test: DataFrameRangeSuite >

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-18 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443239#comment-16443239 ] Jordan Moore commented on SPARK-18057: -- Thanks, [~c...@koeninger.org]. I will forward that

[jira] [Reopened] (SPARK-19941) Spark should not schedule tasks on executors on decommissioning YARN nodes

2018-04-18 Thread Karthik Palaniappan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Palaniappan reopened SPARK-19941: - This should be marked as a duplicate of 

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2018-04-18 Thread Franck Tago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443154#comment-16443154 ] Franck Tago commented on SPARK-23519: - thanks for the suggestion [~shahid] The issue with your

[jira] [Created] (SPARK-24019) AnalysisException for Window function expression to compute derivative

2018-04-18 Thread Barry Becker (JIRA)
Barry Becker created SPARK-24019: Summary: AnalysisException for Window function expression to compute derivative Key: SPARK-24019 URL: https://issues.apache.org/jira/browse/SPARK-24019 Project:

[jira] [Updated] (SPARK-24018) Spark-without-hadoop package fails to create or read parquet files with snappy compression

2018-04-18 Thread Jean-Francis Roy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francis Roy updated SPARK-24018: - Description: On a brand-new installation of Spark 2.3.0 with a user-provided

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-18 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443025#comment-16443025 ] Edwina Lu commented on SPARK-23206: --- We are planning a design discussion for early next week. Please

[jira] [Created] (SPARK-24018) Spark-without-hadoop package fails to create or read parquet files with snappy compression

2018-04-18 Thread Jean-Francis Roy (JIRA)
Jean-Francis Roy created SPARK-24018: Summary: Spark-without-hadoop package fails to create or read parquet files with snappy compression Key: SPARK-24018 URL:

[jira] [Commented] (SPARK-23915) High-order function: array_except(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442960#comment-16442960 ] Apache Spark commented on SPARK-23915: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23915) High-order function: array_except(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23915: Assignee: Apache Spark > High-order function: array_except(x, y) → array >

[jira] [Assigned] (SPARK-23915) High-order function: array_except(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23915: Assignee: (was: Apache Spark) > High-order function: array_except(x, y) → array >

[jira] [Created] (SPARK-24017) Refactor ExternalCatalog to be an interface

2018-04-18 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24017: --- Summary: Refactor ExternalCatalog to be an interface Key: SPARK-24017 URL: https://issues.apache.org/jira/browse/SPARK-24017 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

2018-04-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442948#comment-16442948 ] Xiao Li commented on SPARK-23443: - We need to clean the ExternalCatalog interface at first before the

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-18 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442928#comment-16442928 ] Edwina Lu commented on SPARK-23206: --- [~assia6], there is an initial PR: [[Github] Pull Request #20940

[jira] [Resolved] (SPARK-17508) Setting weightCol to None in ML library causes an error

2018-04-18 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-17508. -- Resolution: Won't Fix Resolving this for now unless there is more interest in the fix >

[jira] [Updated] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23963: Fix Version/s: 2.3.1 2.2.2 > Queries on text-based Hive tables grow disproportionately

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-18 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442838#comment-16442838 ] Cody Koeninger commented on SPARK-18057: [~cricket007] here's a branch with spark 2.1.1 / kafka

[jira] [Commented] (SPARK-24010) Select from table needs read access on DB folder when storage based auth is enabled

2018-04-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442802#comment-16442802 ] Reynold Xin commented on SPARK-24010: - It's better to throw an error message to tell the user the db

[jira] [Commented] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442787#comment-16442787 ] Hyukjin Kwon commented on SPARK-24006: -- Yea, I wouldn't. Just simply wondering how much it could

[jira] [Commented] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-18 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442783#comment-16442783 ] Ruslan Dautkhanov commented on SPARK-23963: --- I thought it's just a matter of a Spark committer

[jira] [Commented] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-18 Thread Xianjin YE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442781#comment-16442781 ] Xianjin YE commented on SPARK-24006: Could you leave this issue open for a while? In my company, we

[jira] [Comment Edited] (SPARK-23933) High-order function: map(array, array) → map<K,V>

2018-04-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436976#comment-16436976 ] Kazuaki Ishizaki edited comment on SPARK-23933 at 4/18/18 4:22 PM: ---

[jira] [Assigned] (SPARK-23913) High-order function: array_intersect(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23913: Assignee: Apache Spark > High-order function: array_intersect(x, y) → array >

[jira] [Assigned] (SPARK-23913) High-order function: array_intersect(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23913: Assignee: (was: Apache Spark) > High-order function: array_intersect(x, y) → array >

[jira] [Commented] (SPARK-23913) High-order function: array_intersect(x, y) → array

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442766#comment-16442766 ] Apache Spark commented on SPARK-23913: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442735#comment-16442735 ] Hyukjin Kwon commented on SPARK-24006: -- Would be good to make a small test with the logic above and

[jira] [Updated] (SPARK-24015) Does method SchedulerBackend.isReady is really well implemented?

2018-04-18 Thread chenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenk updated SPARK-24015: -- Docs Text: (was: The mothod isReady in the interface SchedulerBackend,which implement by

[jira] [Created] (SPARK-24016) Yarn does not update node blacklist in static allocation

2018-04-18 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24016: Summary: Yarn does not update node blacklist in static allocation Key: SPARK-24016 URL: https://issues.apache.org/jira/browse/SPARK-24016 Project: Spark

[jira] [Updated] (SPARK-24015) Does method SchedulerBackend.isReady is really well implemented?

2018-04-18 Thread chenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenk updated SPARK-24015: -- Summary: Does method SchedulerBackend.isReady is really well implemented? (was: Does method

[jira] [Updated] (SPARK-24015) Does method SchedulerBackend.isReady is really well implemented

2018-04-18 Thread chenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenk updated SPARK-24015: -- Summary: Does method SchedulerBackend.isReady is really well implemented (was: Does method

[jira] [Resolved] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24007. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 2.2.2 >

[jira] [Created] (SPARK-24015) Does method BackendReady.isReady is really well implemented

2018-04-18 Thread chenk (JIRA)
chenk created SPARK-24015: - Summary: Does method BackendReady.isReady is really well implemented Key: SPARK-24015 URL: https://issues.apache.org/jira/browse/SPARK-24015 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442568#comment-16442568 ] Wenchen Fan edited comment on SPARK-23989 at 4/18/18 2:47 PM: -- OK now I see

[jira] [Assigned] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23989: Assignee: (was: Apache Spark) > When using `SortShuffleWriter`, the data will be

[jira] [Assigned] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23989: Assignee: Apache Spark > When using `SortShuffleWriter`, the data will be overwritten >

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442620#comment-16442620 ] Apache Spark commented on SPARK-23989: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24012: Assignee: Apache Spark > Union of map and other compatible column >

[jira] [Assigned] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24012: Assignee: (was: Apache Spark) > Union of map and other compatible column >

[jira] [Commented] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442615#comment-16442615 ] Apache Spark commented on SPARK-24012: -- User 'liutang123' has created a pull request for this issue:

[jira] [Updated] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-24012: -- Description: Union of map and other compatible column result in unresolved operator 'Union; exception

[jira] [Commented] (SPARK-23875) Create IndexedSeq wrapper for ArrayData

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442602#comment-16442602 ] Apache Spark commented on SPARK-23875: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442568#comment-16442568 ] Wenchen Fan commented on SPARK-23989: - OK now I see the problem,

[jira] [Assigned] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24014: Assignee: (was: Apache Spark) > Add onStreamingStarted method to StreamingListener >

[jira] [Commented] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442553#comment-16442553 ] Apache Spark commented on SPARK-24014: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24014: Assignee: Apache Spark > Add onStreamingStarted method to StreamingListener >

[jira] [Created] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-24014: --- Summary: Add onStreamingStarted method to StreamingListener Key: SPARK-24014 URL: https://issues.apache.org/jira/browse/SPARK-24014 Project: Spark

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442514#comment-16442514 ] Wenchen Fan commented on SPARK-23989: - it goes to `SortShuffleWriter` and then? We get the correct

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442488#comment-16442488 ] Jacek Laskowski commented on SPARK-23830: - It's about how easy it is to find out that the issue

[jira] [Commented] (SPARK-24013) ApproximatePercentile grinds to a halt on sorted input.

2018-04-18 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442477#comment-16442477 ] Juliusz Sompolski commented on SPARK-24013: --- This hits when trying to create histogram

[jira] [Created] (SPARK-24013) ApproximatePercentile grinds to a halt on sorted input.

2018-04-18 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-24013: - Summary: ApproximatePercentile grinds to a halt on sorted input. Key: SPARK-24013 URL: https://issues.apache.org/jira/browse/SPARK-24013 Project: Spark

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-18 Thread assia ydroudj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442379#comment-16442379 ] assia ydroudj commented on SPARK-23206: --- [~elu], thank you for the shared dos, it works for me Is

[jira] [Commented] (SPARK-23990) Instruments logging improvements - ML regression package

2018-04-18 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442370#comment-16442370 ] Weichen Xu commented on SPARK-23990: [~josephkb] I agree that option 2b looks better. Updating PR...

[jira] [Commented] (SPARK-14682) Provide evaluateEachIteration method or equivalent for spark.ml GBTs

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442368#comment-16442368 ] Apache Spark commented on SPARK-14682: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-14682) Provide evaluateEachIteration method or equivalent for spark.ml GBTs

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14682: Assignee: (was: Apache Spark) > Provide evaluateEachIteration method or equivalent

[jira] [Assigned] (SPARK-14682) Provide evaluateEachIteration method or equivalent for spark.ml GBTs

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14682: Assignee: Apache Spark > Provide evaluateEachIteration method or equivalent for spark.ml

[jira] [Updated] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu updated SPARK-24012: -- Description: Union of map and other compatible column result in unresolved operator 'Union; exception

[jira] [Created] (SPARK-24012) Union of map and other compatible column

2018-04-18 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-24012: - Summary: Union of map and other compatible column Key: SPARK-24012 URL: https://issues.apache.org/jira/browse/SPARK-24012 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24011) Cache rdd's immediate parent ShuffleDependencies to accelerate getShuffleDependencies()

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442259#comment-16442259 ] Apache Spark commented on SPARK-24011: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24011) Cache rdd's immediate parent ShuffleDependencies to accelerate getShuffleDependencies()

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24011: Assignee: Apache Spark > Cache rdd's immediate parent ShuffleDependencies to accelerate

[jira] [Assigned] (SPARK-24011) Cache rdd's immediate parent ShuffleDependencies to accelerate getShuffleDependencies()

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24011: Assignee: (was: Apache Spark) > Cache rdd's immediate parent ShuffleDependencies to

[jira] [Updated] (SPARK-24011) Cache rdd's immediate parent ShuffleDependencies to accelerate getShuffleDependencies()

2018-04-18 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-24011: - Summary: Cache rdd's immediate parent ShuffleDependencies to accelerate getShuffleDependencies() (was: Cache

[jira] [Created] (SPARK-24011) Cache rdd's immediate parent ShuffleDependency to accelerate getShuffleDependencies()

2018-04-18 Thread wuyi (JIRA)
wuyi created SPARK-24011: Summary: Cache rdd's immediate parent ShuffleDependency to accelerate getShuffleDependencies() Key: SPARK-24011 URL: https://issues.apache.org/jira/browse/SPARK-24011 Project: Spark

[jira] [Commented] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-18 Thread Xianjin YE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442224#comment-16442224 ] Xianjin YE commented on SPARK-24006: Haven't launch a large enough job to confirm my assumption..

[jira] [Resolved] (SPARK-23926) High-order function: reverse(x) → array

2018-04-18 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23926. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21034

[jira] [Assigned] (SPARK-23926) High-order function: reverse(x) → array

2018-04-18 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23926: - Assignee: Marek Novotny > High-order function: reverse(x) → array >

[jira] [Comment Edited] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442176#comment-16442176 ] liuxian edited comment on SPARK-23989 at 4/18/18 9:21 AM: --

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442176#comment-16442176 ] liuxian commented on SPARK-23989: - test({color:#6a8759}"groupBy"{color}) { {color:#808080} 

[jira] [Commented] (SPARK-24009) spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'

2018-04-18 Thread ANDY GUAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442165#comment-16442165 ] ANDY GUAN commented on SPARK-24009: --- Try to config *hive.exec.scratchdir* and *hive.exec.stagingdir* in

[jira] [Issue Comment Deleted] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread ANDY GUAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANDY GUAN updated SPARK-24008: -- Comment: was deleted (was: Try to config *hive.exec.scratchdir* and *hive.exec.stagingdir* in

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread ANDY GUAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442164#comment-16442164 ] ANDY GUAN commented on SPARK-24008: --- Try to config *hive.exec.scratchdir* and *hive.exec.stagingdir* in

[jira] [Commented] (SPARK-24010) Select from table needs read access on DB folder when storage based auth is enabled

2018-04-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442146#comment-16442146 ] Rui Li commented on SPARK-24010: Hi [~rxin], I think checking databaseExists is added in SPARK-14869. Do

[jira] [Commented] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442147#comment-16442147 ] Hyukjin Kwon commented on SPARK-24006: -- How many executors would you guess cause an actual problem

[jira] [Created] (SPARK-24010) Select from table needs read access on DB folder when storage based auth is enabled

2018-04-18 Thread Rui Li (JIRA)
Rui Li created SPARK-24010: -- Summary: Select from table needs read access on DB folder when storage based auth is enabled Key: SPARK-24010 URL: https://issues.apache.org/jira/browse/SPARK-24010 Project:

[jira] [Updated] (SPARK-24009) spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'

2018-04-18 Thread chris_j (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris_j updated SPARK-24009: Description: 1.spark-sql -e "INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/ab'row format delimited

[jira] [Commented] (SPARK-23529) Specify hostpath volume and mount the volume in Spark driver and executor pods in Kubernetes

2018-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442050#comment-16442050 ] Apache Spark commented on SPARK-23529: -- User 'madanadit' has created a pull request for this issue:

[jira] [Updated] (SPARK-24009) spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'

2018-04-18 Thread chris_j (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris_j updated SPARK-24009: Description: 1.spark-sql -e "INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/ab'row format delimited

[jira] [Created] (SPARK-24009) spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'

2018-04-18 Thread chris_j (JIRA)
chris_j created SPARK-24009: --- Summary: spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/ab' Key: SPARK-24009 URL: https://issues.apache.org/jira/browse/SPARK-24009 Project: Spark

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441997#comment-16441997 ] Wenchen Fan commented on SPARK-23989: - You have to provide an end-to-end use case to convince us this

[jira] [Updated] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-18 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23989: Attachment: (was: 无标题2.png) > When using `SortShuffleWriter`, the data will be overwritten >

  1   2   >