[jira] [Updated] (SPARK-25696) The storage memory displayed on spark Application UI is incorrect.

2018-10-09 Thread hantiantian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hantiantian updated SPARK-25696: Description: In the reported heartbeat information, the unit of the memory data is bytes, which i

[jira] [Commented] (SPARK-25696) The storage memory displayed on spark Application UI is incorrect.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644527#comment-16644527 ] Apache Spark commented on SPARK-25696: -- User 'httfighter' has created a pull reques

[jira] [Assigned] (SPARK-25696) The storage memory displayed on spark Application UI is incorrect.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25696: Assignee: Apache Spark > The storage memory displayed on spark Application UI is incorrec

[jira] [Assigned] (SPARK-25696) The storage memory displayed on spark Application UI is incorrect.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25696: Assignee: (was: Apache Spark) > The storage memory displayed on spark Application UI

[jira] [Commented] (SPARK-25525) Do not update conf for existing SparkContext in SparkSession.getOrCreate.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644526#comment-16644526 ] Apache Spark commented on SPARK-25525: -- User 'ueshin' has created a pull request fo

[jira] [Created] (SPARK-25696) The storage memory displayed on spark Application UI is incorrect.

2018-10-09 Thread hantiantian (JIRA)
hantiantian created SPARK-25696: --- Summary: The storage memory displayed on spark Application UI is incorrect. Key: SPARK-25696 URL: https://issues.apache.org/jira/browse/SPARK-25696 Project: Spark

[jira] [Commented] (SPARK-25525) Do not update conf for existing SparkContext in SparkSession.getOrCreate.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644521#comment-16644521 ] Apache Spark commented on SPARK-25525: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-25525) Do not update conf for existing SparkContext in SparkSession.getOrCreate.

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644522#comment-16644522 ] Apache Spark commented on SPARK-25525: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-20946) Do not update conf for existing SparkContext in SparkSession.getOrCreate

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644519#comment-16644519 ] Apache Spark commented on SPARK-20946: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-20946) Do not update conf for existing SparkContext in SparkSession.getOrCreate

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644520#comment-16644520 ] Apache Spark commented on SPARK-20946: -- User 'ueshin' has created a pull request fo

[jira] [Comment Edited] (SPARK-25695) Spark history server event log store problem

2018-10-09 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644503#comment-16644503 ] shahid edited comment on SPARK-25695 at 10/10/18 6:07 AM: -- Hi,

[jira] [Commented] (SPARK-25695) Spark history server event log store problem

2018-10-09 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644503#comment-16644503 ] shahid commented on SPARK-25695: Hi, This error seems, eventLog directory doesn't exist

[jira] [Updated] (SPARK-25695) Spark history server event log store problem

2018-10-09 Thread Si Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Si Chen updated SPARK-25695: Description: Envirment:spark 2.0.0 hadoop 2.7.3 spark-default.conf {code:java} spark.driver.extraLibraryP

[jira] [Created] (SPARK-25695) Spark history server event log store problem

2018-10-09 Thread Si Chen (JIRA)
Si Chen created SPARK-25695: --- Summary: Spark history server event log store problem Key: SPARK-25695 URL: https://issues.apache.org/jira/browse/SPARK-25695 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25640) Clarify/Improve EvalType for grouped aggregate and window aggregate

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25640: Target Version/s: 3.0.0 > Clarify/Improve EvalType for grouped aggregate and window aggregate > --

[jira] [Commented] (SPARK-24291) Data source table is not displaying records when files are uploaded to table location

2018-10-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644459#comment-16644459 ] Sean Owen commented on SPARK-24291: --- I don't know. I would not expect this to necessar

[jira] [Commented] (SPARK-24291) Data source table is not displaying records when files are uploaded to table location

2018-10-09 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644435#comment-16644435 ] sandeep katta commented on SPARK-24291: --- [~cloud_fan] [~srowen] is this valid use

[jira] [Commented] (SPARK-21569) Internal Spark class needs to be kryo-registered

2018-10-09 Thread Lijun Cao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644394#comment-16644394 ] Lijun Cao commented on SPARK-21569: --- Register *org.apache.spark.internal.io.FileCommit

[jira] [Commented] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644273#comment-16644273 ] Xiao Li commented on SPARK-25690: - The changes made in https://issues.apache.org/jira/br

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644270#comment-16644270 ] Apache Spark commented on SPARK-25682: -- User 'vanzin' has created a pull request fo

[jira] [Assigned] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25682: Assignee: Apache Spark > Docker images generated from dev build and from dist tarball are

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644269#comment-16644269 ] Apache Spark commented on SPARK-25682: -- User 'vanzin' has created a pull request fo

[jira] [Assigned] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25682: Assignee: (was: Apache Spark) > Docker images generated from dev build and from dist

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644198#comment-16644198 ] Yinan Li commented on SPARK-25682: -- Cool, thanks! > Docker images generated from dev b

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644194#comment-16644194 ] Marcelo Vanzin commented on SPARK-25682: I was busy with other things but I can

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Description: URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlCon

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Summary: URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue (was: URL.setURLStre

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Description: URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlCon

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Description: URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlCon

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Priority: Minor (was: Major) > URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issu

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Description: URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlCon

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection

2018-10-09 Thread Bo Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Yang updated SPARK-25694: Description: URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlCon

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644162#comment-16644162 ] Shixiong Zhu commented on SPARK-25692: -- It may be caused by https://github.com/apac

[jira] [Created] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection

2018-10-09 Thread Bo Yang (JIRA)
Bo Yang created SPARK-25694: --- Summary: URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection Key: SPARK-25694 URL: https://issues.apache.org/jira/browse/SPARK-25694 Project: Spark

[jira] [Updated] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25692: - Description: Looks like the whole test suite is pretty flaky. See: https://amplab.cs.berkeley.e

[jira] [Updated] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25692: Affects Version/s: (was: 2.4.0) 3.0.0 > Flaky test: ChunkFetchIntegrationSuite

[jira] [Created] (SPARK-25693) Fix the multiple "manager" classes in org.apache.spark.deploy.security

2018-10-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25693: -- Summary: Fix the multiple "manager" classes in org.apache.spark.deploy.security Key: SPARK-25693 URL: https://issues.apache.org/jira/browse/SPARK-25693 Project: S

[jira] [Updated] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25692: Priority: Blocker (was: Major) > Flaky test: ChunkFetchIntegrationSuite > ---

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644157#comment-16644157 ] Yinan Li commented on SPARK-25682: -- That looks like to me the only difference. {{bin}},

[jira] [Created] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-09 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-25692: Summary: Flaky test: ChunkFetchIntegrationSuite Key: SPARK-25692 URL: https://issues.apache.org/jira/browse/SPARK-25692 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644117#comment-16644117 ] Sean Owen commented on SPARK-25690: --- I'm unclear what the particular problem is here.

[jira] [Assigned] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25690: - Assignee: (was: Sean Owen) Fix Version/s: (was: 2.4.0) > Analyzer rule "Handle

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644036#comment-16644036 ] Shixiong Zhu commented on SPARK-23390: -- I didn't look at parquet. It may have a sim

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644034#comment-16644034 ] Shixiong Zhu commented on SPARK-23390: -- I think the issue is probably in orc. Any e

[jira] [Updated] (SPARK-25650) Make analyzer rules used in once-policy idempotent

2018-10-09 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25650: Description: Rules like {{HandleNullInputsForUDF}}  (https://issues.apache.org/jira/browse/SPARK-2

[jira] [Created] (SPARK-25691) Analyzer rule "AliasViewChild" does not stabilize

2018-10-09 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25691: --- Summary: Analyzer rule "AliasViewChild" does not stabilize Key: SPARK-25691 URL: https://issues.apache.org/jira/browse/SPARK-25691 Project: Spark Issue Type: S

[jira] [Updated] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25690: Issue Type: Sub-task (was: Bug) Parent: SPARK-25650 > Analyzer rule "HandleNullInputsForU

[jira] [Updated] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25690: Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-14220) > Analyzer rule "HandleNul

[jira] [Updated] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25690: Description: This was fixed in SPARK-24891 and was then broken by SPARK-25044. The tests added in

[jira] [Created] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

2018-10-09 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25690: --- Summary: Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely Key: SPARK-25690 URL: https://issues.apache.org/jira/browse/SPARK-25690

[jira] [Updated] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Description: *RECENT HISTORY* [http://spark-tests.appspot.com/test-details?suite_name=org.apa

[jira] [Updated] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Affects Version/s: 2.4.0 > Flaky test: FileBasedDataSourceSuite >

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643960#comment-16643960 ] Dongjoon Hyun commented on SPARK-23390: --- As reported in SPARK-25688 by [~smilegato

[jira] [Updated] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Priority: Critical (was: Major) > Flaky test: FileBasedDataSourceSuite >

[jira] [Updated] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Summary: Flaky test: FileBasedDataSourceSuite (was: Flaky test: FileBasedDataSourceSuite in S

[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext

2018-10-09 Thread Ankur Gupta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643936#comment-16643936 ] Ankur Gupta commented on SPARK-24523: - Hi [~umayr_nuna] - just checking if you had a

[jira] [Created] (SPARK-25689) Move token renewal logic to driver in yarn-client mode

2018-10-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25689: -- Summary: Move token renewal logic to driver in yarn-client mode Key: SPARK-25689 URL: https://issues.apache.org/jira/browse/SPARK-25689 Project: Spark Is

[jira] [Resolved] (SPARK-25688) Potential resource leak in ORC

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25688. --- Resolution: Duplicate Sorry, but let's gether the same error report into one JIRA. > Potent

[jira] [Commented] (SPARK-25688) Potential resource leak in ORC

2018-10-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643926#comment-16643926 ] Dongjoon Hyun commented on SPARK-25688: --- [~smilegator] . This is a duplication of 

[jira] [Commented] (SPARK-25688) Potential resource leak in ORC

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643910#comment-16643910 ] Xiao Li commented on SPARK-25688: - https://amplab.cs.berkeley.edu/jenkins/job/spark-mast

[jira] [Commented] (SPARK-25688) Potential resource leak in ORC

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643870#comment-16643870 ] Xiao Li commented on SPARK-25688: - It sounds like ORC still has a resource leak even aft

[jira] [Updated] (SPARK-25688) Potential resource leak in ORC

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25688: Summary: Potential resource leak in ORC (was: org.apache.spark.sql.FileBasedDataSourceSuite never pass)

[jira] [Updated] (SPARK-25688) org.apache.spark.sql.FileBasedDataSourceSuite never pass

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25688: Description: http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.FileBasedDataSour

[jira] [Updated] (SPARK-25688) org.apache.spark.sql.FileBasedDataSourceSuite never pass

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25688: Priority: Critical (was: Blocker) > org.apache.spark.sql.FileBasedDataSourceSuite never pass > --

[jira] [Created] (SPARK-25688) org.apache.spark.sql.FileBasedDataSourceSuite never pass

2018-10-09 Thread Xiao Li (JIRA)
Xiao Li created SPARK-25688: --- Summary: org.apache.spark.sql.FileBasedDataSourceSuite never pass Key: SPARK-25688 URL: https://issues.apache.org/jira/browse/SPARK-25688 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-25688) org.apache.spark.sql.FileBasedDataSourceSuite never pass

2018-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25688: Priority: Blocker (was: Major) > org.apache.spark.sql.FileBasedDataSourceSuite never pass > -

[jira] [Commented] (SPARK-19256) Hive bucketing support

2018-10-09 Thread Shay Elbaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643828#comment-16643828 ] Shay Elbaz commented on SPARK-19256: +1 [~tejasp] is this still under progress? >

[jira] [Commented] (SPARK-25493) CRLF Line Separators don't work in multiline CSVs

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643805#comment-16643805 ] Apache Spark commented on SPARK-25493: -- User 'justinuang' has created a pull reques

[jira] [Commented] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643765#comment-16643765 ] Apache Spark commented on SPARK-25559: -- User 'gatorsmile' has created a pull reques

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

2018-10-09 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643762#comment-16643762 ] Bryan Cutler commented on SPARK-25344: -- No I don't have strong feelings, my only pr

[jira] [Commented] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643761#comment-16643761 ] Apache Spark commented on SPARK-25559: -- User 'gatorsmile' has created a pull reques

[jira] [Created] (SPARK-25687) A dataset can store a column as sequence of Vectors but not directly vectors

2018-10-09 Thread Francisco Orchard (JIRA)
Francisco Orchard created SPARK-25687: - Summary: A dataset can store a column as sequence of Vectors but not directly vectors Key: SPARK-25687 URL: https://issues.apache.org/jira/browse/SPARK-25687

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

2018-10-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643663#comment-16643663 ] Imran Rashid commented on SPARK-25344: -- [~hyukjin.kwon] go for it! I dunno if [~br

[jira] [Resolved] (SPARK-25535) Work around bad error checking in commons-crypto

2018-10-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-25535. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22557 [https://gi

[jira] [Assigned] (SPARK-25535) Work around bad error checking in commons-crypto

2018-10-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-25535: Assignee: Marcelo Vanzin > Work around bad error checking in commons-crypto > ---

[jira] [Created] (SPARK-25686) date_trunc Spark SQL function silently returns null if parameters are swapped

2018-10-09 Thread Zack Behringer (JIRA)
Zack Behringer created SPARK-25686: -- Summary: date_trunc Spark SQL function silently returns null if parameters are swapped Key: SPARK-25686 URL: https://issues.apache.org/jira/browse/SPARK-25686 Pro

[jira] [Commented] (SPARK-22388) Limit push down

2018-10-09 Thread Aleksey Zinoviev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643478#comment-16643478 ] Aleksey Zinoviev commented on SPARK-22388: -- [~cloud_fan] Could you please add a

[jira] [Resolved] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24851. --- Resolution: Fixed Fix Version/s: 3.0.0 > Map a Stage ID to it's Associated Job ID in

[jira] [Assigned] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24851: - Assignee: Parth Gandhi > Map a Stage ID to it's Associated Job ID in UI > -

[jira] [Commented] (SPARK-18660) Parquet complains "Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImp

2018-10-09 Thread Paul Praet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643418#comment-16643418 ] Paul Praet commented on SPARK-18660: It's really polluting our logs. Any workaround

[jira] [Commented] (SPARK-20780) Spark Kafka10 Consumer Hangs

2018-10-09 Thread Mary Scott (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643235#comment-16643235 ] Mary Scott commented on SPARK-20780: Hello [~jayadeepj]. I am struggling with the id

[jira] [Updated] (SPARK-25685) Allow running tests in Jenkins in enterprise Git repository

2018-10-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25685: --- Summary: Allow running tests in Jenkins in enterprise Git repository (was: Allow regression testing

[jira] [Commented] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642966#comment-16642966 ] Apache Spark commented on SPARK-25685: -- User 'LantaoJin' has created a pull request

[jira] [Assigned] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25685: Assignee: Apache Spark > Allow regression testing in enterprise Jenkins > ---

[jira] [Assigned] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25685: Assignee: (was: Apache Spark) > Allow regression testing in enterprise Jenkins >

[jira] [Commented] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642963#comment-16642963 ] Apache Spark commented on SPARK-25685: -- User 'LantaoJin' has created a pull request

[jira] [Created] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25685: -- Summary: Allow regression testing in enterprise Jenkins Key: SPARK-25685 URL: https://issues.apache.org/jira/browse/SPARK-25685 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-25683) Make AsyncEventQueue.lastReportTimestamp inital value as the currentTime instead of 0

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25683: Assignee: (was: Apache Spark) > Make AsyncEventQueue.lastReportTimestamp inital value

[jira] [Assigned] (SPARK-25683) Make AsyncEventQueue.lastReportTimestamp inital value as the currentTime instead of 0

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25683: Assignee: Apache Spark > Make AsyncEventQueue.lastReportTimestamp inital value as the cur

[jira] [Commented] (SPARK-25683) Make AsyncEventQueue.lastReportTimestamp inital value as the currentTime instead of 0

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642936#comment-16642936 ] Apache Spark commented on SPARK-25683: -- User 'shivusondur' has created a pull reque

[jira] [Commented] (SPARK-25684) Organize header related codes in CSV datasource

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642922#comment-16642922 ] Apache Spark commented on SPARK-25684: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-25684) Organize header related codes in CSV datasource

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25684: Assignee: Hyukjin Kwon (was: Apache Spark) > Organize header related codes in CSV dataso

[jira] [Assigned] (SPARK-25684) Organize header related codes in CSV datasource

2018-10-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25684: Assignee: Apache Spark (was: Hyukjin Kwon) > Organize header related codes in CSV dataso

[jira] [Resolved] (SPARK-25497) limit operation within whole stage codegen should not consume all the inputs

2018-10-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-25497. -- Resolution: Fixed Fix Version/s: 3.0.0 > limit operation within whole stage cod

[jira] [Assigned] (SPARK-25497) limit operation within whole stage codegen should not consume all the inputs

2018-10-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reassigned SPARK-25497: Assignee: Wenchen Fan > limit operation within whole stage codegen should not con

[jira] [Created] (SPARK-25684) Organize header related codes in CSV datasource

2018-10-09 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-25684: Summary: Organize header related codes in CSV datasource Key: SPARK-25684 URL: https://issues.apache.org/jira/browse/SPARK-25684 Project: Spark Issue Type: I