[jira] [Created] (SPARK-29458) Document scalar functions usage in APIs in SQL getting started.

2019-10-13 Thread Dilip Biswal (Jira)
Dilip Biswal created SPARK-29458: Summary: Document scalar functions usage in APIs in SQL getting started. Key: SPARK-29458 URL: https://issues.apache.org/jira/browse/SPARK-29458 Project: Spark

[jira] [Commented] (SPARK-29458) Document scalar functions usage in APIs in SQL getting started.

2019-10-13 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950727#comment-16950727 ] Dilip Biswal commented on SPARK-29458: -- I will take a look at this one. > Document scalar

[jira] [Commented] (SPARK-29457) Improve tooltip information for Environment Tab

2019-10-13 Thread pavithra ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950729#comment-16950729 ] pavithra ramachandran commented on SPARK-29457: --- i ll work on this > Improve tooltip

[jira] [Created] (SPARK-29457) Improve tooltip information for Environment Tab

2019-10-13 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29457: Summary: Improve tooltip information for Environment Tab Key: SPARK-29457 URL: https://issues.apache.org/jira/browse/SPARK-29457 Project: Spark

[jira] [Commented] (SPARK-29456) Add tooltip information for Session Statistics Table column in JDBC/ODBC Server Tab

2019-10-13 Thread pavithra ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950720#comment-16950720 ] pavithra ramachandran commented on SPARK-29456: --- i will work on this > Add tooltip

[jira] [Created] (SPARK-29456) Add tooltip information for Session Statistics Table column in JDBC/ODBC Server Tab

2019-10-13 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29456: Summary: Add tooltip information for Session Statistics Table column in JDBC/ODBC Server Tab Key: SPARK-29456 URL:

[jira] [Commented] (SPARK-29455) Improve tooltip information for Stages Tab

2019-10-13 Thread Sharanabasappa G Keriwaddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950718#comment-16950718 ] Sharanabasappa G Keriwaddi commented on SPARK-29455: I will work on this > Improve

[jira] [Created] (SPARK-29455) Improve tooltip information for Stages Tab

2019-10-13 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29455: Summary: Improve tooltip information for Stages Tab Key: SPARK-29455 URL: https://issues.apache.org/jira/browse/SPARK-29455 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-29453) Improve tooltip information for SQL tab

2019-10-13 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-29453: -- Comment: was deleted (was: working on this will raise PR soon) > Improve tooltip information

[jira] [Commented] (SPARK-29453) Improve tooltip information for SQL tab

2019-10-13 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950714#comment-16950714 ] Ankit Raj Boudh commented on SPARK-29453: - Started to work on a fix for this. > Improve tooltip

[jira] [Updated] (SPARK-29454) Reduce unsafeProjection call times when read parquet file

2019-10-13 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-29454: - Summary: Reduce unsafeProjection call times when read parquet file (was: Reduce one unsafeProjection

[jira] [Updated] (SPARK-29454) Reduce one unsafeProjection call when read parquet file

2019-10-13 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-29454: - Description: ParquetGroupConverter call unsafeProjection function to covert SpecificInternalRow to

[jira] [Created] (SPARK-29454) Reduce one unsafeProjection call when read parquet file

2019-10-13 Thread Yang Jie (Jira)
Yang Jie created SPARK-29454: Summary: Reduce one unsafeProjection call when read parquet file Key: SPARK-29454 URL: https://issues.apache.org/jira/browse/SPARK-29454 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950686#comment-16950686 ] Sean R. Owen commented on SPARK-26570: -- That doesn't help much, especially as it's a screen shot?

[jira] [Comment Edited] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950658#comment-16950658 ] fengchaoge edited comment on SPARK-26570 at 10/14/19 3:00 AM: -- hello,

[jira] [Updated] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-26570: --- Attachment: image-2019-10-14-10-51-28-374.png > Out of memory when InMemoryFileIndex

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950675#comment-16950675 ] fengchaoge commented on SPARK-26570: !image-2019-10-14-10-51-28-374.png! > Out of memory when

[jira] [Comment Edited] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950672#comment-16950672 ] fengchaoge edited comment on SPARK-26570 at 10/14/19 2:50 AM: --

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950672#comment-16950672 ] fengchaoge commented on SPARK-26570: !image-2019-10-14-10-48-05-063.png!

[jira] [Comment Edited] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950658#comment-16950658 ] fengchaoge edited comment on SPARK-26570 at 10/14/19 2:47 AM: -- hello,

[jira] [Updated] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-26570: --- Attachment: image-2019-10-14-10-47-47-684.png > Out of memory when InMemoryFileIndex

[jira] [Comment Edited] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950658#comment-16950658 ] fengchaoge edited comment on SPARK-26570 at 10/14/19 2:32 AM: -- hello,

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950658#comment-16950658 ] fengchaoge commented on SPARK-26570: hello, [~srowen] [~viirya] jmap logs is shown below:

[jira] [Updated] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-26570: --- Attachment: image-2019-10-14-10-00-27-361.png > Out of memory when InMemoryFileIndex

[jira] [Resolved] (SPARK-29361) Enable DataFrame with streaming source support on DSv1

2019-10-13 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29361. -- Resolution: Invalid Closing this, as there was some explanation that DSv2 is the first one

[jira] [Resolved] (SPARK-29129) Test failure: org.apache.spark.sql.hive.JavaDataFrameSuite (hadoop-2.7/JDK 11 combination)

2019-10-13 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29129. -- Resolution: Invalid We don't support hadoop-2.7 & jdk 11 profile together. Resolving this as

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2019-10-13 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950461#comment-16950461 ] Maxim Gekk commented on SPARK-10848: Nullable = false in user's schema cannot guarantee that nulls

[jira] [Assigned] (SPARK-29308) dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect

2019-10-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29308: Assignee: angerszhu > dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect >

[jira] [Resolved] (SPARK-29308) dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect

2019-10-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29308. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25984

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950406#comment-16950406 ] L. C. Hsieh commented on SPARK-26570: - I think the collected number of filestatus can be more than

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950370#comment-16950370 ] Sean R. Owen commented on SPARK-26570: -- Not sure, those objects aren't large, and 70K of them

[jira] [Commented] (SPARK-29453) Improve tooltip information for SQL tab

2019-10-13 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950364#comment-16950364 ] Sandeep Katta commented on SPARK-29453: --- working on this will raise PR soon > Improve tooltip

[jira] [Created] (SPARK-29453) Improve tooltip information for SQL tab

2019-10-13 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-29453: - Summary: Improve tooltip information for SQL tab Key: SPARK-29453 URL: https://issues.apache.org/jira/browse/SPARK-29453 Project: Spark Issue Type:

[jira] [Created] (SPARK-29452) Improve tootip information for storage tab

2019-10-13 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-29452: - Summary: Improve tootip information for storage tab Key: SPARK-29452 URL: https://issues.apache.org/jira/browse/SPARK-29452 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29452) Improve tootip information for storage tab

2019-10-13 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950344#comment-16950344 ] Sandeep Katta commented on SPARK-29452: --- soon I will raise PR for this  > Improve tootip

[jira] [Reopened] (SPARK-25065) Driver and executors pick the wrong logging configuration file.

2019-10-13 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reopened SPARK-25065: - > Driver and executors pick the wrong logging configuration file. >

[jira] [Updated] (SPARK-25065) Driver and executors pick the wrong logging configuration file.

2019-10-13 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25065: Affects Version/s: (was: 2.3.1) 3.0.0

[jira] [Commented] (SPARK-29449) Add tooltip to Spark WebUI

2019-10-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950324#comment-16950324 ] Yuming Wang commented on SPARK-29449: - cc [~planga82] [~sandeep.katta2007] > Add tooltip to Spark

[jira] [Commented] (SPARK-28867) InMemoryStore checkpoint to speed up replay log file in HistoryServer

2019-10-13 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950307#comment-16950307 ] wuyi commented on SPARK-28867: -- [~irashid] Thanks for putting this as a related issue to  SPARK-20656.  

[jira] [Created] (SPARK-29451) Some queries with divisions in windows are failling in Thrift

2019-10-13 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29451: Summary: Some queries with divisions in windows are failling in Thrift Key: SPARK-29451 URL: https://issues.apache.org/jira/browse/SPARK-29451 Project: Spark

[jira] [Updated] (SPARK-29451) Some queries with divisions in SQL windows are failling in Thrift

2019-10-13 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29451: - Summary: Some queries with divisions in SQL windows are failling in Thrift (was: Some queries

[jira] [Comment Edited] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950276#comment-16950276 ] fengchaoge edited comment on SPARK-26570 at 10/13/19 10:46 AM: --- hello,

[jira] [Updated] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-26570: --- Attachment: image-2019-10-13-18-45-33-770.png > Out of memory when InMemoryFileIndex

[jira] [Commented] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950276#comment-16950276 ] fengchaoge commented on SPARK-26570: hello, [~deshanxiao] [~srowen] spark2.4.3 may also have the

[jira] [Updated] (SPARK-26570) Out of memory when InMemoryFileIndex bulkListLeafFiles

2019-10-13 Thread fengchaoge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-26570: --- Attachment: image-2019-10-13-18-41-22-090.png > Out of memory when InMemoryFileIndex

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-13 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950261#comment-16950261 ] Sandeep Katta commented on SPARK-29435: --- [~XuanYuan] thank you for your comments, I will update

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-13 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950258#comment-16950258 ] Yuanjian Li commented on SPARK-29435: - Great thanks for the reporting and fix by [~koert] and

[jira] [Created] (SPARK-29450) [SS] In streaming aggregation, metric for output rows is not measured in append mode

2019-10-13 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29450: Summary: [SS] In streaming aggregation, metric for output rows is not measured in append mode Key: SPARK-29450 URL: https://issues.apache.org/jira/browse/SPARK-29450