[jira] [Commented] (SPARK-24543) Support any DataType as DDL string for from_json's schema

2018-06-13 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510753#comment-16510753 ] Maxim Gekk commented on SPARK-24543: I am working on the feature at the moment. > S

[jira] [Created] (SPARK-24543) Support any DataType as DDL string for from_json's schema

2018-06-13 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24543: -- Summary: Support any DataType as DDL string for from_json's schema Key: SPARK-24543 URL: https://issues.apache.org/jira/browse/SPARK-24543 Project: Spark Issue T

[jira] [Assigned] (SPARK-24543) Support any DataType as DDL string for from_json's schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24543: Assignee: Apache Spark > Support any DataType as DDL string for from_json's schema >

[jira] [Assigned] (SPARK-24543) Support any DataType as DDL string for from_json's schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24543: Assignee: (was: Apache Spark) > Support any DataType as DDL string for from_json's sc

[jira] [Commented] (SPARK-24543) Support any DataType as DDL string for from_json's schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510769#comment-16510769 ] Apache Spark commented on SPARK-24543: -- User 'MaxGekk' has created a pull request f

[jira] [Updated] (SPARK-24514) Exception while converting RDD to DataFrame

2018-06-13 Thread SHAILENDRA SHAHANE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHAILENDRA SHAHANE updated SPARK-24514: --- Attachment: MongoSparkTest.java > Exception while converting RDD to DataFrame >

[jira] [Updated] (SPARK-24514) Exception while converting RDD to DataFrame

2018-06-13 Thread SHAILENDRA SHAHANE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHAILENDRA SHAHANE updated SPARK-24514: --- Attachment: SparkMongoException.txt > Exception while converting RDD to DataFrame >

[jira] [Commented] (SPARK-24514) Exception while converting RDD to DataFrame

2018-06-13 Thread SHAILENDRA SHAHANE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510875#comment-16510875 ] SHAILENDRA SHAHANE commented on SPARK-24514: Attached Source code and Except

[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once

2018-06-13 Thread LANDAIS Christophe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510917#comment-16510917 ] LANDAIS Christophe commented on SPARK-23486: Hi,   What is the process to

[jira] [Created] (SPARK-24544) Print actual failure cause when look up function failed

2018-06-13 Thread zhoukang (JIRA)
zhoukang created SPARK-24544: Summary: Print actual failure cause when look up function failed Key: SPARK-24544 URL: https://issues.apache.org/jira/browse/SPARK-24544 Project: Spark Issue Type: I

[jira] [Assigned] (SPARK-24544) Print actual failure cause when look up function failed

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24544: Assignee: Apache Spark > Print actual failure cause when look up function failed > --

[jira] [Assigned] (SPARK-24544) Print actual failure cause when look up function failed

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24544: Assignee: (was: Apache Spark) > Print actual failure cause when look up function fail

[jira] [Commented] (SPARK-24544) Print actual failure cause when look up function failed

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510933#comment-16510933 ] Apache Spark commented on SPARK-24544: -- User 'caneGuy' has created a pull request f

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Summary: Function hour not working as expected for hour 2 (was: Function hour not working as expe

[jira] [Created] (SPARK-24545) Function hour not working as expecte for hour 2

2018-06-13 Thread Eric Blanco (JIRA)
Eric Blanco created SPARK-24545: --- Summary: Function hour not working as expecte for hour 2 Key: SPARK-24545 URL: https://issues.apache.org/jira/browse/SPARK-24545 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-24546) InsertIntoDataSourceCommand make dataframe with wrong schema

2018-06-13 Thread yangz (JIRA)
yangz created SPARK-24546: - Summary: InsertIntoDataSourceCommand make dataframe with wrong schema Key: SPARK-24546 URL: https://issues.apache.org/jira/browse/SPARK-24546 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Priority: Minor (was: Major) > Function hour not working as expected for hour 2 in PySpark >

[jira] [Updated] (SPARK-24546) InsertIntoDataSourceCommand make dataframe with wrong schema

2018-06-13 Thread yangz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangz updated SPARK-24546: -- Description: I have a hdfs table with schema  {code:java} hdfs_table(a int, b int, c int){code}   and a kudu

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Commented] (SPARK-24215) Implement __repr__ and _repr_html_ for dataframes in PySpark

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510946#comment-16510946 ] Apache Spark commented on SPARK-24215: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-24546) InsertIntoDataSourceCommand make dataframe with wrong schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24546: Assignee: (was: Apache Spark) > InsertIntoDataSourceCommand make dataframe with wrong

[jira] [Commented] (SPARK-24546) InsertIntoDataSourceCommand make dataframe with wrong schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510957#comment-16510957 ] Apache Spark commented on SPARK-24546: -- User 'zheh12' has created a pull request fo

[jira] [Assigned] (SPARK-24546) InsertIntoDataSourceCommand make dataframe with wrong schema

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24546: Assignee: Apache Spark > InsertIntoDataSourceCommand make dataframe with wrong schema > -

[jira] [Created] (SPARK-24547) Spark on K8s

2018-06-13 Thread Ray Burgemeestre (JIRA)
Ray Burgemeestre created SPARK-24547: Summary: Spark on K8s Key: SPARK-24547 URL: https://issues.apache.org/jira/browse/SPARK-24547 Project: Spark Issue Type: Improvement Compo

[jira] [Updated] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

2018-06-13 Thread Ray Burgemeestre (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Burgemeestre updated SPARK-24547: - Summary: Spark on K8s docker-image-tool.sh improvements (was: Spark on K8s ) > Spark o

[jira] [Updated] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

2018-06-13 Thread Ray Burgemeestre (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Burgemeestre updated SPARK-24547: - Description: *Context* PySpark support for Spark on k8s was merged with [https://githu

[jira] [Commented] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510971#comment-16510971 ] Apache Spark commented on SPARK-24547: -- User 'rayburgemeestre' has created a pull r

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24547: Assignee: Apache Spark > Spark on K8s docker-image-tool.sh improvements > ---

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24547: Assignee: (was: Apache Spark) > Spark on K8s docker-image-tool.sh improvements >

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Description: Hello, I tried to get the hour out of a date and it works except if the hour is 2. I

[jira] [Created] (SPARK-24548) JavaPairRDD to Dataset in SPARK returns error

2018-06-13 Thread Jackson (JIRA)
Jackson created SPARK-24548: --- Summary: JavaPairRDD to Dataset in SPARK returns error Key: SPARK-24548 URL: https://issues.apache.org/jira/browse/SPARK-24548 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510999#comment-16510999 ] Eric Blanco commented on SPARK-24545: - Ok apparenlty this is due to a change of hour

[jira] [Resolved] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco resolved SPARK-24545. - Resolution: Not A Bug > Function hour not working as expected for hour 2 in PySpark > --

[jira] [Updated] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

2018-06-13 Thread Jackson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jackson updated SPARK-24548: Component/s: Java API Summary: JavaPairRDD to Dataset in SPARK generates ambiguous results (was:

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Attachment: image-2018-06-13-13-52-06-165.png > Function hour not working as expected for hour 2 i

[jira] [Commented] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511003#comment-16511003 ] Eric Blanco commented on SPARK-24545: - !image-2018-06-13-13-52-06-165.png! In Scala

[jira] [Updated] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco updated SPARK-24545: Attachment: image-2018-06-13-13-53-21-185.png > Function hour not working as expected for hour 2 i

[jira] [Reopened] (SPARK-24545) Function hour not working as expected for hour 2 in PySpark

2018-06-13 Thread Eric Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Blanco reopened SPARK-24545: - > Function hour not working as expected for hour 2 in PySpark >

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2018-06-13 Thread Sina Madani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511033#comment-16511033 ] Sina Madani commented on SPARK-650: --- I too have this problem. It seems that Apache Flink

[jira] [Updated] (SPARK-24538) ByteArrayDecimalType support push down to the data sources

2018-06-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24538: Summary: ByteArrayDecimalType support push down to the data sources (was: Decimal type support pu

[jira] [Commented] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511034#comment-16511034 ] Yuming Wang commented on SPARK-24549: - I'm working on this > 32BitDecimalType and 6

[jira] [Created] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-24549: --- Summary: 32BitDecimalType and 64BitDecimalType support push down to the data sources Key: SPARK-24549 URL: https://issues.apache.org/jira/browse/SPARK-24549 Project: Sp

[jira] [Commented] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511049#comment-16511049 ] Apache Spark commented on SPARK-24549: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24549: Assignee: Apache Spark > 32BitDecimalType and 64BitDecimalType support push down to the d

[jira] [Assigned] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24549: Assignee: (was: Apache Spark) > 32BitDecimalType and 64BitDecimalType support push do

[jira] [Issue Comment Deleted] (SPARK-24549) 32BitDecimalType and 64BitDecimalType support push down to the data sources

2018-06-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24549: Comment: was deleted (was: I'm working on this) > 32BitDecimalType and 64BitDecimalType support p

[jira] [Commented] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

2018-06-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511056#comment-16511056 ] Tomasz Gawęda commented on SPARK-24548: --- IMHO names should be distinct, in other c

[jira] [Resolved] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24479. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21504 [https://gi

[jira] [Assigned] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24479: Assignee: Arun Mahadevan > Register StreamingQueryListener in Spark Conf > -

[jira] [Created] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-24550: --- Summary: Registration of K8s specific metrics Key: SPARK-24550 URL: https://issues.apache.org/jira/browse/SPARK-24550 Project: Spark Issue Type

[jira] [Updated] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Description: Spark by default offers a specific set of metrics for monitoring. It

[jira] [Updated] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Description: Spark by default offers a specific set of metrics for monitoring. It

[jira] [Updated] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Description: Spark by default offers a specific set of metrics for monitoring. It

[jira] [Updated] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Issue Type: New Feature (was: Bug) > Registration of K8s specific metrics > -

[jira] [Updated] (SPARK-24550) Registration of K8s specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Description: Spark by default offers a specific set of metrics for monitoring. It

[jira] [Updated] (SPARK-24550) Add support for Kubernetes specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Summary: Add support for Kubernetes specific metrics (was: Registration of Kubern

[jira] [Updated] (SPARK-24550) Registration of Kubernetes specific metrics

2018-06-13 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24550: Summary: Registration of Kubernetes specific metrics (was: Registration of K8s sp

[jira] [Created] (SPARK-24551) Add Integration tests for Secrets

2018-06-13 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-24551: --- Summary: Add Integration tests for Secrets Key: SPARK-24551 URL: https://issues.apache.org/jira/browse/SPARK-24551 Project: Spark Issue Type: I

[jira] [Resolved] (SPARK-24500) UnsupportedOperationException when trying to execute Union plan with Stream of children

2018-06-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24500. - Resolution: Fixed Fix Version/s: 2.4.0 > UnsupportedOperationException when trying to exe

[jira] [Issue Comment Deleted] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

2018-06-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Gawęda updated SPARK-24548: -- Comment: was deleted (was: IMHO names should be distinct, in other cases it's hard to query fo

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

2018-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511365#comment-16511365 ] Marcelo Vanzin commented on SPARK-24539: >From a previous chat with [~tgraves] t

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511452#comment-16511452 ] Wenchen Fan commented on SPARK-24528: - It's a different problem. Spark makes a trad

[jira] [Updated] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-13 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-24439: --- Priority: Minor (was: Major) > Add distanceMeasure to BisectingKMeans in PySpark >

[jira] [Assigned] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24439: Assignee: (was: Apache Spark) > Add distanceMeasure to BisectingKMeans in PySpark > -

[jira] [Commented] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511482#comment-16511482 ] Apache Spark commented on SPARK-24439: -- User 'huaxingao' has created a pull request

[jira] [Assigned] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24439: Assignee: Apache Spark > Add distanceMeasure to BisectingKMeans in PySpark >

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-13 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511483#comment-16511483 ] Ohad Raviv commented on SPARK-24528: I understand the tradeoff, the question is how

[jira] [Commented] (SPARK-22239) User-defined window functions with pandas udf

2018-06-13 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511495#comment-16511495 ] Li Jin commented on SPARK-22239: [~hyukjin.kwon] I actually don't think this Jira is don

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511533#comment-16511533 ] Wenchen Fan commented on SPARK-24528: - I have 2 ideas: 1. provide an option to let S

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-13 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511578#comment-16511578 ] Ohad Raviv commented on SPARK-24528: I think the 2nd point better suits my usecase.

[jira] [Updated] (SPARK-24552) Task attempt ids are reused when stages are retried

2018-06-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-24552: -- Description: When stages are retried due to shuffle failures, task attempt ids are reused. This cause

[jira] [Created] (SPARK-24552) Task attempt ids are reused when stages are retried

2018-06-13 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-24552: - Summary: Task attempt ids are reused when stages are retried Key: SPARK-24552 URL: https://issues.apache.org/jira/browse/SPARK-24552 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-24552) Task attempt ids are reused when stages are retried

2018-06-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-24552: -- Description: When stages are retried due to shuffle failures, task attempt numbers are reused. This c

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-24552: -- Summary: Task attempt numbers are reused when stages are retried (was: Task attempt ids are reused wh

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511590#comment-16511590 ] Ryan Blue commented on SPARK-24552: --- cc [~vanzin], [~henryr], [~cloud_fan] > Task att

[jira] [Assigned] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24552: Assignee: (was: Apache Spark) > Task attempt numbers are reused when stages are retri

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511593#comment-16511593 ] Apache Spark commented on SPARK-24552: -- User 'rdblue' has created a pull request fo

[jira] [Assigned] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24552: Assignee: Apache Spark > Task attempt numbers are reused when stages are retried > --

[jira] [Resolved] (SPARK-24235) create the top-of-task RDD sending rows to the remote buffer

2018-06-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24235. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21428 [https://gi

[jira] [Assigned] (SPARK-24235) create the top-of-task RDD sending rows to the remote buffer

2018-06-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24235: Assignee: Jose Torres > create the top-of-task RDD sending rows to the remote buffer > --

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-06-13 Thread Ankur Gupta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511618#comment-16511618 ] Ankur Gupta commented on SPARK-24415: - I am planning to work on this JIRA > Stage p

[jira] [Created] (SPARK-24553) Job UI redirect causing http 302 error

2018-06-13 Thread Steven Kallman (JIRA)
Steven Kallman created SPARK-24553: -- Summary: Job UI redirect causing http 302 error Key: SPARK-24553 URL: https://issues.apache.org/jira/browse/SPARK-24553 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-24525) Provide an option to limit MemorySink memory usage

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24525: Assignee: Apache Spark > Provide an option to limit MemorySink memory usage > ---

[jira] [Commented] (SPARK-24525) Provide an option to limit MemorySink memory usage

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511655#comment-16511655 ] Apache Spark commented on SPARK-24525: -- User 'mukulmurthy' has created a pull reque

[jira] [Assigned] (SPARK-24525) Provide an option to limit MemorySink memory usage

2018-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24525: Assignee: (was: Apache Spark) > Provide an option to limit MemorySink memory usage >

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511674#comment-16511674 ] Imran Rashid commented on SPARK-24552: -- I wouldn't call this a bug in the scheduler

[jira] [Created] (SPARK-24554) Add MapType Support for Arrow in PySpark

2018-06-13 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-24554: Summary: Add MapType Support for Arrow in PySpark Key: SPARK-24554 URL: https://issues.apache.org/jira/browse/SPARK-24554 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-24554) Add MapType Support for Arrow in PySpark

2018-06-13 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511691#comment-16511691 ] Bryan Cutler commented on SPARK-24554: -- There still is work to be done to add a Map

[jira] [Updated] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-06-13 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-23874: - Description: Version 0.10.0 will allow for the following improvements and bug fixes: * Allow fo

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511695#comment-16511695 ] Jiang Xingbo commented on SPARK-24552: -- IIUC stageAttemptId + taskAttemptId shall p

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-13 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511695#comment-16511695 ] Jiang Xingbo edited comment on SPARK-24552 at 6/13/18 9:47 PM: ---

[jira] [Resolved] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24531. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > HiveExternalCatalogVersio

[jira] [Commented] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511730#comment-16511730 ] Dongjoon Hyun commented on SPARK-24530: --- Hi, [~mengxr] . I got the following loca

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24530: -- Description: I generated python docs from master locally using `make html`. However, the gene

[jira] [Commented] (SPARK-5152) Let metrics.properties file take an hdfs:// path

2018-06-13 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511724#comment-16511724 ] John Zhuge commented on SPARK-5152: --- SPARK-7169 alleviated this issue, however, still f

[jira] [Comment Edited] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511730#comment-16511730 ] Dongjoon Hyun edited comment on SPARK-24530 at 6/13/18 10:28 PM: -

  1   2   >