[jira] [Commented] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630672#comment-16630672 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] thanks so much for your hard work, let me look

[jira] [Commented] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629320#comment-16629320 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] Thanks for your reply. At first, what my PR

[jira] [Commented] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628119#comment-16628119 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] if you like, you can propose a PR based my PR

[jira] [Commented] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627934#comment-16627934 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] any updates on your PR, I can propose a PR as

[jira] [Created] (SPARK-25518) Spark kafka delegation token supported

Mingjie Tang created SPARK-25518: Summary: Spark kafka delegation token supported Key: SPARK-25518 URL: https://issues.apache.org/jira/browse/SPARK-25518 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567393#comment-16567393 ] Mingjie Tang commented on SPARK-24615: -- >From user's perspective, user only concern about the GPU

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519622#comment-16519622 ] Mingjie Tang commented on SPARK-24374: -- [~mengxr].  [~jerryshao] create a SISP [SPARK-24615] for

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508814#comment-16508814 ] Mingjie Tang commented on SPARK-24093: -- The related topic could be viewed when users want to look

[jira] [Created] (SPARK-24479) Register StreamingQueryListener in Spark Conf

Mingjie Tang created SPARK-24479: Summary: Register StreamingQueryListener in Spark Conf Key: SPARK-24479 URL: https://issues.apache.org/jira/browse/SPARK-24479 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494154#comment-16494154 ] Mingjie Tang commented on SPARK-24093: -- I made a PR for this

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491285#comment-16491285 ] Mingjie Tang commented on SPARK-24093: -- i can add a PR for this. > Make some fields of

[jira] [Created] (SPARK-23674) Add Spark ML Listener for Tracking ML Pipeline Status

Mingjie Tang created SPARK-23674: Summary: Add Spark ML Listener for Tracking ML Pipeline Status Key: SPARK-23674 URL: https://issues.apache.org/jira/browse/SPARK-23674 Project: Spark Issue

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284485#comment-16284485 ] Mingjie Tang commented on SPARK-18673: -- Hi Steve, the hive side is updated.

[jira] [Comment Edited] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273745#comment-16273745 ] Mingjie Tang edited comment on SPARK-22587 at 12/5/17 12:01 AM: we can

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273745#comment-16273745 ] Mingjie Tang commented on SPARK-22587: -- we can update the compareFS by considering the authority.

[jira] [Commented] (SPARK-19771) Support OR-AND amplification in Locality Sensitive Hashing (LSH)

[ https://issues.apache.org/jira/browse/SPARK-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893117#comment-15893117 ] Mingjie Tang commented on SPARK-19771: -- (1) because you need to explode each tuple. For example

[jira] [Commented] (SPARK-18454) Changes to improve Nearest Neighbor Search for LSH

[ https://issues.apache.org/jira/browse/SPARK-18454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893087#comment-15893087 ] Mingjie Tang commented on SPARK-18454: -- [~yunn] the current multi-probe NNS can be improved without

[jira] [Comment Edited] (SPARK-19771) Support OR-AND amplification in Locality Sensitive Hashing (LSH)

[ https://issues.apache.org/jira/browse/SPARK-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889180#comment-15889180 ] Mingjie Tang edited comment on SPARK-19771 at 3/1/17 12:30 AM: --- If we

[jira] [Commented] (SPARK-19771) Support OR-AND amplification in Locality Sensitive Hashing (LSH)

[ https://issues.apache.org/jira/browse/SPARK-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889180#comment-15889180 ] Mingjie Tang commented on SPARK-19771: -- If we follow the AND-OR framework, one optimization is here:

[jira] [Commented] (SPARK-18454) Changes to improve Nearest Neighbor Search for LSH

[ https://issues.apache.org/jira/browse/SPARK-18454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879539#comment-15879539 ] Mingjie Tang commented on SPARK-18454: -- [~yunn] I leave some comments on the document. the build

[jira] [Commented] (SPARK-18392) LSH API, algorithm, and documentation follow-ups

[ https://issues.apache.org/jira/browse/SPARK-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867064#comment-15867064 ] Mingjie Tang commented on SPARK-18392: -- Sure, AND-amp is important and basic for current LSH. We can

[jira] [Commented] (SPARK-18392) LSH API, algorithm, and documentation follow-ups

[ https://issues.apache.org/jira/browse/SPARK-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867041#comment-15867041 ] mingjie tang commented on SPARK-18392: -- [~yunn] are you working on the BitSampling &

[jira] [Updated] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingjie tang updated SPARK-18372: - Fix Version/s: 2.0.2 > .Hive-staging folders created from Spark hiveContext are not getting

[jira] [Updated] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingjie tang updated SPARK-18372: - Attachment: _thumb_37664.png the staging directory fail to be removed when hive table in the

[jira] [Commented] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649935#comment-15649935 ] mingjie tang commented on SPARK-18372: -- This bug can be reproduced by the following codes: val

[jira] [Commented] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649397#comment-15649397 ] mingjie tang commented on SPARK-18372: -- Solution: This bug is reported by customers. The reason is

[jira] [Updated] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingjie tang updated SPARK-18372: - Description: Steps to reproduce: 1. Launch spark-shell 2. Run the following

[jira] [Commented] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

[ https://issues.apache.org/jira/browse/SPARK-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649386#comment-15649386 ] mingjie tang commented on SPARK-18372: -- the PR is https://github.com/apache/spark/pull/15819 >

[jira] [Created] (SPARK-18372) .Hive-staging folders created from Spark hiveContext are not getting cleaned up

mingjie tang created SPARK-18372: Summary: .Hive-staging folders created from Spark hiveContext are not getting cleaned up Key: SPARK-18372 URL: https://issues.apache.org/jira/browse/SPARK-18372