[jira] [Commented] (SPARK-2090) spark-shell input text entry not showing on REPL

2016-02-17 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151802#comment-15151802 ] Lantao Jin commented on SPARK-2090: --- Richard is right, this is the permissions problem on home

[jira] [Comment Edited] (SPARK-2090) spark-shell input text entry not showing on REPL

2016-02-17 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151802#comment-15151802 ] Lantao Jin edited comment on SPARK-2090 at 2/18/16 6:35 AM: Richard is right,

[jira] [Comment Edited] (SPARK-2090) spark-shell input text entry not showing on REPL

2016-02-17 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151802#comment-15151802 ] Lantao Jin edited comment on SPARK-2090 at 2/18/16 6:36 AM: Richard is right,

[jira] [Commented] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read from directory failed

2016-11-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656317#comment-15656317 ] Lantao Jin commented on SPARK-18227: [~marmbrus], oh, yeah. That's the root cause, thanks. > Parquet

[jira] [Closed] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read from directory failed

2016-11-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin closed SPARK-18227. -- > Parquet file stream sink create a hidden directory "_spark_metadata" cause > the DataFrame read from

[jira] [Commented] (SPARK-8560) Executors page displays negative active tasks

2016-11-23 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692346#comment-15692346 ] Lantao Jin commented on SPARK-8560: --- It still displays negative active tasks in UI of Spark-2.0.1 >

[jira] [Commented] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed

2016-11-02 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629047#comment-15629047 ] Lantao Jin commented on SPARK-18227: I also have an idea that add a "spark.parquet.metadata.dir" or

[jira] [Created] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed

2016-11-02 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-18227: -- Summary: Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed Key: SPARK-18227 URL: https://issues.apache.org/jira/browse/SPARK-18227

[jira] [Comment Edited] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed

2016-11-02 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628997#comment-15628997 ] Lantao Jin edited comment on SPARK-18227 at 11/2/16 1:39 PM: - hadoop fs -ls

[jira] [Updated] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed

2016-11-02 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-18227: --- Description: When we set an out directory as a streaming sink with parquet format in structured

[jira] [Commented] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read failed

2016-11-02 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628997#comment-15628997 ] Lantao Jin commented on SPARK-18227: hadoop fs -ls hdfs:///path/out Found 3 items -rw-r--r-- 3 hdfs

[jira] [Updated] (SPARK-18227) Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame read from directory failed

2016-11-02 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-18227: --- Summary: Parquet file stream sink create a hidden directory "_spark_metadata" cause the DataFrame

[jira] [Commented] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-03-24 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939897#comment-15939897 ] Lantao Jin commented on SPARK-19995: Great patch! I meet this issue recently. And it's looks good to

[jira] [Commented] (SPARK-20680) Spark-sql do not support for void column datatype of view

2017-05-11 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006314#comment-16006314 ] Lantao Jin commented on SPARK-20680: Hi [~jiangxb1987] and [~hvanhovell], I fixed with a very simple

[jira] [Comment Edited] (SPARK-20680) Spark-sql do not support for void column datatype of view

2017-05-11 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006314#comment-16006314 ] Lantao Jin edited comment on SPARK-20680 at 5/11/17 12:12 PM: -- Hi

[jira] [Comment Edited] (SPARK-20680) Spark-sql do not support for void column datatype of view

2017-05-11 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006314#comment-16006314 ] Lantao Jin edited comment on SPARK-20680 at 5/11/17 12:12 PM: -- Hi

[jira] [Updated] (SPARK-20680) Spark-sql do not support for void column datatype of view

2017-05-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-20680: --- Summary: Spark-sql do not support for void column datatype of view (was: Spark-sql do not support

[jira] [Created] (SPARK-20680) Spark-sql do not support for column datatype of void in a HIVE view

2017-05-09 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-20680: -- Summary: Spark-sql do not support for column datatype of void in a HIVE view Key: SPARK-20680 URL: https://issues.apache.org/jira/browse/SPARK-20680 Project: Spark

[jira] [Created] (SPARK-20816) MetricsConfig doen't trim the properties file cause the exception very confused

2017-05-19 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-20816: -- Summary: MetricsConfig doen't trim the properties file cause the exception very confused Key: SPARK-20816 URL: https://issues.apache.org/jira/browse/SPARK-20816

[jira] [Commented] (SPARK-20816) MetricsConfig doen't trim the properties file cause the exception very confused

2017-05-20 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018500#comment-16018500 ] Lantao Jin commented on SPARK-20816: So, how about just trim the property of class name when reflect?

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043160#comment-16043160 ] Lantao Jin edited comment on SPARK-21023 at 6/8/17 6:06 PM: The purpose is

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043160#comment-16043160 ] Lantao Jin commented on SPARK-21023: *The purpose is making the default configuration loaded

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043191#comment-16043191 ] Lantao Jin commented on SPARK-21023: I think {{--conf}} couldn't help this. Because from the view of

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043164#comment-16043164 ] Lantao Jin commented on SPARK-21023: {quote} Another option is to have a config option {quote} LGTM

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043172#comment-16043172 ] Lantao Jin commented on SPARK-21023: {quote} Another option is to have a config option {quote} Oh,

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043179#comment-16043179 ] Lantao Jin commented on SPARK-21023: {quote} it may break existing applications {quote} I really know

[jira] [Created] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-21023: -- Summary: Ignore to load default properties file is not a good choice from the perspective of system Key: SPARK-21023 URL: https://issues.apache.org/jira/browse/SPARK-21023

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043124#comment-16043124 ] Lantao Jin commented on SPARK-21023: [~vanzin] I suggest to change the current behavior and offer a

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043124#comment-16043124 ] Lantao Jin edited comment on SPARK-21023 at 6/8/17 5:45 PM: [~vanzin] I

[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045539#comment-16045539 ] Lantao Jin commented on SPARK-21048: Thank you for pointing out about JIRA. Confusing name can be

[jira] [Closed] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin closed SPARK-21023. -- Resolution: Not A Problem Close as NotAProblem and new solution see

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045452#comment-16045452 ] Lantao Jin commented on SPARK-21023: If current behavior should be keep. I can add an environment

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045452#comment-16045452 ] Lantao Jin edited comment on SPARK-21023 at 6/10/17 8:18 AM: - If current

[jira] [Updated] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-21023: --- Description: The default properties file {{spark-defaults.conf}} shouldn't be ignore to load even

[jira] [Created] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-21048: -- Summary: Add an option --merged-properties-file to distinguish the configuration loading behavior Key: SPARK-21048 URL: https://issues.apache.org/jira/browse/SPARK-21048

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045452#comment-16045452 ] Lantao Jin edited comment on SPARK-21023 at 6/10/17 8:16 AM: - If current

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045483#comment-16045483 ] Lantao Jin edited comment on SPARK-21023 at 6/10/17 10:15 AM: -- [~cloud_fan]

[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045502#comment-16045502 ] Lantao Jin commented on SPARK-21048: Will push a PR soon after a short discussing. > Add an option

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045441#comment-16045441 ] Lantao Jin commented on SPARK-21023: I modify the description with adding two tables to illustrate

[jira] [Updated] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-21023: --- Description: The default properties file {{spark-defaults.conf}} shouldn't be ignore to load even

[jira] [Updated] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-21023: --- Description: The default properties file {{spark-defaults.conf}} shouldn't be ignore to load even

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045483#comment-16045483 ] Lantao Jin commented on SPARK-21023: [~cloud_fan] {{--properties-file}} and

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045495#comment-16045495 ] Lantao Jin commented on SPARK-21023: [~cloud_fan], I come to see what you mean. Maybe add a

[jira] [Commented] (SPARK-20973) insert table fail caused by unable to fetch data definition file from remote hdfs

2017-09-19 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172627#comment-16172627 ] Lantao Jin commented on SPARK-20973: Please file a PR, refer to [How to contribute to

[jira] [Created] (SPARK-22079) Serializer in HiveOutputWriter miss loading job configuration

2017-09-20 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-22079: -- Summary: Serializer in HiveOutputWriter miss loading job configuration Key: SPARK-22079 URL: https://issues.apache.org/jira/browse/SPARK-22079 Project: Spark

[jira] [Created] (SPARK-22744) Add a configuration to show the application submit hostname

2017-12-08 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-22744: -- Summary: Add a configuration to show the application submit hostname Key: SPARK-22744 URL: https://issues.apache.org/jira/browse/SPARK-22744 Project: Spark

[jira] [Updated] (SPARK-22744) Add a configuration to show the application submit hostname

2017-12-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-22744: --- Description: In MapReduce, we can get the submit hostname via checking the value of configuration

[jira] [Updated] (SPARK-24292) Proxy user cannot connect to HiveMetastore in local mode

2018-05-16 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-24292: --- Component/s: Spark Core > Proxy user cannot connect to HiveMetastore in local mode >

[jira] [Created] (SPARK-24292) Proxy user cannot connect to HiveMetastore in local mode

2018-05-16 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-24292: -- Summary: Proxy user cannot connect to HiveMetastore in local mode Key: SPARK-24292 URL: https://issues.apache.org/jira/browse/SPARK-24292 Project: Spark Issue

[jira] [Commented] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2018-05-16 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477006#comment-16477006 ] Lantao Jin commented on SPARK-19995: [~rajeshhadoop], I filed a PR

[jira] [Issue Comment Deleted] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2018-05-16 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-19995: --- Comment: was deleted (was: [~rajeshhadoop], I filed a PR https://github.com/apache/spark/pull/21343

[jira] [Created] (SPARK-24349) obtainDelegationTokens() exits JVM if Driver use JDBC instead of using metastore

2018-05-22 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-24349: -- Summary: obtainDelegationTokens() exits JVM if Driver use JDBC instead of using metastore Key: SPARK-24349 URL: https://issues.apache.org/jira/browse/SPARK-24349

[jira] [Resolved] (SPARK-24292) Proxy user cannot connect to HiveMetastore in local mode

2018-05-21 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-24292. Resolution: Duplicate > Proxy user cannot connect to HiveMetastore in local mode >

[jira] [Closed] (SPARK-24349) obtainDelegationTokens() exits JVM if Driver use JDBC instead of using metastore

2018-05-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin closed SPARK-24349. -- > obtainDelegationTokens() exits JVM if Driver use JDBC instead of using > metastore >

[jira] [Resolved] (SPARK-24349) obtainDelegationTokens() exits JVM if Driver use JDBC instead of using metastore

2018-05-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-24349. Resolution: Not A Problem delegationTokensRequired has been checked in SparkSQLCLIDriver.scala >

[jira] [Updated] (SPARK-24349) obtainDelegationTokens() exits JVM if Driver use JDBC instead of using metastore

2018-05-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-24349: --- Description: In [SPARK-23639|https://issues.apache.org/jira/browse/SPARK-23639], use --proxy-user

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335424#comment-16335424 ] Lantao Jin commented on SPARK-23187: Hi [~jerryshao], do you have time to look at it? > Accumulator

[jira] [Created] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-22 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-23187: -- Summary: Accumulator object can not be sent from Executor to Driver Key: SPARK-23187 URL: https://issues.apache.org/jira/browse/SPARK-23187 Project: Spark Issue

[jira] [Updated] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23187: --- Affects Version/s: 2.3.1 > Accumulator object can not be sent from Executor to Driver >

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-22 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335431#comment-16335431 ] Lantao Jin commented on SPARK-23187: The simplest way to verify is just set a fixed value in

[jira] [Comment Edited] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-27 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342463#comment-16342463 ] Lantao Jin edited comment on SPARK-23187 at 1/28/18 6:15 AM: - [~jerryshao]

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-27 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342463#comment-16342463 ] Lantao Jin commented on SPARK-23187: [~jerryshao] {quota} So I guess it is because UI doesn't display

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: ExecutorTab2.png > Additional Memory Tuning Metrics > >

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin edited comment on SPARK-23206 at 2/9/18 2:28 AM: Yes, we have done

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin commented on SPARK-23206: Yes, we have done the similar things: !Screen Shot 2018-02-09

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: (was: Screen Shot 2018-02-09 at 10.21.19.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: Screen Shot 2018-02-09 at 10.21.19.png > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: ExecutorsTab2.png > Additional Memory Tuning Metrics >

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin edited comment on SPARK-23206 at 2/9/18 7:56 AM: Yes, we have done

[jira] [Created] (SPARK-23385) Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI

2018-02-10 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-23385: -- Summary: Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI Key: SPARK-23385 URL: https://issues.apache.org/jira/browse/SPARK-23385

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: (was: ExecutorTab2.png) > Additional Memory Tuning Metrics >

[jira] [Created] (SPARK-23353) Allow ExecutorMetricsUpdate events to be logged to the event log with sampling

2018-02-07 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-23353: -- Summary: Allow ExecutorMetricsUpdate events to be logged to the event log with sampling Key: SPARK-23353 URL: https://issues.apache.org/jira/browse/SPARK-23353 Project:

[jira] [Resolved] (SPARK-25054) Enable MetricsServlet sink for Executor

2018-08-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-25054. Resolution: Won't Fix > Enable MetricsServlet sink for Executor >

[jira] [Closed] (SPARK-25054) Enable MetricsServlet sink for Executor

2018-08-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin closed SPARK-25054. -- > Enable MetricsServlet sink for Executor > --- > >

[jira] [Created] (SPARK-25054) Enable MetricsServlet sink for Executor

2018-08-07 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25054: -- Summary: Enable MetricsServlet sink for Executor Key: SPARK-25054 URL: https://issues.apache.org/jira/browse/SPARK-25054 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575860#comment-16575860 ] Lantao Jin commented on SPARK-25084: I offer other fix way.

[jira] [Updated] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Summary: Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

[jira] [Created] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete location in event log

2018-09-06 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25357: -- Summary: Abbreviated metadata in DataSourceScanExec results in incomplete location in event log Key: SPARK-25357 URL: https://issues.apache.org/jira/browse/SPARK-25357

[jira] [Updated] (SPARK-25357) Abbreviated simpleString in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Summary: Abbreviated simpleString in DataSourceScanExec results in incomplete information in event

[jira] [Updated] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Description: Field {{metadata}} removed from {{SparkPlanInfo}} in SPARK-17701. Corresponding, this

[jira] [Updated] (SPARK-25357) Add metadata to SparkPlanInfo to dump more information like file path to event log

2018-09-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Summary: Add metadata to SparkPlanInfo to dump more information like file path to event log (was:

[jira] [Updated] (SPARK-25357) Add metadata to SparkPlanInfo to dump more information like file path to event log

2018-09-10 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Description: Field {{metadata}} removed from {{SparkPlanInfo}} in SPARK-17701. Corresponding, this

[jira] [Commented] (SPARK-24750) HiveCaseSensitiveInferenceMode with INFER_AND_SAVE will show WRITE permission denied even if select table operation

2018-07-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534616#comment-16534616 ] Lantao Jin commented on SPARK-24750: I think the default value of HiveCaseSensitiveInferenceMode

[jira] [Updated] (SPARK-24750) HiveCaseSensitiveInferenceMode with INFER_AND_SAVE will show WRITE permission denied even if select table operation

2018-07-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-24750: --- Description: The default HiveCaseSensitiveInferenceMode is INFER_AND_SAVE. In this mode, even if I

[jira] [Created] (SPARK-24750) HiveCaseSensitiveInferenceMode with INFER_AND_SAVE will show WRITE permission denied even if select table operation

2018-07-06 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-24750: -- Summary: HiveCaseSensitiveInferenceMode with INFER_AND_SAVE will show WRITE permission denied even if select table operation Key: SPARK-24750 URL:

[jira] [Commented] (SPARK-24750) HiveCaseSensitiveInferenceMode with INFER_AND_SAVE will show WRITE permission denied even if select table operation

2018-07-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534614#comment-16534614 ] Lantao Jin commented on SPARK-24750: [~smilegator] [~cloud_fan] [~jerryshao] >

[jira] [Updated] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23653: --- Description: [SPARK-4871|https://issues.apache.org/jira/browse/SPARK-4871] has already added the

[jira] [Updated] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23653: --- Attachment: Screen Shot 2018-03-12 at 20.16.14.png > Show sql statement in spark SQL UI >

[jira] [Updated] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23653: --- Attachment: Screen Shot 2018-03-12 at 14.25.51.png > Show sql statement in spark SQL UI >

[jira] [Created] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-23653: -- Summary: Show sql statement in spark SQL UI Key: SPARK-23653 URL: https://issues.apache.org/jira/browse/SPARK-23653 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23653: --- Attachment: Screen Shot 2018-03-12 at 20.16.23.png > Show sql statement in spark SQL UI >

[jira] [Commented] (SPARK-23653) Show sql statement in spark SQL UI

2018-03-12 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395241#comment-16395241 ] Lantao Jin commented on SPARK-23653: The best way is displaying the intact sql statement in SQL Tab

[jira] [Updated] (SPARK-22744) Cannot get the submit hostname of application

2018-03-21 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-22744: --- Summary: Cannot get the submit hostname of application (was: Set the application submit hostname by

[jira] [Updated] (SPARK-22744) Set the application submit hostname by default

2018-03-21 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-22744: --- Description: In MapReduce, we can get the submit hostname via checking the value of configuration

[jira] [Updated] (SPARK-22744) Set the application submit hostname by default

2018-03-21 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-22744: --- Summary: Set the application submit hostname by default (was: Add a configuration to show the

[jira] [Reopened] (SPARK-22744) Cannot get the submit hostname of application

2018-03-21 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin reopened SPARK-22744: I think its very useful if we can get the submit host of application. Change some code to prevent

[jira] [Resolved] (SPARK-22744) Cannot get the submit hostname of application

2018-03-23 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-22744. Resolution: Won't Fix Close it while found a work around way. > Cannot get the submit hostname of

[jira] [Created] (SPARK-25865) Add GC information to ExecutorMetrics

2018-10-29 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25865: -- Summary: Add GC information to ExecutorMetrics Key: SPARK-25865 URL: https://issues.apache.org/jira/browse/SPARK-25865 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-25685) Allow regression testing in enterprise Jenkins

2018-10-09 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25685: -- Summary: Allow regression testing in enterprise Jenkins Key: SPARK-25685 URL: https://issues.apache.org/jira/browse/SPARK-25685 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25685) Allow running tests in Jenkins in enterprise Git repository

2018-10-09 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25685: --- Summary: Allow running tests in Jenkins in enterprise Git repository (was: Allow regression

  1   2   3   4   >