[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836951#comment-16836951 ] nichunen commented on KYLIN-3866: - Committed to 2.6.x branch > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v2.6.2, v3.0.0-alpha > > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792399#comment-16792399 ] Guangxu Cheng commented on KYLIN-3866: -- Thanks for reviewing and committing, [~Shaofengshi] :) > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v2.6.2 > > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792379#comment-16792379 ] Shaofeng SHI commented on KYLIN-3866: - I agree with you! +1 I have merged the patch to master and 2.6.x branch, thanks for making Kylin better, Guangxu! > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792377#comment-16792377 ] ASF subversion and git services commented on KYLIN-3866: Commit 3b05e1066647609bea6c026a7ef44af429c2ec63 in kylin's branch refs/heads/2.6.x from Guangxu Cheng [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3b05e10 ] KYLIN-3866 Whether to set mapreduce.application.classpath is determined by the user Signed-off-by: shaofengshi > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792376#comment-16792376 ] ASF subversion and git services commented on KYLIN-3866: Commit d23d2bddab3263d39059492b91f71d8091198057 in kylin's branch refs/heads/master from Guangxu Cheng [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d23d2bd ] KYLIN-3866 Whether to set mapreduce.application.classpath is determined by the user Signed-off-by: shaofengshi > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791612#comment-16791612 ] Guangxu Cheng commented on KYLIN-3866: -- Add parameter "kylin.engine.mr.use-local-classpath" and set the default value to true, mainly to keep the compatibility of versions. For old users, upgrading kylin to a higher version has no effect on them. For new users, this feature can be turned off with this parameter if it is not needed. If the changes can be listed in the upgrade document to remind the old user.I personally think it is okay to remove this logic. WDYT? Thanks > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791575#comment-16791575 ] Shaofeng SHI commented on KYLIN-3866: - Now I know the problem, thank you for the information! The change will be good; I see you add a config parameter to keep the old behavior, is that parameter is necessary? I mean, if the "mapreduce.application.classpath" is absent, can we just let it go (totally not get the classpath from local configuration)? > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791409#comment-16791409
]
Guangxu Cheng commented on KYLIN-3866:
--
{quote}While, I still not understand the requirment: in which case the
"mapreduce.application.classpath" is absent, and still not allow to use local
cmd to get the classpath? Usually Kylin is installed in a Hadoop client node,
which need the MR/Hive/HBase be well configured, the classpath configuration
should be correct for all Hadoop node. Could you please elaborate the problem
you encountered, and how common it is? Thank you!
{quote}
[~Shaofengshi] Thanks for your review.
1. "mapreduce.application.classpath" is not a required parameter. If the
parameter is not configured, the default value will be used. see:
[https://github.com/apache/hadoop/blob/5d8c8cd568ae54c1be366e786c738b63bd53e271/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L219]
In our cluster, we use the default value and do not need the user to configure
the parameter on the client side.
2. We are not deploying kylin on the nodes of the MR cluster because the MR
cluster is maintained by other departments within Tencent and we do not have
permission to deploy other applications on the nodes of the MR cluster. We also
don't know the installation path of the MR cluster.
In Tencent, Kylin is deployed separately, Hadoop/HBase/Hive client is deployed
on the kylin node, but the path of deploying Hadoop client may not be the same
as the MR cluster. So, the path obtained by the command "mapred classpath" is
only the path of Hadoop client not the path of the MR cluster deployment.
Now, kylin forces the user to configure this parameter. If it is not
configured, the path obtained by using the command "mapred classpath" (which
may be wrong value) will be assigned to the parameter, which I think it is
unreasonable.
So, it may be more reasonable for the user to decide whether to use the
parameter.
> Whether to set mapreduce.application.classpath is determined by the user
>
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Priority: Major
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter
> "mapreduce.application.classpath", then kylin will use the command "mapred
> classpath" to get the local classpath and set it to
> "mapreduce.application.classpath", However, it will cause the following two
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to
> configure this parameter or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791260#comment-16791260 ] Shaofeng SHI commented on KYLIN-3866: - Hi [~andrewcheng], welcome to report issue to Kylin! While, I still not understand the requirment: in which case the "mapreduce.application.classpath" is absent, and still not allow to use local cmd to get the classpath? Usually Kylin is installed in a Hadoop client node, which need the MR/Hive/HBase be well configured, the classpath configuration should be correct for all Hadoop node. Could you please elaborate the problem you encountered, and how common it is? Thank you! > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789357#comment-16789357 ] Guangxu Cheng commented on KYLIN-3866: -- [~Shaofengshi] mind taking a look at it ? thanks > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
