[
https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943
]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 4:41 PM:
-----------------------------------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add
property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties.
Then I changed a line
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server",
clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to
hadoop-site.xml, unfortunately when the createClientSideProperties() is to be
writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<<
FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0
sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at
org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that
currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec
clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only
have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that
functionality for client. In other words, the current "hadoop-common" prefix
from whirr-hadoop-default.properties is to be separated into a "hadoop-client"?
I think so, because on the server side we don't want to add a
hadoop.socks.server property.
I am right?
was (Author: tibor.kiss):
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add
property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties.
Then I changed a line
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server",
clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to
hadoop-site.xml, unfortunately when the createClientSideProperties() is to be
writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<<
FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0
sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at
org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that
currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec
clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only
have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that
functionality for client. I am right?
> Add a new optional c parameter for being able to configure the port of socks
> connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which
> contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed
> a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this
> 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which
> contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded
> value.)
> In order to run multiple clusters you only need to override this paramter
> knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
