performance in a relatively big cluster.
This behavior is very easy to reproduce by simply running terasort on a
cluster.
Any suggestion to fix this problem?
On Wed, Jan 8, 2014 at 11:18 AM, Jian Fang wrote:
> Thanks Vinod for your quick response. It is running in non-secure mode.
>
> I still
hould do so as the user, not the job-id. It is not just logging, I'd be
> surprised if jobs pass. Do you have permissions enabled on HDFS?
>
> Oh, or is this in non-secure mode (i.e. without kerberos)?
>
> +Vinod
>
>
> On Jan 7, 2014, at 5:14 PM, Jian Fang
> wrote:
Hi,
I looked at Hadoop 1.X source code and found some logic that I could not
understand.
In the org.apache.hadoop.mapred.Child class, there were two UGIs defined as
follows.
UserGroupInformation current = UserGroupInformation.getCurrentUser();
current.addToken(jt);
UserGroupInformat
Output Format Counters
Bytes Written=1
Thanks,
John
On Tue, Oct 22, 2013 at 2:44 PM, Jian Fang wrote:
> Hi,
>
> I have the same problem. I compared Hadoop 2.2.0 with Hadoop 1.0.3 and it
> turned out that the terasort for MR2 is 2 times slower than th
Hi,
I have the same problem. I compared Hadoop 2.2.0 with Hadoop 1.0.3 and it
turned out that the terasort for MR2 is 2 times slower than that in MR1. I
cannot really believe it.
The cluster has 20 nodes with 19 data nodes. My Hadoop 2.2.0 cluster
configurations are as follows.
mapreduc
4 PM, Jian Fang wrote:
> Ok, seems there is a jira for this issue.
>
> https://issues.apache.org/jira/browse/YARN-800
>
>
> On Mon, Sep 9, 2013 at 3:39 PM, Jian Fang
> wrote:
>
>> Hi,
>>
>> I need to use the web services in application master, for ex
Ok, seems there is a jira for this issue.
https://issues.apache.org/jira/browse/YARN-800
On Mon, Sep 9, 2013 at 3:39 PM, Jian Fang wrote:
> Hi,
>
> I need to use the web services in application master, for example,
>
> curl
> http://10.6.179.230:9026/proxy/application_137876
John
>
> This exception should indicate error from the container process. If the
> container process exits with non-zero exit code, it will be logged.
> In case of such errors, you'd better look at the per-container log see
> what's happening there.
>
> Jian
>
Hi,
I need to use the web services in application master, for example,
curl
http://10.6.179.230:9026/proxy/application_1378761541170_0003/ws/v1/mapreduce/info
This used to work fine, but after I upgraded to Hadoop 2.1.0-beta, I got
the following errors.
Error 500 Connection refused
HTTP ER
Hi,
I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error
messages as follows.
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: bash:
/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_000
Ok, I figured out the solution. I need to make sure my own shutdown hook
has higher priority than the one in FileSystem.
On Thu, Apr 11, 2013 at 12:08 PM, Jian Fang
wrote:
> Hi,
>
> We have some customized code to flush data during hadoop shutdown process,
> but we run into t
Hi,
We have some customized code to flush data during hadoop shutdown process,
but we run into the following errors with Hadoop 2.0.3-alpha
Exception in thread "Thread-3" java.lang.ExceptionInInitializerError
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2323)
Thanks Harsh for your quick response.
On Tue, Apr 9, 2013 at 11:45 AM, Harsh J wrote:
> These are part of the HDFS-347 branch which should be merged to trunk
> soon (and then hopefully, onto one of the upcoming 2.x releases, via a
> backport).
>
> On Tue, Apr 9, 2013 at 10:0
Hi,
I saw HDFS-4354 and HDFS-4416 have been used in Cloudera's CDH4.2 to
improve short circuit read. Is there any plan to apply them to Hadoop 2.0.X
branches? Seems the short circuit read implementation in current Hadoop
2.0.x branches are quite different from HDFS-4354 and HDFS-4416.
Thanks in a
instead on the fuller
> dfs.http.address in 1.x and dfs.namenode.http.address in 2.x.
>
> On Sat, Aug 11, 2012 at 3:45 AM, Jian Fang
> wrote:
> > Thanks Joey for the clarification. I will ask our hadoop admin to change
> > that.
> > But it would be great if this could
for
> HFTP.
>
> You should make sure that your settings for dfs.http.address and
> dfs.info.port are in sync. So change one of those to match the port
> number of the other.
>
> -Joey
>
> On Fri, Aug 10, 2012 at 5:41 PM, Jian Fang
> wrote:
> > Hi Joey,
> >
tp.address?
Thanks,
Jian
On Fri, Aug 10, 2012 at 5:08 PM, Joey Echeverria wrote:
> Can you post your NN logs? It looks like the NN is not actually
> started or is listening on another port for HTTP.
>
> -Joey
>
> On Fri, Aug 10, 2012 at 2:38 PM, Jian Fang
> wrote:
>
Already did that. Connection was rejected.
On Fri, Aug 10, 2012 at 2:24 PM, Joey Echeverria wrote:
> Try:
>
> $ telnet pnjhadoopnn01 50070
>
> -Joey
>
> On Fri, Aug 10, 2012 at 1:10 PM, Jian Fang
> wrote:
> > Here is the property in hdfs-site.xml
> >
r what its
> dfs.http.address may really be using for its port?
>
> On Fri, Aug 10, 2012 at 7:48 PM, Jian Fang
> wrote:
> > Hi Harsh,
> >
> > Seems the -p requires the root privilege, which I don't have. I run
> > "netstat -a | grep 50070", but d
i, Aug 10, 2012 at 1:50 AM, Harsh J wrote:
> Jian,
>
> From your NN, can you get us the output "netstat -anp | grep 50070"?
>
> On Fri, Aug 10, 2012 at 9:29 AM, Jian Fang
> wrote:
> > Thanks Harsh. But there is no firewall there, the two clusters are on the
>
ll isn't the cause of the connection refused on port 50070 on
> the NN and ports 50075 on the DNs here?
>
> On Fri, Aug 10, 2012 at 1:47 AM, Jian Fang
> wrote:
> > Hi,
> >
> > We have a hadoop cluster of version 0.20.2 in production. Now we have
> > another n
Hi,
We have a hadoop cluster of version 0.20.2 in production. Now we have
another new Hadoop cluster using cloudera's CDH3U4. We like to run distcp
to copy files between the two clusters. Since the hadoop versions are
different, we have to use hftp protocol to copy files based on the hadoop
docume
22 matches
Mail list logo
