reduce no response
Hi all, Do terasort use 4 reduce 8 map , when all maps finish reduce no response. Debug messages like: 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7126 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7126 12/02/02 01:42:47 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7127 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7127 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7128 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7128 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7129 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7129 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7130 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7130 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7131 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7131 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getTaskCompletionEvents 1 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7132 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7132 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getJobStatus 0 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7133 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7133 The information and any attached documents contained in this message may be confidential and/or legally privileged. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, or reproduction is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender immediately by return e-mail and destroy all copies of the original message.
reduce no response
Hi all, Do terasort use 4 reduce 8 map , when all maps finish reduce no response. Debug messages like: 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7126 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7126 12/02/02 01:42:47 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7127 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7127 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7128 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7128 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7129 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7129 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7130 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7130 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getJobStatus 1 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7131 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7131 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getTaskCompletionEvents 1 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7132 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7132 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getJobStatus 0 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop sending #7133 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:9001 from hadoop got value #7133 The information and any attached documents contained in this message may be confidential and/or legally privileged. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, or reproduction is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender immediately by return e-mail and destroy all copies of the original message.
Re: reduce no response
Jinyan, Am not sure what your problem here seems to be - the client hanging or the job itself hanging. Could you provide us some more information on what state the job is hung in, or expand on the job client hang? Having a jstack also helps whenever you run into a JVM hang. On Wed, Feb 1, 2012 at 3:46 PM, Jinyan Xu wrote: > Hi all, > > Do terasort use 4 reduce 8 map , when all maps finish reduce no response. > > Debug messages like: > > 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7126 > 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7126 > 12/02/02 01:42:47 DEBUG ipc.RPC: Call: getJobStatus 1 > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7127 > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7127 > 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7128 > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7128 > 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getJobStatus 1 > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7129 > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7129 > 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7130 > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7130 > 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getJobStatus 1 > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7131 > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7131 > 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getTaskCompletionEvents 1 > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7132 > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7132 > 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getJobStatus 0 > 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7133 > 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7133 > > > > The information and any attached documents contained in this message > may be confidential and/or legally privileged. The message is > intended solely for the addressee(s). If you are not the intended > recipient, you are hereby notified that any use, dissemination, or > reproduction is strictly prohibited and may be unlawful. If you are > not the intended recipient, please contact the sender immediately by > return e-mail and destroy all copies of the original message. -- Harsh J Customer Ops. Engineer Cloudera | http://tiny.cloudera.com/about
Re: reduce no response
Hi Jinyan, To quickly check if it has to do with resolving the ip address, can you check what is the hostname of the system? A quick hack would be to rename host itself as "localhost" if you are using cloudera's pseudo cluster hadoop for testing. Thanks, -Idris On Wed, Feb 1, 2012 at 4:24 PM, Harsh J wrote: > Jinyan, > > Am not sure what your problem here seems to be - the client hanging or > the job itself hanging. Could you provide us some more information on > what state the job is hung in, or expand on the job client hang? > Having a jstack also helps whenever you run into a JVM hang. > > On Wed, Feb 1, 2012 at 3:46 PM, Jinyan Xu wrote: > > Hi all, > > > > Do terasort use 4 reduce 8 map , when all maps finish reduce no response. > > > > Debug messages like: > > > > 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7126 > > 12/02/02 01:42:47 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7126 > > 12/02/02 01:42:47 DEBUG ipc.RPC: Call: getJobStatus 1 > > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7127 > > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7127 > > 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 > > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7128 > > 12/02/02 01:42:48 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7128 > > 12/02/02 01:42:48 DEBUG ipc.RPC: Call: getJobStatus 1 > > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7129 > > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7129 > > 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getTaskCompletionEvents 0 > > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7130 > > 12/02/02 01:42:49 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7130 > > 12/02/02 01:42:49 DEBUG ipc.RPC: Call: getJobStatus 1 > > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7131 > > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7131 > > 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getTaskCompletionEvents 1 > > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7132 > > 12/02/02 01:42:50 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7132 > > 12/02/02 01:42:50 DEBUG ipc.RPC: Call: getJobStatus 0 > > 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop sending #7133 > > 12/02/02 01:42:51 DEBUG ipc.Client: IPC Client (47) connection to > localhost/127.0.0.1:9001 from hadoop got value #7133 > > > > > > > > The information and any attached documents contained in this message > > may be confidential and/or legally privileged. The message is > > intended solely for the addressee(s). If you are not the intended > > recipient, you are hereby notified that any use, dissemination, or > > reproduction is strictly prohibited and may be unlawful. If you are > > not the intended recipient, please contact the sender immediately by > > return e-mail and destroy all copies of the original message. > > > > -- > Harsh J > Customer Ops. Engineer > Cloudera | http://tiny.cloudera.com/about >
