Re: println in MapReduce job

2015-09-25 Thread xeonmailinglist 

I have found the error. It was a bug in my code.

On 09/24/2015 07:47 PM, xeonmailinglist wrote:
No, I am looking to the right place. Here is the output of a maptask. 
I also thought that the print would come up, but it isn't showing. 
That is why I am asking.



[1] Output from from one map task.

```

Log Type: stderr

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 317

Java HotSpot(TM) Client VM warning: You have loaded library 
/home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have 
disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', 
or link it with '-z noexecstack'.

Log Type: stdout

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 0

Log Type: syslog

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 2604

2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
2015-09-24 12:45:05,139 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from 
hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 
10 second(s).
2015-09-24 12:45:05,413 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: 
Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: 
mapreduce.job, Service: job_1443113036547_0001, Ident: 
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: 
Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: 
mapreduce.cluster.local.dir for child: 
/tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] 
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. 
Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task:  Using 
ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: 
Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: 
Task:attempt_1443113036547_0001_m_00_0 is done. And is in the process of 
committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task 
attempt_1443113036547_0001_m_00_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of 
task 'attempt_1443113036547_0001_m_00_0' to 
hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_00
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 
'attempt_1443113036547_0001_m_00_0' done.
2015-09-24 12:45:11,135 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics 
system...
2015-09-24 12:45:11,136 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
stopped.
2015-09-24 12:45:11,136 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
shutdown complete.

```

On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
You log statement should show up in the container log for that Map 
class within the data node. I'm guessing you aren't looking in the 
right place.


On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist 
> wrote:


Does anyone know this question about logging in MapReduce. I
can't find an example or an explanation on how to print something
inside user map and reduce functions.

On 09/24/2015 04:19 PM, xeonmailinglist wrote:


Hi,

1.

I have this example of MapReduce [1], and I want to print
info in the stdout and in a log file. It seems that the logs
isn’t print anything. How can I make my class print these words?

2.

I also have set in the |yarn-site.xml| to retain log.
Although the logs are retained in the |/app-logs| dir, the
|userlogs| dir is deleted at the end of the job execution.
How can I make MapReduce to not delete files in the
|userlogs| dir?

I am using Yarn.

Thanks,

[1] Wordcount exampla with just the map part.

|public class MyWordCount { public static class MyMap extends
Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
final static IntWritable one = new IntWritable(1); private Text
word = new Text(); public void map(LongWritable key, Text value,
OutputCollector output, Reporter reporter)
throws IOException {

Re: println in MapReduce job

2015-09-24 Thread Rich Haase 
To unsubscribe from this list send an email to 
user-unsubscr...@hadoop.apache.org.

https://hadoop.apache.org/mailing_lists.html

On Sep 24, 2015, at 9:40 AM, sukesh kumar 
> wrote:

unsubscribe

On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist 
> wrote:

Hi,

  1.
I have this example of MapReduce [1], and I want to print info in the stdout 
and in a log file. It seems that the logs isn’t print anything. How can I make 
my class print these words?
  2.
I also have set in the yarn-site.xml to retain log. Although the logs are 
retained in the /app-logs dir, the userlogs dir is deleted at the end of the 
job execution. How can I make MapReduce to not delete files in the userlogs dir?

I am using Yarn.

Thanks,

[1] Wordcount exampla with just the map part.


public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info("HERE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}

public void run(Context context) throws IOException, 
InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " 
Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), 
context);
}
} finally {
cleanup(context);
}
}

public void cleanup(Mapper.Context context) {}
}


[2] yarn-site.xml


 yarn.log-aggregation-enable true 

 yarn.nodemanager.log.retain-seconds 
90 
 yarn.nodemanager.remote-app-log-dir 
/app-logs 


​



--
Thanks & Best Regards
Sukesh Kumar

Re: println in MapReduce job

2015-09-24 Thread sukesh kumar 
unsubscribe

On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist 
wrote:

> Hi,
>
>1.
>
>I have this example of MapReduce [1], and I want to print info in the
>stdout and in a log file. It seems that the logs isn’t print anything. How
>can I make my class print these words?
>2.
>
>I also have set in the yarn-site.xml to retain log. Although the logs
>are retained in the /app-logs dir, the userlogs dir is deleted at the
>end of the job execution. How can I make MapReduce to not delete files in
>the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, 
> InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " 
> Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), 
> context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> 
>  yarn.log-aggregation-enable 
> true 
>  yarn.nodemanager.log.retain-seconds 
> 90 
>  yarn.nodemanager.remote-app-log-dir 
> /app-logs 
>
> ​
>



-- 
Thanks & Best Regards
Sukesh Kumar

Re: println in MapReduce job

2015-09-24 Thread Peyman Mohajerian 
You log statement should show up in the container log for that Map class
within the data node. I'm guessing you aren't looking in the right place.

On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist 
wrote:

> Does anyone know this question about logging in MapReduce. I can't find an
> example or an explanation on how to print something inside user map and
> reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
>1.
>
>I have this example of MapReduce [1], and I want to print info in the
>stdout and in a log file. It seems that the logs isn’t print anything. How
>can I make my class print these words?
>2.
>
>I also have set in the yarn-site.xml to retain log. Although the logs
>are retained in the /app-logs dir, the userlogs dir is deleted at the
>end of the job execution. How can I make MapReduce to not delete files in
>the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, 
> InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " 
> Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), 
> context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> 
>  yarn.log-aggregation-enable 
> true 
>  yarn.nodemanager.log.retain-seconds 
> 90 
>  yarn.nodemanager.remote-app-log-dir 
> /app-logs 
>
> ​
>
>
>

Re: println in MapReduce job

2015-09-24 Thread xeonmailinglist 
No, I am looking to the right place. Here is the output of a maptask. I 
also thought that the print would come up, but it isn't showing. That is 
why I am asking.



[1] Output from from one map task.

```

Log Type: stderr

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 317

Java HotSpot(TM) Client VM warning: You have loaded library 
/home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have 
disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', 
or link it with '-z noexecstack'.

Log Type: stdout

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 0

Log Type: syslog

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 2604

2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
2015-09-24 12:45:05,139 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from 
hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 
10 second(s).
2015-09-24 12:45:05,413 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: 
Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: 
mapreduce.job, Service: job_1443113036547_0001, Ident: 
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: 
Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: 
mapreduce.cluster.local.dir for child: 
/tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] 
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. 
Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task:  Using 
ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: 
Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: 
Task:attempt_1443113036547_0001_m_00_0 is done. And is in the process of 
committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task 
attempt_1443113036547_0001_m_00_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of 
task 'attempt_1443113036547_0001_m_00_0' to 
hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_00
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 
'attempt_1443113036547_0001_m_00_0' done.
2015-09-24 12:45:11,135 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics 
system...
2015-09-24 12:45:11,136 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
stopped.
2015-09-24 12:45:11,136 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system 
shutdown complete.


```

On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
You log statement should show up in the container log for that Map 
class within the data node. I'm guessing you aren't looking in the 
right place.


On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist 
> wrote:


Does anyone know this question about logging in MapReduce. I can't
find an example or an explanation on how to print something inside
user map and reduce functions.

On 09/24/2015 04:19 PM, xeonmailinglist wrote:


Hi,

1.

I have this example of MapReduce [1], and I want to print
info in the stdout and in a log file. It seems that the logs
isn’t print anything. How can I make my class print these words?

2.

I also have set in the |yarn-site.xml| to retain log.
Although the logs are retained in the |/app-logs| dir, the
|userlogs| dir is deleted at the end of the job execution.
How can I make MapReduce to not delete files in the
|userlogs| dir?

I am using Yarn.

Thanks,

[1] Wordcount exampla with just the map part.

|public class MyWordCount { public static class MyMap extends
Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
final static IntWritable one = new IntWritable(1); private Text
word = new Text(); public void map(LongWritable key, Text value,
OutputCollector output, Reporter reporter)
throws IOException { StringTokenizer itr = new
StringTokenizer(value.toString()); System.out.println("HERRE");

Re: println in MapReduce job

2015-09-24 Thread xeonmailinglist 
Does anyone know this question about logging in MapReduce. I can't find 
an example or an explanation on how to print something inside user map 
and reduce functions.


On 09/24/2015 04:19 PM, xeonmailinglist wrote:


Hi,

1.

I have this example of MapReduce [1], and I want to print info in
the stdout and in a log file. It seems that the logs isn’t print
anything. How can I make my class print these words?

2.

I also have set in the |yarn-site.xml| to retain log. Although the
logs are retained in the |/app-logs| dir, the |userlogs| dir is
deleted at the end of the job execution. How can I make MapReduce
to not delete files in the |userlogs| dir?

I am using Yarn.

Thanks,

[1] Wordcount exampla with just the map part.

|public class MyWordCount { public static class MyMap extends Mapper { 
Log log = LogFactory.getLog(MyWordCount.class); private final static 
IntWritable one = new IntWritable(1); private Text word = new Text(); 
public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) throws IOException { 
StringTokenizer itr = new StringTokenizer(value.toString()); 
System.out.println("HERRE"); log.info("HERE"); while 
(itr.hasMoreTokens()) { word.set(itr.nextToken()); 
output.collect(word, one); } } public void run(Context context) throws 
IOException, InterruptedException { setup(context); try { while 
(context.nextKeyValue()) { System.out.println("Key: " + 
context.getCurrentKey() + " Value: " + context.getCurrentValue()); 
map(context.getCurrentKey(), context.getCurrentValue(), context); } } 
finally { cleanup(context); } } public void cleanup(Mapper.Context 
context) {} } |


[2] yarn-site.xml

|  
yarn.log-aggregation-enable true 
  
yarn.nodemanager.log.retain-seconds 90 
  
yarn.nodemanager.remote-app-log-dir 
/app-logs  |

​