[
https://issues.apache.org/jira/browse/PIG-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199320#comment-14199320
]
Eyal Allweil commented on PIG-3399:
-----------------------------------
Here it is:
Testsuite: org.apache.pig.test.TestCommit
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.792 sec
------------- Standard Output ---------------
2014-11-06 01:04:10,686 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: file:///
2014-11-06 01:04:11,324 [main] WARN org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2014-11-06 01:04:11,360 [main] WARN org.apache.pig.newplan.BaseOperatorPlan -
Encountered Warning IMPLICIT_CAST_TO_INT 2 time(s).
2014-11-06 01:04:11,379 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig features used in the script: COGROUP,GROUP_BY,ORDER_BY,FILTER
2014-11-06 01:04:11,413 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,
GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2014-11-06 01:04:11,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
2014-11-06 01:04:11,507 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil -
Choosing to move algebraic foreach to combiner
2014-11-06 01:04:11,522 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizerMR
- Using Secondary Key Optimization for MapReduce node scope-49
2014-11-06 01:04:11,525 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POPackage(JoinPackager)
2014-11-06 01:04:11,537 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2014-11-06 01:04:11,538 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2014-11-06 01:04:11,577 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings
are added to the job
2014-11-06 01:04:11,585 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-11-06 01:04:11,587 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Reduce phase detected, estimating # of required reducers.
2014-11-06 01:04:11,587 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Using reducer estimator:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-11-06 01:04:11,590 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=145
2014-11-06 01:04:11,591 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2014-11-06 01:04:11,608 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-11-06 01:04:11,611 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Key [pig.schematuple] is false, will not generate code.
2014-11-06 01:04:11,611 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Starting process to move generated code to distributed cacche
2014-11-06 01:04:11,611 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Distributed cache not supported or needed in local mode. Setting key
[pig.schematuple.local.dir] with code temp directory:
/tmp/pig_junit_tmp2035031198/1415228651611-0
2014-11-06 01:04:11,689 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-11-06 01:04:11,706 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2014-11-06 01:04:11,712 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- No job jar file set. User classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
2014-11-06 01:04:11,745 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:11,745 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:11,752 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2014-11-06 01:04:11,753 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-11-06 01:04:11,755 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:11,756 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:11,756 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-11-06 01:04:12,000 [Thread-2] INFO org.apache.hadoop.util.ProcessTree -
setsid exited with exit code 0
2014-11-06 01:04:12,016 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@22762e43
2014-11-06 01:04:12,034 [Thread-2] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader -
Current split being processed
file:/tmp/pig_junit_tmp2035031198/tmp2814450914960679041voter:0+82
2014-11-06 01:04:12,041 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
io.sort.mb = 100
2014-11-06 01:04:12,070 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
data buffer = 79691776/99614720
2014-11-06 01:04:12,070 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
record buffer = 262144/327680
2014-11-06 01:04:12,083 [Thread-2] INFO org.apache.pig.data.SchemaTupleBackend
- Key [pig.schematuple] was not set... will not generate code.
2014-11-06 01:04:12,098 [Thread-2] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map
- Aliases being processed per job phase (AliasName[line,offset]): M:
a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R:
2014-11-06 01:04:12,105 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
Starting flush of map output
2014-11-06 01:04:12,114 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
Finished spill 0
2014-11-06 01:04:12,116 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:12,201 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0001
2014-11-06 01:04:12,201 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases a,b,c,d,e
2014-11-06 01:04:12,201 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R:
2014-11-06 01:04:12,202 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2014-11-06 01:04:12,203 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Running jobs are [job_local_0001]
2014-11-06 01:04:14,978 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:14,979 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0001_m_000000_0' done.
2014-11-06 01:04:14,993 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@12e194e2
2014-11-06 01:04:14,997 [Thread-2] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader -
Current split being processed
file:/tmp/pig_junit_tmp2035031198/tmp5809809739981971182student:0+63
2014-11-06 01:04:14,997 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
io.sort.mb = 100
2014-11-06 01:04:15,026 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
data buffer = 79691776/99614720
2014-11-06 01:04:15,026 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
record buffer = 262144/327680
2014-11-06 01:04:15,034 [Thread-2] WARN org.apache.pig.data.SchemaTupleBackend
- SchemaTupleBackend has already been initialized
2014-11-06 01:04:15,044 [Thread-2] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map
- Aliases being processed per job phase (AliasName[line,offset]): M:
a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R:
2014-11-06 01:04:15,045 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
Starting flush of map output
2014-11-06 01:04:15,048 [Thread-2] INFO org.apache.hadoop.mapred.MapTask -
Finished spill 0
2014-11-06 01:04:15,051 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
2014-11-06 01:04:15,205 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 12% complete
2014-11-06 01:04:15,206 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Running jobs are [job_local_0001]
2014-11-06 01:04:17,985 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:17,987 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0001_m_000001_0' done.
2014-11-06 01:04:18,001 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@46ace11e
2014-11-06 01:04:18,001 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:18,006 [Thread-2] INFO org.apache.hadoop.mapred.Merger -
Merging 2 sorted segments
2014-11-06 01:04:18,017 [Thread-2] INFO org.apache.hadoop.mapred.Merger -
Down to the last merge-pass, with 2 segments left of total size: 204 bytes
2014-11-06 01:04:18,017 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:18,033 [Thread-2] WARN org.apache.pig.data.SchemaTupleBackend
- SchemaTupleBackend has already been initialized
2014-11-06 01:04:18,064 [Thread-2] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce
- Aliases being processed per job phase (AliasName[line,offset]): M:
a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R:
2014-11-06 01:04:18,068 [Thread-2] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:18,069 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:18,069 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task
attempt_local_0001_r_000000_0 is allowed to commit now
2014-11-06 01:04:18,071 [Thread-2] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of
task 'attempt_local_0001_r_000000_0' to file:/tmp/temp-2074079481/tmp1973256487
2014-11-06 01:04:20,999 [Thread-2] INFO
org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2014-11-06 01:04:21,000 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0001_r_000000_0' done.
2014-11-06 01:04:21,214 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2014-11-06 01:04:21,228 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings
are added to the job
2014-11-06 01:04:21,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-11-06 01:04:21,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Reduce phase detected, estimating # of required reducers.
2014-11-06 01:04:21,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Using reducer estimator:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-11-06 01:04:21,230 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=164
2014-11-06 01:04:21,230 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2014-11-06 01:04:21,237 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-11-06 01:04:21,261 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-11-06 01:04:21,265 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2014-11-06 01:04:21,268 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- No job jar file set. User classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
2014-11-06 01:04:21,277 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:21,277 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:21,277 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-11-06 01:04:21,347 [Thread-9] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c9f91aa
2014-11-06 01:04:21,352 [Thread-9] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader -
Current split being processed
file:/tmp/temp-2074079481/tmp1973256487/part-r-00000:0+164
2014-11-06 01:04:21,352 [Thread-9] INFO org.apache.hadoop.mapred.MapTask -
io.sort.mb = 100
2014-11-06 01:04:21,382 [Thread-9] INFO org.apache.hadoop.mapred.MapTask -
data buffer = 79691776/99614720
2014-11-06 01:04:21,382 [Thread-9] INFO org.apache.hadoop.mapred.MapTask -
record buffer = 262144/327680
2014-11-06 01:04:21,391 [Thread-9] WARN org.apache.pig.data.SchemaTupleBackend
- SchemaTupleBackend has already been initialized
2014-11-06 01:04:21,398 [Thread-9] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map
- Aliases being processed per job phase (AliasName[line,offset]): M:
h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4]
2014-11-06 01:04:21,399 [Thread-9] INFO org.apache.hadoop.mapred.MapTask -
Starting flush of map output
2014-11-06 01:04:21,412 [Thread-9] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine
- Aliases being processed per job phase (AliasName[line,offset]): M:
h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4]
2014-11-06 01:04:21,413 [Thread-9] INFO org.apache.hadoop.mapred.MapTask -
Finished spill 0
2014-11-06 01:04:21,415 [Thread-9] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:21,761 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0002
2014-11-06 01:04:21,761 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases g,h
2014-11-06 01:04:21,762 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4]
2014-11-06 01:04:24,342 [Thread-9] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:24,344 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0002_m_000000_0' done.
2014-11-06 01:04:24,355 [Thread-9] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@36be5783
2014-11-06 01:04:24,355 [Thread-9] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:24,356 [Thread-9] INFO org.apache.hadoop.mapred.Merger -
Merging 1 sorted segments
2014-11-06 01:04:24,356 [Thread-9] INFO org.apache.hadoop.mapred.Merger -
Down to the last merge-pass, with 1 segments left of total size: 61 bytes
2014-11-06 01:04:24,356 [Thread-9] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:24,365 [Thread-9] WARN org.apache.pig.data.SchemaTupleBackend
- SchemaTupleBackend has already been initialized
2014-11-06 01:04:24,375 [Thread-9] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce
- Aliases being processed per job phase (AliasName[line,offset]): M:
h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4]
2014-11-06 01:04:24,376 [Thread-9] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:24,377 [Thread-9] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:24,377 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task
attempt_local_0002_r_000000_0 is allowed to commit now
2014-11-06 01:04:24,379 [Thread-9] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of
task 'attempt_local_0002_r_000000_0' to file:/tmp/temp-2074079481/tmp-588398347
2014-11-06 01:04:24,770 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 37% complete
2014-11-06 01:04:24,771 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Running jobs are [job_local_0002]
2014-11-06 01:04:27,353 [Thread-9] INFO
org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2014-11-06 01:04:27,355 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0002_r_000000_0' done.
2014-11-06 01:04:27,775 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2014-11-06 01:04:27,782 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings
are added to the job
2014-11-06 01:04:27,784 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-11-06 01:04:27,784 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Reduce phase detected, estimating # of required reducers.
2014-11-06 01:04:27,784 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Using reducer estimator:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-11-06 01:04:27,785 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=51
2014-11-06 01:04:27,786 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2014-11-06 01:04:27,793 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-11-06 01:04:27,817 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-11-06 01:04:27,819 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2014-11-06 01:04:27,819 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- No job jar file set. User classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
2014-11-06 01:04:27,828 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:27,828 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:27,828 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-11-06 01:04:27,922 [Thread-12] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@594d7f1e
2014-11-06 01:04:27,926 [Thread-12] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader -
Current split being processed
file:/tmp/temp-2074079481/tmp-588398347/part-r-00000:0+51
2014-11-06 01:04:27,926 [Thread-12] INFO org.apache.hadoop.mapred.MapTask -
io.sort.mb = 100
2014-11-06 01:04:28,018 [Thread-12] INFO org.apache.hadoop.mapred.MapTask -
data buffer = 79691776/99614720
2014-11-06 01:04:28,019 [Thread-12] INFO org.apache.hadoop.mapred.MapTask -
record buffer = 262144/327680
2014-11-06 01:04:28,025 [Thread-12] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been
initialized
2014-11-06 01:04:28,029 [Thread-12] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map
- Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4]
C: R:
2014-11-06 01:04:28,030 [Thread-12] INFO org.apache.hadoop.mapred.MapTask -
Starting flush of map output
2014-11-06 01:04:28,031 [Thread-12] INFO org.apache.hadoop.mapred.MapTask -
Finished spill 0
2014-11-06 01:04:28,032 [Thread-12] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0003_m_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:28,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0003
2014-11-06 01:04:28,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases i
2014-11-06 01:04:28,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: i[9,4] C: R:
2014-11-06 01:04:30,917 [Thread-12] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:30,918 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0003_m_000000_0' done.
2014-11-06 01:04:30,929 [Thread-12] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@517c51fe
2014-11-06 01:04:30,929 [Thread-12] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:30,930 [Thread-12] INFO org.apache.hadoop.mapred.Merger -
Merging 1 sorted segments
2014-11-06 01:04:30,930 [Thread-12] INFO org.apache.hadoop.mapred.Merger -
Down to the last merge-pass, with 1 segments left of total size: 66 bytes
2014-11-06 01:04:30,930 [Thread-12] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:30,934 [Thread-12] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been
initialized
2014-11-06 01:04:30,942 [Thread-12] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce
- Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4]
C: R:
2014-11-06 01:04:30,943 [Thread-12] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0003_r_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:30,943 [Thread-12] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:30,944 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task
attempt_local_0003_r_000000_0 is allowed to commit now
2014-11-06 01:04:30,945 [Thread-12] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of
task 'attempt_local_0003_r_000000_0' to file:/tmp/temp-2074079481/tmp-219746025
2014-11-06 01:04:31,326 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 62% complete
2014-11-06 01:04:31,326 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Running jobs are [job_local_0003]
2014-11-06 01:04:33,925 [Thread-12] INFO
org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2014-11-06 01:04:33,927 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0003_r_000000_0' done.
2014-11-06 01:04:34,330 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 75% complete
2014-11-06 01:04:34,337 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings
are added to the job
2014-11-06 01:04:34,338 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-11-06 01:04:34,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Reduce phase detected, estimating # of required reducers.
2014-11-06 01:04:34,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2014-11-06 01:04:34,350 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-11-06 01:04:34,361 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-11-06 01:04:34,367 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2014-11-06 01:04:34,369 [JobControl] WARN org.apache.hadoop.mapred.JobClient
- No job jar file set. User classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
2014-11-06 01:04:34,377 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:34,377 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:34,377 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2014-11-06 01:04:34,424 [Thread-15] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@46c807db
2014-11-06 01:04:34,427 [Thread-15] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader -
Current split being processed
file:/tmp/temp-2074079481/tmp-588398347/part-r-00000:0+51
2014-11-06 01:04:34,428 [Thread-15] INFO org.apache.hadoop.mapred.MapTask -
io.sort.mb = 100
2014-11-06 01:04:34,462 [Thread-15] INFO org.apache.hadoop.mapred.MapTask -
data buffer = 79691776/99614720
2014-11-06 01:04:34,462 [Thread-15] INFO org.apache.hadoop.mapred.MapTask -
record buffer = 262144/327680
2014-11-06 01:04:34,467 [Thread-15] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been
initialized
2014-11-06 01:04:34,469 [Thread-15] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map
- Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4]
C: R:
2014-11-06 01:04:34,473 [Thread-15] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:34,473 [Thread-15] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:34,475 [Thread-15] INFO org.apache.hadoop.mapred.MapTask -
Starting flush of map output
2014-11-06 01:04:34,477 [Thread-15] INFO org.apache.hadoop.mapred.MapTask -
Finished spill 0
2014-11-06 01:04:34,478 [Thread-15] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0004_m_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:34,862 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0004
2014-11-06 01:04:34,862 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases i
2014-11-06 01:04:34,862 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: i[9,4] C: R:
2014-11-06 01:04:37,413 [Thread-15] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:37,414 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0004_m_000000_0' done.
2014-11-06 01:04:37,430 [Thread-15] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@707123aa
2014-11-06 01:04:37,430 [Thread-15] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:37,432 [Thread-15] INFO org.apache.hadoop.mapred.Merger -
Merging 1 sorted segments
2014-11-06 01:04:37,432 [Thread-15] INFO org.apache.hadoop.mapred.Merger -
Down to the last merge-pass, with 1 segments left of total size: 57 bytes
2014-11-06 01:04:37,432 [Thread-15] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:37,448 [Thread-15] WARN
org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been
initialized
2014-11-06 01:04:37,452 [Thread-15] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce
- Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4]
C: R:
2014-11-06 01:04:37,452 [Thread-15] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0004_r_000000_0 is done. And is in the process of commiting
2014-11-06 01:04:37,453 [Thread-15] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2014-11-06 01:04:37,453 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task
attempt_local_0004_r_000000_0 is allowed to commit now
2014-11-06 01:04:37,455 [Thread-15] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of
task 'attempt_local_0004_r_000000_0' to file:/tmp/temp-2074079481/tmp-1319698666
2014-11-06 01:04:37,866 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 87% complete
2014-11-06 01:04:37,866 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Running jobs are [job_local_0004]
2014-11-06 01:04:40,422 [Thread-15] INFO
org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2014-11-06 01:04:40,423 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task
'attempt_local_0004_r_000000_0' done.
2014-11-06 01:04:40,874 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2014-11-06 01:04:40,882 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.4 0.14.0-SNAPSHOT eyal 2014-11-06 01:04:11 2014-11-06 01:04:40
COGROUP,GROUP_BY,ORDER_BY,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime
MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime
MedianReducetime Alias Feature Outputs
job_local_0001 1 1 n/a n/a n/a n/a n/a n/a
n/a n/a a,b,c,d,e COGROUP
job_local_0002 1 1 n/a n/a n/a n/a n/a n/a
n/a n/a g,h GROUP_BY,COMBINER
job_local_0003 1 1 n/a n/a n/a n/a n/a n/a
n/a n/a i SAMPLER
job_local_0004 1 1 n/a n/a n/a n/a n/a n/a
n/a n/a i ORDER_BY
file:/tmp/temp-2074079481/tmp-1319698666,
Input(s):
Successfully read 4 records from:
"/tmp/pig_junit_tmp2035031198/tmp5809809739981971182student"
Successfully read 3 records from:
"/tmp/pig_junit_tmp2035031198/tmp2814450914960679041voter"
Output(s):
Successfully stored 2 records in: "file:/tmp/temp-2074079481/tmp-1319698666"
Counters:
Total records written : 2
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_local_0001 -> job_local_0002,
job_local_0002 -> job_local_0003,
job_local_0003 -> job_local_0004,
job_local_0004
2014-11-06 01:04:40,894 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
2014-11-06 01:04:40,898 [main] WARN org.apache.pig.data.SchemaTupleBackend -
SchemaTupleBackend has already been initialized
2014-11-06 01:04:40,900 [main] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2014-11-06 01:04:40,900 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2014-11-06 01:04:40,931 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: file:///
2014-11-06 01:04:41,049 [main] WARN org.apache.pig.newplan.BaseOperatorPlan -
Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s).
------------- ---------------- ---------------
Testcase: testCheckin1 took 30.624 sec
Testcase: testCheckin2 took 0.154 sec
Caused an ERROR
Unable to store alias i
org.apache.pig.PigException: ERROR 1002: Unable to store alias i
at org.apache.pig.PigServer.storeEx(PigServer.java:1038)
at org.apache.pig.PigServer.store(PigServer.java:997)
at org.apache.pig.PigServer.store(PigServer.java:965)
at org.apache.pig.test.TestCommit.testCheckin2(TestCommit.java:103)
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output
Location Validation Failed for: 'testCheckin2-output.txt More info to follow:
Output directory testCheckin2-output.txt already exists
at
org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:95)
at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at
org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1767)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1443)
at org.apache.pig.PigServer.storeEx(PigServer.java:1033)
Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output
directory testCheckin2-output.txt already exists
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at
org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)
> Missing required library: 'build/ivy/lib/Pig/javacc-4.2.jar' pig
> ----------------------------------------------------------------
>
> Key: PIG-3399
> URL: https://issues.apache.org/jira/browse/PIG-3399
> Project: Pig
> Issue Type: Bug
> Components: build
> Affects Versions: 0.11.1
> Reporter: vikram s
> Attachments: PIG-3399-1.patch, PIG-3399-2.patch
>
>
> 1.Got latest version by git pull
> 2.mvn clean eclipse-files
> 3.Imported to eclipse
> 4.Got error Missing required library: 'build/ivy/lib/Pig/javacc-4.2.jar'
> pig
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
