Hi,
I am trying to run the first pig sample from the Hadoop - The definitive guide
- book.
Unfortunately that doesn't work for me.
I downloaded 0.12.0 and got the impression it should work with Hadoop 2.2.
>http://pig.apache.org/releases.html#14+October%2C+2013%3A+release+0.12.0+available
>14 October, 2013: release 0.12.0 available
>This release include several new features such as ASSERT operator, Streaming
>UDF, new AvroStorage, IN/CASE >operator, BigInteger/BigDecimal data type,
>support for Windows.
>Note
>This release works with Hadoop 0.20.X, 1.X, 0.23.X and 2.X
I use Hadoop 2.x.
>snow:bin mkamp$ which hadoop
>/Users/mkamp/hadoop-2.2.0/bin//hadoop
>snow:bin mkamp$ echo $HADOOP_HOME
>/Users/mkamp/hadoop-2.2.0
But no matter if HADOOP_HOME is set or not I get a couple of errors and it
doesn't work if I run the script:
>records = LOAD 'micro-tab/sample.txt'
>AS (year:chararray, temperature:int, quality:int);
>DUMP records;
All hell breaks lose and there is a lot of output, but most seems meaningless,
warnings about settings that are deprecated in Hadoop, but still delivered by
default this way.
Hard to say what is relevant. Here are some excerpts, full output attached as
file.
From the logfile:
>Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found
>interface >org.apache.hadoop.mapreduce.JobContext, but class was expected
>at
>>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigO>utputFormat.java:225)
>ERROR 1066: Unable to open iterator for alias records
>From the console:
>2014-01-09 22:24:45,976 [main] WARN
>org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
>>JobControl (not using hadoop 0.20 ?)
>java.lang.NoSuchFieldException: runnerState
>at java.lang.Class.getDeclaredField(Class.java:1918)
But as a little googling indicated, this is business as usual?
>2014-01-09 22:24:49,228 [JobControl] ERROR
>org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error >while trying to
>run jobs.
>java.lang.IncompatibleClassChangeError: Found interface
>org.apache.hadoop.mapreduce.JobContext, but class >was expected
>Input(s):
>Failed to read data from "hdfs://localhost/user/mkamp/micro-tab/sample.txt"
That last one looks interesting. Maybe I am using it wrong and the reported
errors are not related? I wanted to read from the local file system.
So I also changed the script to read from hdfs, but that didn't change the
error.
Any ideas where to go from here?
Is it possible to run the latest Hadoop binary download and the latest Pig
binary download together?
snow:ncdc mkamp$ pig ~/ma/test.pig
2014-01-09 22:24:43,855 [main] INFO org.apache.pig.Main - Apache Pig version
0.12.0 (r1529718) compiled Oct 07 2013, 12:20:14
2014-01-09 22:24:43,856 [main] INFO org.apache.pig.Main - Logging error
messages to: /Users/mkamp/ncdc/pig_1389302683853.log
2014-01-09 22:24:44.100 java[26715:1003] Unable to load realm info from
SCDynamicStore
2014-01-09 22:24:44,109 [main] WARN org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2014-01-09 22:24:44,381 [main] INFO org.apache.pig.impl.util.Utils - Default
bootup file /Users/mkamp/.pigbootup not found
2014-01-09 22:24:44,585 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address
2014-01-09 22:24:44,586 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2014-01-09 22:24:44,586 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: hdfs://localhost
2014-01-09 22:24:44,589 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.used.genericoptionsparser is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used
2014-01-09 22:24:45,027 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
map-reduce job tracker at: localhost:8021
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use
mapreduce.jobtracker.maxtasks.perjob
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.system.dir is
deprecated. Instead, use mapreduce.jobtracker.system.dir
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.max.attempts
is deprecated. Instead, use mapreduce.reduce.maxattempts
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use
mapreduce.tasktracker.map.tasks.maximum
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.local.dir.minspacekill is deprecated. Instead, use
mapreduce.tasktracker.local.dir.minspacekill
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.jobtracker.job.history.block.size is deprecated. Instead, use
mapreduce.jobtracker.jobhistory.block.size
2014-01-09 22:24:45,566 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.backup.address is
deprecated. Instead, use dfs.namenode.backup.address
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.name.edits.dir is
deprecated. Instead, use dfs.namenode.edits.dir
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.task.timeout is
deprecated. Instead, use mapreduce.task.timeout
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.task.tracker.task-controller is deprecated. Instead, use
mapreduce.tasktracker.taskcontroller
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.local.dir.minspacestart is deprecated. Instead, use
mapreduce.tasktracker.local.dir.minspacestart
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
jobclient.progress.monitor.poll.interval is deprecated. Instead, use
mapreduce.client.progressmonitor.pollinterval
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.shuffle.merge.percent is deprecated. Instead, use
mapreduce.reduce.shuffle.merge.percent
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.block.size is
deprecated. Instead, use dfs.blocksize
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.df.interval is
deprecated. Instead, use fs.df.interval
2014-01-09 22:24:45,567 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.skip.reduce.max.skip.groups is deprecated. Instead, use
mapreduce.reduce.skip.maxgroups
2014-01-09 22:24:45,568 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.map.max.attempts is
deprecated. Instead, use mapreduce.map.maxattempts
2014-01-09 22:24:45,568 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile is
deprecated. Instead, use mapreduce.task.profile
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit
is deprecated. Instead, use mapreduce.job.counters.max
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.child.tmp is
deprecated. Instead, use mapreduce.task.tmp.dir
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.replication.min is
deprecated. Instead, use dfs.namenode.replication.min
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.safemode.threshold.pct
is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
dfs.https.client.keystore.resource is deprecated. Instead, use
dfs.client.https.keystore.resource
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use
mapreduce.tasktracker.reduce.tasks.maximum
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.limit.kb is
deprecated. Instead, use mapreduce.task.userlog.limit.kb
2014-01-09 22:24:45,569 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
2014-01-09 22:24:45,570 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.reduce.parallel.copies is deprecated. Instead, use
mapreduce.reduce.shuffle.parallelcopies
2014-01-09 22:24:45,570 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
2014-01-09 22:24:45,570 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.skip.attempts.to.start.skipping is deprecated. Instead, use
mapreduce.task.skip.start.attempts
2014-01-09 22:24:45,570 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.min.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-01-09 22:24:45,570 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
jobclient.completion.poll.interval is deprecated. Instead, use
mapreduce.client.completion.pollinterval
2014-01-09 22:24:45,571 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - jobclient.output.filter is
deprecated. Instead, use mapreduce.client.output.filter
2014-01-09 22:24:45,571 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use
mapreduce.job.committer.setup.cleanup.needed
2014-01-09 22:24:45,571 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.max.objects is
deprecated. Instead, use dfs.namenode.max.objects
2014-01-09 22:24:45,571 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2014-01-09 22:24:45,571 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.expiry.interval is deprecated. Instead, use
mapreduce.jobtracker.expire.trackers.interval
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.datanode.max.xcievers is
deprecated. Instead, use dfs.datanode.max.transfer.threads
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.submit.replication is
deprecated. Instead, use mapreduce.client.submit.file.replication
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.jobtracker.taskScheduler is deprecated. Instead, use
mapreduce.jobtracker.taskscheduler
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.temp.dir is
deprecated. Instead, use mapreduce.cluster.temp.dir
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated.
Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.output.compression.codec is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.codec
2014-01-09 22:24:45,572 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.retain.hours
is deprecated. Instead, use mapreduce.job.userlog.retain.hours
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.healthChecker.interval is deprecated. Instead, use
mapreduce.tasktracker.healthchecker.interval
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use
mapreduce.jobtracker.retiredjobs.cache.size
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.max.tracker.failures
is deprecated. Instead, use mapreduce.job.maxtaskfailures.per.tracker
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.replication.considerLoad
is deprecated. Instead, use dfs.namenode.replication.considerLoad
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.acls.enabled is
deprecated. Instead, use mapreduce.cluster.acls.enabled
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.reduce.slowstart.completed.maps is deprecated. Instead, use
mapreduce.job.reduce.slowstart.completedmaps
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.handler.count is deprecated. Instead, use
mapreduce.jobtracker.handler.count
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.attempts is
deprecated. Instead, use mapreduce.job.end-notification.retry.attempts
2014-01-09 22:24:45,573 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.http.address is
deprecated. Instead, use dfs.namenode.http-address
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.dir
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.name.dir.restore is
deprecated. Instead, use dfs.namenode.name.dir.restore
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.https.address is
deprecated. Instead, use dfs.namenode.https-address
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.healthChecker.script.timeout is deprecated. Instead, use
mapreduce.tasktracker.healthchecker.script.timeout
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.shuffle.connect.timeout is deprecated. Instead, use
mapreduce.reduce.shuffle.connect.timeout
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.backup.http.address is
deprecated. Instead, use dfs.namenode.backup.http-address
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.secondary.http.address
is deprecated. Instead, use dfs.namenode.secondary.http-address
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.umaskmode is deprecated.
Instead, use fs.permissions.umask-mode
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
2014-01-09 22:24:45,574 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.replication.interval is
deprecated. Instead, use dfs.namenode.replication.interval
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.name.dir is deprecated.
Instead, use dfs.namenode.name.dir
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.indexcache.mb is deprecated. Instead, use
mapreduce.tasktracker.indexcache.mb
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - keep.failed.task.files is
deprecated. Instead, use mapreduce.task.files.preserve.failedtasks
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.heartbeats.in.second
is deprecated. Instead, use mapreduce.jobtracker.heartbeats.in.second
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.permissions is
deprecated. Instead, use dfs.permissions.enabled
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use
mapreduce.job.speculative.slowtaskthreshold
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.checkpoint.dir is
deprecated. Instead, use dfs.namenode.checkpoint.dir
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.dns.interface is deprecated. Instead, use
mapreduce.tasktracker.dns.interface
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use
mapreduce.job.speculative.slownodethreshold
2014-01-09 22:24:45,575 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.https.need.client.auth
is deprecated. Instead, use dfs.client.https.need-auth
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.checkpoint.edits.dir is
deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.hours
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use
mapreduce.job.jvm.numtasks
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
topology.node.switch.mapping.impl is deprecated. Instead, use
net.topology.node.switch.mapping.impl
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.task.cache.levels is
deprecated. Instead, use mapreduce.jobtracker.taskcache.levels
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.instrumentation is deprecated. Instead, use
mapreduce.tasktracker.instrumentation
2014-01-09 22:24:45,576 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.access.time.precision is
deprecated. Instead, use dfs.namenode.accesstime.precision
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.job.queue.name is
deprecated. Instead, use mapreduce.job.queuename
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.reduce.child.log.level is deprecated. Instead, use
mapreduce.reduce.log.level
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.balance.bandwidthPerSec
is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.active
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.map.output.compression.codec is deprecated. Instead, use
mapreduce.map.output.compress.codec
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.task.tracker.http.address is deprecated. Instead, use
mapreduce.tasktracker.http.address
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapreduce.jobtracker.split.metainfo.maxsize is deprecated. Instead, use
mapreduce.job.split.metainfo.maxsize
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.reduces
is deprecated. Instead, use mapreduce.task.profile.reduces
2014-01-09 22:24:45,577 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.inmem.merge.threshold
is deprecated. Instead, use mapreduce.reduce.merge.inmem.threshold
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.safemode.extension is
deprecated. Instead, use dfs.namenode.safemode.extension
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.checkpoint.period is
deprecated. Instead, use dfs.namenode.checkpoint.period
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is
deprecated. Instead, use dfs.bytes-per-checksum
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.data.dir is deprecated.
Instead, use dfs.datanode.data.dir
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir is
deprecated. Instead, use mapreduce.cluster.local.dir
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.interval is
deprecated. Instead, use mapreduce.job.end-notification.retry.interval
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.output.compression.type is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.type
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - io.sort.spill.percent is
deprecated. Instead, use mapreduce.map.sort.spill.percent
2014-01-09 22:24:45,578 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.permissions.supergroup
is deprecated. Instead, use dfs.permissions.superusergroup
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - tasktracker.http.threads is
deprecated. Instead, use mapreduce.tasktracker.http.threads
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.compress.map.output
is deprecated. Instead, use mapreduce.map.output.compress
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.reduce.input.buffer.percent is deprecated. Instead, use
mapreduce.reduce.input.buffer.percent
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, use
mapreduce.tasktracker.tasks.sleeptimebeforesigkill
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.tasktracker.dns.nameserver is deprecated. Instead, use
mapreduce.tasktracker.dns.nameserver
2014-01-09 22:24:45,579 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.read.timeout
is deprecated. Instead, use mapreduce.reduce.shuffle.read.timeout
2014-01-09 22:24:45,580 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.max.tracker.blacklists is deprecated. Instead, use
mapreduce.jobtracker.tasktracker.maxblacklists
2014-01-09 22:24:45,580 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - topology.script.number.args
is deprecated. Instead, use net.topology.script.number.args
2014-01-09 22:24:45,580 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use
mapreduce.reduce.shuffle.input.buffer.percent
2014-01-09 22:24:45,580 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.merge.recordsBeforeProgress is deprecated. Instead, use
mapreduce.task.merge.progress.records
2014-01-09 22:24:45,580 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - dfs.write.packet.size is
deprecated. Instead, use dfs.client-write-packet-size
2014-01-09 22:24:45,581 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.jobtracker.restart.recover is deprecated. Instead, use
mapreduce.jobtracker.restart.recover
2014-01-09 22:24:45,581 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
2014-01-09 22:24:45,581 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use
mapreduce.jobtracker.jobhistory.lru.cache.size
2014-01-09 22:24:45,581 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.map.child.log.level
is deprecated. Instead, use mapreduce.map.log.level
2014-01-09 22:24:45,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.task.tracker.report.address is deprecated. Instead, use
mapreduce.tasktracker.report.address
2014-01-09 22:24:45,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.speculative.execution.speculativeCap is deprecated. Instead, use
mapreduce.job.speculative.speculativecap
2014-01-09 22:24:45,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.skip.map.max.skip.records is deprecated. Instead, use
mapreduce.map.skip.maxrecords
2014-01-09 22:24:45,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.maps is
deprecated. Instead, use mapreduce.task.profile.maps
2014-01-09 22:24:45,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.jobtracker.instrumentation is deprecated. Instead, use
mapreduce.jobtracker.instrumentation
2014-01-09 22:24:45,649 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig features used in the script: UNKNOWN
2014-01-09 22:24:45,706 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite,
GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer,
LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer,
PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter,
StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-01-09 22:24:45,885 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
2014-01-09 22:24:45,917 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2014-01-09 22:24:45,917 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2014-01-09 22:24:45,946 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - session.id is deprecated.
Instead, use dfs.metrics.session-id
2014-01-09 22:24:45,948 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics -
Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-01-09 22:24:45,976 [main] WARN
org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1918)
at
org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:98)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:287)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-01-09 22:24:45,979 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig script settings are added to the job
2014-01-09 22:24:45,986 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-01-09 22:24:45,988 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job5125392973250111582.jar
2014-01-09 22:24:49,133 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job5125392973250111582.jar created
2014-01-09 22:24:49,133 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated.
Instead, use mapreduce.job.jar
2014-01-09 22:24:49,150 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-01-09 22:24:49,161 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Key [pig.schematuple] is false, will not generate code.
2014-01-09 22:24:49,161 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Starting process to move generated code to distributed cache
2014-01-09 22:24:49,163 [main] INFO org.apache.pig.data.SchemaTupleFrontend -
Setting key [pig.schematuple.classes] with classes to deserialize []
2014-01-09 22:24:49,163 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2014-01-09 22:24:49,200 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2014-01-09 22:24:49,221 [JobControl] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
2014-01-09 22:24:49,228 [JobControl] ERROR
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to
run jobs.
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at
org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:695)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
2014-01-09 22:24:49,228 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2014-01-09 22:24:49,234 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop
immediately on failure.
2014-01-09 22:24:49,235 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job null has failed! Stop running all dependent jobs
2014-01-09 22:24:49,235 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2014-01-09 22:24:49,247 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
exception from backend error: Unexpected System Error Occured:
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at
org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:695)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
2014-01-09 22:24:49,248 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil
- 1 map reduce job(s) failed!
2014-01-09 22:24:49,249 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.2.0 0.12.0 mkamp 2014-01-09 22:24:45 2014-01-09 22:24:49 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A records MAP_ONLY Message: Unexpected System Error Occured:
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at
org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:695)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
hdfs://localhost/tmp/temp1899281183/tmp903904220,
Input(s):
Failed to read data from "hdfs://localhost/user/mkamp/micro-tab/sample.txt"
Output(s):
Failed to produce result in "hdfs://localhost/tmp/temp1899281183/tmp903904220"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null
2014-01-09 22:24:49,250 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2014-01-09 22:24:49,256 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
1066: Unable to open iterator for alias records
Details at logfile: /Users/mkamp/ncdc/pig_1389302683853.log1950 0 1
1950 22 1
1950 -11 1
1949 111 1
1949 78 1
