[
https://issues.apache.org/jira/browse/HIVE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840921#comment-13840921
]
cyril liao commented on HIVE-5235:
----------------------------------
in my environment, the problem appeared when a int type column contains null
values. it was solved after i change the null values to -1 as default.
> Infinite loop with ORC file and Hive 0.11
> -----------------------------------------
>
> Key: HIVE-5235
> URL: https://issues.apache.org/jira/browse/HIVE-5235
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.11.0
> Environment: Gentoo linux with Hortonworks Hadoop
> hadoop-1.1.2.23.tar.gz and Apache Hive 0.11d
> Reporter: Iván de Prado
> Priority: Blocker
> Attachments: gendata.py
>
>
> We are using Hive 0.11 with ORC file format and we get some tasks blocked in
> some kind of infinite loop. They keep working indefinitely when we set a huge
> task expiry timeout. If we the expiry time to 600 second, the taks fail
> because of not reporting progress, and finally, the Job fails.
> That is not consistent, and some times between jobs executions the behavior
> changes. It happen for different queries.
> We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The taks
> that is blocked keeps consuming 100% of CPU usage, and the stack trace is
> always the same consistently. Everything points to some kind of infinite
> loop. My guessing is that it has some relation to the ORC file. Maybe some
> pointer is not right when writing generating some kind of infinite loop when
> reading. Or maybe there is a bug in the reading stage.
> More information below. The stack trace:
> {noformat}
> "main" prio=10 tid=0x00007f20a000a800 nid=0x1ed2 runnable [0x00007f20a8136000]
> java.lang.Thread.State: RUNNABLE
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:256)
> - locked <0x00000000f42a6ca0> (a java.util.zip.ZStreamRef)
> at
> org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.java:64)
> at
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:128)
> at
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:143)
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(SerializationUtils.java:54)
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(SerializationUtils.java:65)
> at
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(RunLengthIntegerReader.java:66)
> at
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLengthIntegerReader.java:81)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:332)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:802)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1214)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:71)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:46)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:300)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
> - eliminated <0x00000000e1459700> (a
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
> - locked <0x00000000e1459700> (a
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
> We have seen the same stack trace repeatedly for several executions of jstack.
> The log file for this kind of task is the following:
> {noformat}
> 2013-09-04 23:12:34,332 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
> the native-hadoop library
> 2013-09-04 23:12:34,681 INFO org.apache.hadoop.mapred.TaskRunner: Creating
> symlink:
> /hd/hd6/mapred/local/taskTracker/distcache/1758137359311570022_-1138434677_1812098585/master/tmp/hive-datasalt/hive_2013-09-04_22-45-13_829_2202639092470021957/-mr-10004/22027cf8-f583-41d7-adb8-e7e74922d113
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/HIVE_PLAN22027cf8-f583-41d7-adb8-e7e74922d113
> 2013-09-04 23:12:34,718 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/com
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/com
> 2013-09-04 23:12:34,733 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javolution
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javolution
> 2013-09-04 23:12:34,743 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/org
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/org
> 2013-09-04 23:12:34,756 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/job.jar
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/job.jar
> 2013-09-04 23:12:34,768 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javax
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javax
> 2013-09-04 23:12:34,779 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/hive-exec-log4j.properties
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/hive-exec-log4j.properties
> 2013-09-04 23:12:34,790 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/META-INF
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/META-INF
> 2013-09-04 23:12:34,796 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javaewah
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javaewah
> 2013-09-04 23:12:34,813 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/.job.jar.crc
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/.job.jar.crc
> 2013-09-04 23:12:35,127 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-09-04 23:12:35,412 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-09-04 23:12:35,473 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d0fa836
> 2013-09-04 23:12:36,927 INFO
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28/000798_0
> 2013-09-04 23:12:36,927 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 72
> 2013-09-04 23:12:36,936 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
> 100
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: data buffer =
> 79691776/99614720
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: record buffer
> = 262144/327680
> 2013-09-04 23:12:37,036 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy:
> Snappy native library is available
> 2013-09-04 23:12:37,036 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy:
> Snappy native library loaded
> 2013-09-04 23:12:37,046 INFO ExecMapper: maximum memory = 559284224
> 2013-09-04 23:12:37,046 INFO ExecMapper: conf classpath =
> [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
> file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/,
> file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
>
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
>
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,047 INFO ExecMapper: thread classpath =
> [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
> file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/,
> file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
>
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
>
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,085 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,089 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump
> TS
> struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,091 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-29
> 2013-09-04 23:12:37,092 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-30
> 2013-09-04 23:12:37,093 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-31
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-01
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-02
> 2013-09-04 23:12:37,095 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-03
> 2013-09-04 23:12:37,096 INFO ExecMapper:
> <MAP>Id =7
> <Children>
> <TS>Id =3
> <Children>
> <SEL>Id =2
> <Children>
> <GBY>Id =1
> <Children>
> <RS>Id =0
> <Parent>Id = 1 null<\Parent>
> <\RS>
> <\Children>
> <Parent>Id = 2 null<\Parent>
> <\GBY>
> <\Children>
> <Parent>Id = 3 null<\Parent>
> <\SEL>
> <\Children>
> <Parent>Id = 7 null<\Parent>
> <\TS>
> <\Children>
> <\MAP>
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Initializing Self 7 MAP
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 3 TS
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 3 TS initialized
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 3
> TS
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing child 2 SEL
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing Self 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> SELECT
> struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Operator 2 SEL initialized
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing children of 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing child 1 GBY
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing Self 1 GBY
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Operator 1 GBY initialized
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing children of 1 GBY
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing child 0 RS
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing Self 0 RS
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = -1
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Operator 0 RS initialized
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initialization Done 0 RS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initialization Done 1 GBY
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initialization Done 2 SEL
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 3 TS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Initialization Done 7 MAP
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Processing alias hits for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1 rows
> 2013-09-04 23:12:37,165 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1 rows
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1 rows
> 2013-09-04 23:12:37,168 INFO ExecMapper: ExecMapper: processing 1 rows: used
> memory = 151151936
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO ExecMapper: ExecMapper: processing 10 rows: used
> memory = 151151936
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO ExecMapper: ExecMapper: processing 100 rows:
> used memory = 151151936
> 2013-09-04 23:12:37,247 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1000 rows
> 2013-09-04 23:12:37,250 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000 rows
> 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1000 rows
> 2013-09-04 23:12:37,251 INFO ExecMapper: ExecMapper: processing 1000 rows:
> used memory = 151151936
> 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 10000 rows
> 2013-09-04 23:12:37,642 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10000 rows
> 2013-09-04 23:12:37,644 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 10000 rows
> 2013-09-04 23:12:37,644 INFO ExecMapper: ExecMapper: processing 10000 rows:
> used memory = 153032176
> 2013-09-04 23:12:38,467 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 100000 rows
> 2013-09-04 23:12:38,468 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100000 rows
> 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 100000 rows
> 2013-09-04 23:12:38,469 INFO ExecMapper: ExecMapper: processing 100000 rows:
> used memory = 125346272
> 2013-09-04 23:12:40,626 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO ExecMapper: ExecMapper: processing 1000000 rows:
> used memory = 123383952
> {noformat}
> For this particular example, the task was killed 6 hours after the last log
> line.
> The strace of the process when it is in the infinite loop has the following
> looking:
> {noformat}
> Process 7887 attached with 24 threads
> [pid 7930] futex(0x7f20a06df28c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid 7922] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7920] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 8333] accept(61, <unfinished ...>
> [pid 7908] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7907] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7906] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7905] futex(0x7f20a00fa554, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>
> [pid 7903] futex(0x7f20a00f5254, FUTEX_WAIT_PRIVATE, 417, NULL <unfinished
> ...>
> [pid 7904] futex(0x7f20a00f7c54, FUTEX_WAIT_PRIVATE, 387, NULL <unfinished
> ...>
> [pid 7902] futex(0x7f20a7513dc0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid 7901] futex(0x7f20a00a6f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid 7899] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7898] futex(0x7f20a0026554, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7900] futex(0x7f20a00a4b54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid 7897] futex(0x7f20a0024754, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7896] futex(0x7f20a0022a54, FUTEX_WAIT_PRIVATE, 41, NULL <unfinished
> ...>
> [pid 7895] futex(0x7f20a0020c54, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7894] futex(0x7f20a001ef54, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
> ...>
> [pid 7893] futex(0x7f20a001d154, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7887] futex(0x7f20a81389d0, FUTEX_WAIT, 7890, NULL <unfinished ...>
> [pid 7892] futex(0x7f20a001b454, FUTEX_WAIT_PRIVATE, 35, NULL <unfinished
> ...>
> [pid 7891] futex(0x7f20a0019654, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
> ...>
> [pid 7906] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373836, 993325000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 43598000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 93858000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 144101000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 194344000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 244659000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 294924000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 345198000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 395462000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 445768000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 495983000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 546226000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 596489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 646798000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 697063000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 747370000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 797688000},
> ffffffff <unfinished ...>
> [pid 7899] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 782077000},
> ffffffff <unfinished ...>
> [pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 848014000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 898283000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 948489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 998751000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 49086000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 99351000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 149589000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 199852000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 250180000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 300489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 350762000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 401078000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 451258000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 501527000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 551767000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 601970000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 652161000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 702290000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 752523000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 802739000},
> ffffffff <unfinished ...>
> [pid 7899] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 782336000},
> ffffffff <unfinished ...>
> [pid 7922] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] futex(0x7f20a0550554, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a0550550, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7907] <... restart_syscall resumed> ) = 0
> [pid 7907] futex(0x7f20a0550528, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] write(56, "\0\0\0\335\0\0\2.\0\4ping\0\0\0\2\0&org.apache.h"...,
> 225 <unfinished ...>
> [pid 7907] read(56, <unfinished ...>
> [pid 7922] <... write resumed> ) = 225
> [pid 7907] <... read resumed> 0x7f2098000b70, 8192) = -1 EAGAIN (Resource
> temporarily unavailable)
> [pid 7922] futex(0x7f20a009d054, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a009d050, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7899] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e54, FUTEX_WAIT_PRIVATE, 53, NULL <unfinished
> ...>
> [pid 7907] epoll_ctl(59, EPOLL_CTL_ADD, 56, {...} <unfinished ...>
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7907] <... epoll_ctl resumed> ) = 0
> [pid 7899] <... futex resumed> ) = 0
> [pid 7907] epoll_wait(59, {?} 0x7f20a0544000, 4096, 60000) = 1
> [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ <unfinished ...>
> [pid 7907] futex(0x7f20a0551e54, FUTEX_WAIT_PRIVATE, 11, NULL <unfinished
> ...>
> [pid 7899] <... mprotect resumed> ) = 0
> [pid 7890] futex(0x7f20a000b854, FUTEX_WAIT_PRIVATE, 349, NULL <unfinished
> ...>
> [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ|PROT_WRITE) = 0
> [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_NONE) = 0
> [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_READ) = 0
> [pid 7899] futex(0x7f20a0551e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a0551e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7899] futex(0x7f20a06d3e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a06d3e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> [pid 7899] <... futex resumed> ) = 1
> [pid 7907] futex(0x7f20a0551e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7899] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7899] <... futex resumed> ) = 1
> [pid 7922] <... futex resumed> ) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373839, 794194000},
> ffffffff <unfinished ...>
> [pid 7922] futex(0x7f20a06d3a54, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid 7907] futex(0x7f20a000b854, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a000b850, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7890] <... futex resumed> ) = 0
> [pid 7890] futex(0x7f20a000b828, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7907] epoll_ctl(59, EPOLL_CTL_DEL, 56, {...} <unfinished ...>
> [pid 7890] <... futex resumed> ) = 0
> [pid 7907] <... epoll_ctl resumed> ) = 0
> [pid 7907] epoll_wait(59, {}, 4096, 0) = 0
> [pid 7907] read(56, "\0\0\2.\0\0\0\0\0\7boolean\1", 8192) = 18
> [pid 7907] futex(0x7f20a06d3a54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a06d3a50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] futex(0x7f20a0550554,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1107, {1378373848,
> 793674000}, ffffffff <unfinished ...>
> [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] futex(0x7f20a06d3a54,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373841, 794758000},
> ffffffff <unfinished ...>
> [pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 852867000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 903052000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 953226000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 3436000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 53695000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 103966000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 154232000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 204466000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 254802000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 305067000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 355318000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 405623000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 455817000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 506130000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 556395000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 606661000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 656898000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 707177000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
