This is using side input, yes. But it is standard practice. For example, map side joins do basically the same.
Specifically wrt opportunistic execution this should be fine. Hdfs does not disallow opening the same file for reading by multiple tasks iirc. Sent from my phone. On Oct 9, 2014 1:02 PM, "Ken Krugler" <kkrugler_li...@transpac.com> wrote: > The BtJob's BtMapper has some "interesting" logic in its setup routine, > where it looks like it's creating a side-channel: > > /* > * actually this is kind of dangerous because this routine thinks we > need > * to create file name for our current job and this will use -m- so > it's > * just serendipity we are calling it from the mapper too as the > QJob did. > */ > Path qInputPath = > new Path(qJobPath, FileOutputFormat.getUniqueFile(context, > QJob.OUTPUT_QHAT, > "")); > blockNum = context.getTaskAttemptID().getTaskID().getId(); > > SequenceFileValueIterator<DenseBlockWritable> qhatInput = > new SequenceFileValueIterator<DenseBlockWritable>(qInputPath, > true, > conf); > closeables.addFirst(qhatInput); > > It wouldn't surprise me if this fails when speculative execution is turned > on. > > -- Ken > > > > From: Dmitriy Lyubimov > > Sent: October 9, 2014 11:31:42am PDT > > To: user@mahout.apache.org > > Subject: Re: SSVD: lease conflict due to 2 attempts using the same dir > > > > wow. good to know. however, it would seem to me like a bug in > > MultipleOutputs? either way, it doesn't seem anything to do with the > Mahout > > code itself. > > > > On Thu, Oct 9, 2014 at 10:32 AM, Yang <teddyyyy...@gmail.com> wrote: > > > >> yes, that's what I'm saying, I disabled speculative, it works for now > (kind > >> of is a hack) > >> > >> > >> also yes, this is hadoop 2.0 with YARN > >> > >> this has nothing to do with overwrite mode. the 2 attempts are run > >> simultaneously because they are speculative runs > >> > >> On Wed, Oct 8, 2014 at 12:07 AM, Serega Sheypak < > serega.shey...@gmail.com> > >> wrote: > >> > >>> owned by > >>> DFSClient_attempt_1412632629877_7081_*m_000000_1_**196610264_1 > >>> *but is > >>> accessed by > >>> DFSClient_attempt_1412632629877_7081_*m_000000_0_**1051299648_1* > >>> at > >>> > >>> Looks like two clients try to write to the same dir and they are trying > >> to > >>> write to the same file. > >>> > >>> 2014-10-08 6:24 GMT+04:00 Dmitriy Lyubimov <dlie...@gmail.com>: > >>> > >>>> Usually map reduce output format would issue a much clearer > >> message.this > >>> on > >>>> the other hand looks like a name node operation conflict. > >>>> > >>>> Sent from my phone. > >>>> On Oct 7, 2014 3:23 PM, "Suneel Marthi" <smar...@apache.org> wrote: > >>>> > >>>>> Have u tried running with -ow (overwrite) option, that should clear > >> all > >>>>> tmpdir between successive runs ?? > >>>>> > >>>>> The SSVD code does clear the tmpdir when -ow is specified. > >>>>> > >>>>> On Tue, Oct 7, 2014 at 5:55 PM, Yang <teddyyyy...@gmail.com> wrote: > >>>>> > >>>>>> we are running mahout ssvd, with a --tempDir parameter, > >>>>>> but we are constantly getting this error below. > >>>>>> > >>>>>> the following seems to suggest that the 2 attempts in mapper are > >>> trying > >>>>> to > >>>>>> access the same tmp dir on hdfs. > >>>>>> > >>>>>> if this is the cause, then ssvd has a bug in that different > >> attempts > >>>>> should > >>>>>> use different temp dirs . right? > >>>>>> > >>>>>> > >>>>>> I'm trying now without the --tempDir param > >>>>>> > >>>>>> thanks > >>>>>> Yang > >>>>>> > >>>>>> Error: > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): > >>>>>> Lease mismatch on > >>>>> /user/myuser/CIReco/shoes/ssvd/tmp/ssvd/Bt-job/Q-m-00000 > >>>>>> owned by > >> DFSClient_attempt_1412632629877_7081_m_000000_1_196610264_1 > >>>> but > >>>>> is > >>>>>> accessed by > >>>> DFSClient_attempt_1412632629877_7081_m_000000_0_1051299648_1 > >>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2967) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3016) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2996) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:647) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > >>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at > >>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2020) at > >>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2016) at > >>>>>> > >>>>>> java.security.AccessController.doPrivileged(Native Method) at > >>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650) > >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2014) at > >>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1410) at > >>>>>> org.apache.hadoop.ipc.Client.call(Client.java:1363) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > >>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at > >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) > >>>>>> at com.sun.proxy.$Proxy15.complete(Unknown Source) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116) > >>>>>> at > >>>>> > >> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70) > >>>>>> at > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103) > >>>>>> at > >>>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1279) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1501) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.mapred.lib.MultipleOutputs.close(MultipleOutputs.java:537) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.mahout.common.IOUtils$MultipleOutputsCloseableAdapter.close(IOUtils.java:193) > >>>>>> at org.apache.mahout.common.IOUtils.close(IOUtils.java:128) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.cleanup(BtJob.java:295) > >>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at > >>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at > >>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at > >>>>>> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at > >>>>>> java.security.AccessController.doPrivileged(Native Method) at > >>>>>> javax.security.auth.Subject.doAs(Subject.java:415) at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650) > >>>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) > >>>>>> > >>>>> > >>>> > >>> > >> > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > > > > > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > > > > > >
