答复: 答复: spark append files to the same hdfs dir issue for LeaseExpiredException
Thanks for your email My situation I, there is a hive table partitioned by five minutes, I want to write data every 30s into the hdfs location where the table located. So I when the first batch is delay, then the next batch may have the chance to touch the _SUCCESS file at the same time. Then may be crash for spark for exception. 发件人: Charles O. Bajomo [mailto:charles.baj...@pretechconsulting.co.uk] 发送时间: 2017年2月28日 20:10 收件人: 邓刚[产品技术中心] 抄送: user; d...@spark.apache.org 主题: Re: 答复: spark append files to the same hdfs dir issue for LeaseExpiredException Unless this is a managed hive table I would expect you can just MSCK REPAIR the table to get the new partition. of course you will need to change the schema to reflect the new partition Kind Regards From: "Triones,Deng(vip.com)" mailto:triones.d...@vipshop.com>> To: "Charles O. Bajomo" mailto:charles.baj...@pretechconsulting.co.uk>> Cc: "user" mailto:user@spark.apache.org>>, d...@spark.apache.org<mailto:d...@spark.apache.org> Sent: Tuesday, 28 February, 2017 10:47:47 Subject: 答复: spark append files to the same hdfs dir issue for LeaseExpiredException I am writing data to hdfs file, also the hdfs dir is a hive partition file dir. Hive does not support sub dirs.. for example my partition folder is ***/dt=20170224/hm=1400 that means I need to write all the data between 1400 to 1500 to the same folder. 发件人: Charles O. Bajomo [mailto:charles.baj...@pretechconsulting.co.uk] 发送时间: 2017年2月28日 18:04 收件人: 邓刚[产品技术中心] 抄送: user; d...@spark.apache.org<mailto:d...@spark.apache.org> 主题: Re: spark append files to the same hdfs dir issue for LeaseExpiredException I see this problem as well with the _temporary directory but from what I have been able to gather, there is no way around it in that situation apart from making sure all reducers write to different folders. In the past I partitioned by executor id. I don't know if this is the best way though. Kind Regards From: "Triones,Deng(vip.com)" mailto:triones.d...@vipshop.com>> To: "user" mailto:user@spark.apache.org>>, d...@spark.apache.org<mailto:d...@spark.apache.org> Sent: Tuesday, 28 February, 2017 09:35:00 Subject: spark append files to the same hdfs dir issue for LeaseExpiredException Hi dev and users Now I am running spark streaming , (spark version 2.0.2) to write file to hdfs. When my spark.streaming.concurrentJobs is more than one. Like 20. I meet the exception as below. We know that when the batch finished, there will be a _SUCCESS file. As I guess my spark application, if one batch is slow, and the another one run at the same time, two spark streaming batch may be try to make use of the _SUCCESS file at the same time. So the error as below happened Anyone knows that whether I am right. Or any suggestions to avoid this problem? Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on ***/dt=20170224/hm=1400 /_SUCCESS (inode 17483293037): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_**_*, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3362) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3450) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3420) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:691) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.complete(AuthorizationProviderProxyClientProtocol.java:219) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:520) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.i
答复: spark append files to the same hdfs dir issue for LeaseExpiredException
I am writing data to hdfs file, also the hdfs dir is a hive partition file dir. Hive does not support sub dirs.. for example my partition folder is ***/dt=20170224/hm=1400 that means I need to write all the data between 1400 to 1500 to the same folder. 发件人: Charles O. Bajomo [mailto:charles.baj...@pretechconsulting.co.uk] 发送时间: 2017年2月28日 18:04 收件人: 邓刚[产品技术中心] 抄送: user; d...@spark.apache.org 主题: Re: spark append files to the same hdfs dir issue for LeaseExpiredException I see this problem as well with the _temporary directory but from what I have been able to gather, there is no way around it in that situation apart from making sure all reducers write to different folders. In the past I partitioned by executor id. I don't know if this is the best way though. Kind Regards From: "Triones,Deng(vip.com)" mailto:triones.d...@vipshop.com>> To: "user" mailto:user@spark.apache.org>>, d...@spark.apache.org<mailto:d...@spark.apache.org> Sent: Tuesday, 28 February, 2017 09:35:00 Subject: spark append files to the same hdfs dir issue for LeaseExpiredException Hi dev and users Now I am running spark streaming , (spark version 2.0.2) to write file to hdfs. When my spark.streaming.concurrentJobs is more than one. Like 20. I meet the exception as below. We know that when the batch finished, there will be a _SUCCESS file. As I guess my spark application, if one batch is slow, and the another one run at the same time, two spark streaming batch may be try to make use of the _SUCCESS file at the same time. So the error as below happened Anyone knows that whether I am right. Or any suggestions to avoid this problem? Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on ***/dt=20170224/hm=1400 /_SUCCESS (inode 17483293037): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_**_*, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3362) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3450) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3420) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:691) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.complete(AuthorizationProviderProxyClientProtocol.java:219) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:520) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy28.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy29.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:319) at org.apache.parquet.hadoop.ParquetOutputCommitt
spark append files to the same hdfs dir issue for LeaseExpiredException
Hi dev and users Now I am running spark streaming , (spark version 2.0.2) to write file to hdfs. When my spark.streaming.concurrentJobs is more than one. Like 20. I meet the exception as below. We know that when the batch finished, there will be a _SUCCESS file. As I guess my spark application, if one batch is slow, and the another one run at the same time, two spark streaming batch may be try to make use of the _SUCCESS file at the same time. So the error as below happened Anyone knows that whether I am right. Or any suggestions to avoid this problem? Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on ***/dt=20170224/hm=1400 /_SUCCESS (inode 17483293037): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_**_*, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3362) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3450) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3420) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:691) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.complete(AuthorizationProviderProxyClientProtocol.java:219) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:520) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy28.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:404) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy29.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2116) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2100) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:319) at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46) at org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:222) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelationCommand.scala:144) ... 40 more Thanks Triones 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without
撤回: How to deal with string column data for spark mlib?
邓刚[技术中心] 将撤回邮件“How to deal with string column data for spark mlib?”。 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
答复: How to deal with string column data for spark mlib?
Hi spark dev, I am using spark 2 to write orc file to hdfs. I have one questions about the savemode. My use case is this. When I write data into hdfs. If one task failed I hope the file that the task created should be delete and the retry task can write all data, that is to say, If I have the data 1 to 100 in this task, when the task which write 1 to 100 failed at first, then the task scheduler reschedule the partition task , the data in hdfs should only have the data 1 to 100. Not double 1 and so on. If so which kind of savemode should I use. I the FileFormatWriter.scala the file name rule contains one UUID,so I am in mistake.. Thanks Triones 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
question about the data frame save mode to make the data exactly one
Hi spark dev, I am using spark 2 to write orc file to hdfs. I have one questions about the savemode. My use case is this. When I write data into hdfs. If one task failed I hope the file that the task created should be delete and the retry task can write all data, that is to say, If I have the data 1 to 100 in this task, when the task which write 1 to 100 failed at first, then the task scheduler reschedule the partition task , the data in hdfs should only have the data 1 to 100. Not double 1 and so on. If so which kind of savemode should I use. I the FileFormatWriter.scala the file name rule contains one UUID,so I am in mistake.. Thanks Triones 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
答复: 答复: 答复: 答复: spark streaming context trigger invoke stop why?
$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:42)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:34)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:218)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:218)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:218)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:217)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Could not compute split, block
input-22-1452641669000 not found
at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
... 3 more
16/01/13 07:35:42 INFO
[org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking
stop(stopGracefully=false) from shutdown hook
16/01/13 07:35:42 INFO
[org.apache.spark.streaming.scheduler.ReceiverTracker---sparkDriver-akka.actor.default-dispatcher-4]:
Sent stop signal to all 42 receivers
发件人: Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com]
发送时间: 2016年1月16日 6:28
收件人: 邓刚[技术中心]
抄送: Yogesh Mahajan; user
主题: Re: 答复: 答复: 答复: spark streaming context trigger invoke stop why?
I see. So when your job fails, `jsc.awaitTermination();` will throw an
exception. Then you app main method will exit and trigger the shutdown hook and
call `jsc.stop()`.
On Thu, Jan 14, 2016 at 10:20 PM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
Thanks for your response .
Our code as below :
public void process(){
logger.info<http://logger.info>("streaming process start !!!");
SparkConf sparkConf = createSparkConf(this.getClass().getSimpleName());
JavaStreamingContext jsc = this.createJavaStreamingContext(sparkConf);
if(this.streamingListener != null){
jsc.addStreamingListener(this.streamingListener);
}
JavaPairDStream allKafkaWindowData =
this.sparkReceiverDStream.createReceiverDStream(jsc,this.streamingConf.getWindowDuration(),
this.streamingConf.getSlideDuration());
this.businessProcess(allKafkaWindowData);
this.sleep();
jsc.start();
jsc.awaitTermination();
发件人: Shixiong(Ryan) Zhu
[mailto:shixi...@databricks.com<mailto:shixi...@databricks.com>]
发送时间: 2016年1月15日 6:02
收件人: 邓刚[技术中心]
抄送: Yogesh Mahajan; user
主题: Re: 答复: 答复: spark streaming context trigger invoke stop why?
Could you show your codes? Did you use `StreamingContext.awaitTermination`? If
so, it will return if any exception happens.
On Wed, Jan 13, 2016 at 11:47 PM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by
myself. So I believe somewhere of
答复: 答复: 答复: spark streaming context trigger invoke stop why?
Thanks for your response .
Our code as below :
public void process(){
logger.info("streaming process start !!!");
SparkConf sparkConf = createSparkConf(this.getClass().getSimpleName());
JavaStreamingContext jsc = this.createJavaStreamingContext(sparkConf);
if(this.streamingListener != null){
jsc.addStreamingListener(this.streamingListener);
}
JavaPairDStream allKafkaWindowData =
this.sparkReceiverDStream.createReceiverDStream(jsc,this.streamingConf.getWindowDuration(),
this.streamingConf.getSlideDuration());
this.businessProcess(allKafkaWindowData);
this.sleep();
jsc.start();
jsc.awaitTermination();
发件人: Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com]
发送时间: 2016年1月15日 6:02
收件人: 邓刚[技术中心]
抄送: Yogesh Mahajan; user
主题: Re: 答复: 答复: spark streaming context trigger invoke stop why?
Could you show your codes? Did you use `StreamingContext.awaitTermination`? If
so, it will return if any exception happens.
On Wed, Jan 13, 2016 at 11:47 PM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by
myself. So I believe somewhere of the driver kill itself
发件人: 邓刚[技术中心]
发送时间: 2016年1月14日 15:45
收件人: 'Yogesh Mahajan'
抄送: user
主题: 答复: 答复: spark streaming context trigger invoke stop why?
Thanks for your response, ApplicationMaster is only for yarn mode. I am using
standalone mode. Could you kindly please let me know where trigger the shutdown
hook?
发件人: Yogesh Mahajan [mailto:ymaha...@snappydata.io]
发送时间: 2016年1月14日 12:42
收件人: 邓刚[技术中心]
抄送: user
主题: Re: 答复: spark streaming context trigger invoke stop why?
All the action happens in ApplicationMaster expecially in run method
Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes
ApplicationMaster#finish method. You can also try System.exit in your program
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io/>
On Thu, Jan 14, 2016 at 9:56 AM, Yogesh Mahajan
mailto:ymaha...@snappydata.io>> wrote:
Hi Triones,
Check the org.apache.spark.util.ShutdownHookManager : It adds this ShutDownHook
when you start a StreamingContext
Here is the code in StreamingContext.start()
shutdownHookRef = ShutdownHookManager.addShutdownHook(
StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)
Also looke at the following def in StreamingContext which actually stops the
context from shutdown hook :
private def stopOnShutdown(): Unit = {
val stopGracefully =
conf.getBoolean("spark.streaming.stopGracefullyOnShutdown", false)
logInfo(s"Invoking stop(stopGracefully=$stopGracefully) from shutdown hook")
// Do not stop SparkContext, let its own shutdown hook stop it
stop(stopSparkContext = false, stopGracefully = stopGracefully)
}
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io>
On Thu, Jan 14, 2016 at 8:55 AM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
More info
I am using spark version 1.5.2
发件人: Triones,Deng(vip.com<http://vip.com>)
[mailto:triones.d...@vipshop.com<mailto:triones.d...@vipshop.com>]
发送时间: 2016年1月14日 11:24
收件人: user
主题: spark streaming context trigger invoke stop why?
Hi all
As I saw the driver log, the task failed 4 times in a stage, the stage
will be dropped when the input block was deleted before make use of. After that
the StreamingContext invoke stop. Does anyone know what kind of akka message
trigger the stop or which code trigger the shutdown hook?
Thanks
Driver log:
Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times
[org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking
stop(stopGracefully=false) from shutdown hook
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privilege
答复: 答复: spark streaming context trigger invoke stop why?
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by
myself. So I believe somewhere of the driver kill itself
发件人: 邓刚[技术中心]
发送时间: 2016年1月14日 15:45
收件人: 'Yogesh Mahajan'
抄送: user
主题: 答复: 答复: spark streaming context trigger invoke stop why?
Thanks for your response, ApplicationMaster is only for yarn mode. I am using
standalone mode. Could you kindly please let me know where trigger the shutdown
hook?
发件人: Yogesh Mahajan [mailto:ymaha...@snappydata.io]
发送时间: 2016年1月14日 12:42
收件人: 邓刚[技术中心]
抄送: user
主题: Re: 答复: spark streaming context trigger invoke stop why?
All the action happens in ApplicationMaster expecially in run method
Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes
ApplicationMaster#finish method. You can also try System.exit in your program
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io/>
On Thu, Jan 14, 2016 at 9:56 AM, Yogesh Mahajan
mailto:ymaha...@snappydata.io>> wrote:
Hi Triones,
Check the org.apache.spark.util.ShutdownHookManager : It adds this ShutDownHook
when you start a StreamingContext
Here is the code in StreamingContext.start()
shutdownHookRef = ShutdownHookManager.addShutdownHook(
StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)
Also looke at the following def in StreamingContext which actually stops the
context from shutdown hook :
private def stopOnShutdown(): Unit = {
val stopGracefully =
conf.getBoolean("spark.streaming.stopGracefullyOnShutdown", false)
logInfo(s"Invoking stop(stopGracefully=$stopGracefully) from shutdown hook")
// Do not stop SparkContext, let its own shutdown hook stop it
stop(stopSparkContext = false, stopGracefully = stopGracefully)
}
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io>
On Thu, Jan 14, 2016 at 8:55 AM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
More info
I am using spark version 1.5.2
发件人: Triones,Deng(vip.com<http://vip.com>)
[mailto:triones.d...@vipshop.com<mailto:triones.d...@vipshop.com>]
发送时间: 2016年1月14日 11:24
收件人: user
主题: spark streaming context trigger invoke stop why?
Hi all
As I saw the driver log, the task failed 4 times in a stage, the stage
will be dropped when the input block was deleted before make use of. After that
the StreamingContext invoke stop. Does anyone know what kind of akka message
trigger the stop or which code trigger the shutdown hook?
Thanks
Driver log:
Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times
[org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking
stop(stopGracefully=false) from shutdown hook
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (includi
答复: 答复: spark streaming context trigger invoke stop why?
Thanks for your response, ApplicationMaster is only for yarn mode. I am using
standalone mode. Could you kindly please let me know where trigger the shutdown
hook?
发件人: Yogesh Mahajan [mailto:ymaha...@snappydata.io]
发送时间: 2016年1月14日 12:42
收件人: 邓刚[技术中心]
抄送: user
主题: Re: 答复: spark streaming context trigger invoke stop why?
All the action happens in ApplicationMaster expecially in run method
Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes
ApplicationMaster#finish method. You can also try System.exit in your program
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io/>
On Thu, Jan 14, 2016 at 9:56 AM, Yogesh Mahajan
mailto:ymaha...@snappydata.io>> wrote:
Hi Triones,
Check the org.apache.spark.util.ShutdownHookManager : It adds this ShutDownHook
when you start a StreamingContext
Here is the code in StreamingContext.start()
shutdownHookRef = ShutdownHookManager.addShutdownHook(
StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)
Also looke at the following def in StreamingContext which actually stops the
context from shutdown hook :
private def stopOnShutdown(): Unit = {
val stopGracefully =
conf.getBoolean("spark.streaming.stopGracefullyOnShutdown", false)
logInfo(s"Invoking stop(stopGracefully=$stopGracefully) from shutdown hook")
// Do not stop SparkContext, let its own shutdown hook stop it
stop(stopSparkContext = false, stopGracefully = stopGracefully)
}
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io<http://snappydata.io>
On Thu, Jan 14, 2016 at 8:55 AM, Triones,Deng(vip.com<http://vip.com>)
mailto:triones.d...@vipshop.com>> wrote:
More info
I am using spark version 1.5.2
发件人: Triones,Deng(vip.com<http://vip.com>)
[mailto:triones.d...@vipshop.com<mailto:triones.d...@vipshop.com>]
发送时间: 2016年1月14日 11:24
收件人: user
主题: spark streaming context trigger invoke stop why?
Hi all
As I saw the driver log, the task failed 4 times in a stage, the stage
will be dropped when the input block was deleted before make use of. After that
the StreamingContext invoke stop. Does anyone know what kind of akka message
trigger the stop or which code trigger the shutdown hook?
Thanks
Driver log:
Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times
[org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking
stop(stopGracefully=false) from shutdown hook
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
This communication is intended only for the addressee(s) and may contain
information that is privileged and confidential. You are hereby notified that,
if you are not an intended recipient listed above, or an authorized employee or
agent of an addressee of this communication responsible for delivering e-mail
messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is
strictly prohibited. If you have received this communication in error, please
notify us immediately by a reply e-mail addressed to the sender and permanently
delete the original e-mail communication and any attachments from all storage
devices without making or otherwise retaining a copy.
答复: spark streaming context trigger invoke stop why?
More info I am using spark version 1.5.2 发件人: Triones,Deng(vip.com) [mailto:triones.d...@vipshop.com] 发送时间: 2016年1月14日 11:24 收件人: user 主题: spark streaming context trigger invoke stop why? Hi all As I saw the driver log, the task failed 4 times in a stage, the stage will be dropped when the input block was deleted before make use of. After that the StreamingContext invoke stop. Does anyone know what kind of akka message trigger the stop or which code trigger the shutdown hook? Thanks Driver log: Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times [org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking stop(stopGracefully=false) from shutdown hook 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy. 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
spark streaming context trigger invoke stop why?
Hi all As I saw the driver log, the task failed 4 times in a stage, the stage will be dropped when the input block was deleted before make use of. After that the StreamingContext invoke stop. Does anyone know what kind of akka message trigger the stop or which code trigger the shutdown hook? Thanks Driver log: Job aborted due to stage failure: Task 410 in stage 215.0 failed 4 times [org.apache.spark.streaming.StreamingContext---Thread-0]: Invoking stop(stopGracefully=false) from shutdown hook 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
spark straggle task
Hi All We run an application with version 1.4.1 standalone mode. We saw two tasks in one stage which runs very slow seems it is hang. We know that the JobScheduler have the function to assign the straggle task to another node. But what we saw it does not reassign. So we want to know is there anyone know that is there ant para to open this function. What's more, we saw that these two straggle tasks hangs at socket read. Is it because it cannot be interrupted so the reassign function does not work, the Thread stack as below: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:152) java.net.SocketInputStream.read(SocketInputStream.java:122) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) com.vip.logview.tsdb.HttpClientUtils.postHandler(HttpClientUtils.java:80) com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:75) com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:30) org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) org.apache.spark.scheduler.Task.run(Task.scala:70) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thanks 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
