unsubscribe

[Kang]

-- 


Best Regards!

Kang 

Re: Going it alone.

[Qi Kang]

Well man, check your attitude, you’re way over the line


> On Apr 16, 2020, at 13:26, jane thorpe  wrote:
> 
> F*U*C*K O*F*F 
> C*U*N*T*S
> 
> 
> 
> On Thursday, 16 April 2020 Kelvin Qin  > wrote:
> 
> No wonder I said why I can't understand what the mail expresses, it feels 
> like a joke……
> 
> 
> 
> 
> 
> 在 2020-04-16 02:28:49，seemanto.ba...@nomura.com.INVALID 
>  写道：
> 
> Have we been tricked by a bot ?
> 
>  
> From: Matt Smith mailto:matt.smith...@gmail.com>> 
> Sent: Wednesday, April 15, 2020 2:23 PM
> To: jane thorpe
> Cc: dh.lo...@gmail.com ; user@spark.apache.org 
> ; janethor...@aol.com 
> ; em...@yeikel.com 
> Subject: Re: Going it alone.
> 
>  
> CAUTION EXTERNAL EMAIL: DO NOT CLICK ON LINKS OR OPEN ATTACHMENTS THAT ARE 
> UNEXPECTED OR SENT FROM UNKNOWN SENDERS. IF IN DOUBT REPORT TO SPAM 
> SUBMISSIONS.
> 
> This is so entertaining.
> 
>  
> 1. Ask for help
> 
> 2. Compare those you need help from to a lower order primate.
> 
> 3. Claim you provided information you did not
> 
> 4. Explain that providing any information would be "too revealing"
> 
> 5. ???
> 
>  
> Can't wait to hear what comes next, but please keep it up.  This is a bright 
> spot in my day.
> 
>  
>  
> On Tue, Apr 14, 2020 at 4:47 PM jane thorpe  > wrote:
> 
> I did write a long email in response to you.
> But then I deleted it because I felt it would be too revealing. 
> 
> 
> 
> 
> 
> On Tuesday, 14 April 2020 David Hesson  > wrote:
> 
> I want to know  if Spark is headed in my direction.
> 
> You are implying  Spark could be. 
> 
>  
> What direction are you headed in, exactly? I don't feel as if anything were 
> implied when you were asked for use cases or what problem you are solving. 
> You were asked to identify some use cases, of which you don't appear to have 
> any.
> 
>  
> On Tue, Apr 14, 2020 at 4:49 PM jane thorpe  > wrote:
> 
> That's what  I want to know,  Use Cases. 
> I am looking for  direction as I described and I want to know  if Spark is 
> headed in my direction.  
> 
> You are implying  Spark could be.
> 
> So tell me about the USE CASES and I'll do the rest.
> 
> On Tuesday, 14 April 2020 yeikel valdes  > wrote:
> 
> It depends on your use case. What are you trying to solve? 
> 
>  
> 
>  On Tue, 14 Apr 2020 15:36:50 -0400 janethor...@aol.com.INVALID 
>  wrote 
> 
> Hi, 
> 
> I consider myself to be quite good in Software Development especially using 
> frameworks.
> 
> I like to get my hands  dirty. I have spent the last few months understanding 
> modern frameworks and architectures.
> 
> I am looking to invest my energy in a product where I don't have to relying 
> on the monkeys which occupy this space  we call software development.
> 
> I have found one that meets my requirements.
> 
> Would Apache Spark be a good Tool for me or  do I need to be a member of a 
> team to develop  products  using Apache Spark  ?
> 
> 
> 
> 
>  
> PLEASE READ: This message is for the named person's use only. It may contain 
> confidential, proprietary or legally privileged information. No 
> confidentiality or privilege is waived or lost by any mistransmission. If you 
> receive this message in error, please delete it and all copies from your 
> system, destroy any hard copies and notify the sender. You must not, directly 
> or indirectly, use, disclose, distribute, print, or copy any part of this 
> message if you are not the intended recipient. Nomura Holding America Inc., 
> Nomura Securities International, Inc, and their respective subsidiaries each 
> reserve the right to monitor all e-mail communications through its networks. 
> Any views expressed in this message are those of the individual sender, 
> except where the message states otherwise and the sender is authorized to 
> state the views of such entity. Unless otherwise stated, any pricing 
> information in this message is indicative only, is subject to change and does 
> not constitute an offer to deal at any price quoted. Any reference to the 
> terms of executed transactions should be treated as preliminary only and 
> subject to our formal written confirmation.

NullPointerException at FileBasedWriteAheadLogRandomReader

[Kang Minwoo]

Hello, Users.

While I use a write-ahead logs in spark streaming, I got an error that is a 
NullPointerException at FileBasedWriteAheadLogRandomReader.scala:48[1]

[1]: 
https://github.com/apache/spark/blob/v2.4.4/streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLogRandomReader.scala#L48

 Full stack trace
Caused by: org.apache.spark.SparkException: Could not read data from write 
ahead log record 
FileBasedWriteAheadLogSegment(hdfs://.../receivedData/0/log-...,...)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org$apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:145)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:173)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD$$anonfun$compute$1.apply(WriteAheadLogBackedBlockRDD.scala:173)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.compute(WriteAheadLogBackedBlockRDD.scala:173)
//...
at 
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
at 
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
at 
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at 
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
... 3 more
Caused by: java.lang.NullPointerException
at 
org.apache.spark.streaming.util.FileBasedWriteAheadLogRandomReader.close(FileBasedWriteAheadLogRandomReader.scala:48)
at 
org.apache.spark.streaming.util.FileBasedWriteAheadLog.read(FileBasedWriteAheadLog.scala:122)
at 
org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org$apache$spark$streaming$rdd$WriteAheadLogBackedBlockRDD$$getBlockFromWriteAheadLog$1(WriteAheadLogBackedBlockRDD.scala:142)
... 50 more


- Spark version: 2.4.4
- Hadoop version: 2.7.1
- spark conf
- "spark.streaming.receiver.writeAheadLog.enable" -> "true"

Did I do something wrong?

Best regards,
Minwoo Kang

ETL and workflow management on Spark

[William Kang]

Hi,
We are moving into adopting the full stack of Spark. So far, we have used
Shark to do some ETL work, which is not bad but is not prefect either. We
ended writing UDF and UDGF, UDAF that can be avoided if we could use Pig.

Do you have any suggestions with the ETL solution in Spark stack?

And did any one have a working work flow management solution with Spark?

Many thanks.


Cao

Hadoop 2.3 Centralized Cache vs RDD

[William Kang]

Hi,
Any comments or thoughts on the implications of the newly released feature
from Hadoop 2.3 on the centralized cache? How different it is from RDD?

Many thanks.


Cao