Re: Unsubscribe

[Wesley]

please send an empty email to:
dev-unsubscr...@spark.apache.org
user-unsubscr...@spark.apache.org

for unsubscribing yourself from the lists.

Thanks.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: unsubscribe

[Wesley]

please send an empty email to: user-unsubscr...@spark.apache.org for 
unsubscribing.


thanks.



unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Adding an indexed column

[Wesley Miao]

One way I can see is to -

1. get rdd from your df
2. call rdd.zipWithIndex to get a new rdd
3. turn your new rdd to a new df

On Fri, May 29, 2015 at 5:43 AM, Cesar Flores ces...@gmail.com wrote:


 Assuming that I have the next data frame:

 flag | price
 --
 1|47.808764653746
 1|47.808764653746
 1|31.9869279512204
 1|47.7907893713564
 1|16.7599200038239
 1|16.7599200038239
 1|20.3916014172137

 How can I create a data frame with an extra indexed column as the next one:

 flag | price  | index
 --|---
 1|47.808764653746 | 0
 1|47.808764653746 | 1
 1|31.9869279512204| 2
 1|47.7907893713564| 3
 1|16.7599200038239| 4
 1|16.7599200038239| 5
 1|20.3916014172137| 6

 --
 Cesar Flores

Re: Using Log4j for logging messages inside lambda functions

[Wesley Miao]

The reason it didn't work for you is that the function you registered with
someRdd.map will be running on the worker/executor side, not in your
driver's program. Then you need to be careful to not accidentally close
over some objects instantiated from your driver's program, like the log
object in your sample code above. You can look for more information online
to understand more the concept of Closure so that you can understand to
the bottom of it why it didn't work for you at first place.

The usual solution to this type of problems is to instantiate the objects
you want to use in your map functions from within your map functions. You
can define a factory object that you can create your log object from.

On Mon, May 25, 2015 at 11:05 PM, Spico Florin spicoflo...@gmail.com
wrote:

 Hello!
   I would like to use the logging mechanism provided by the log4j, but I'm
 getting the
 Exception in thread main org.apache.spark.SparkException: Task not
 serializable - Caused by: java.io.NotSerializableException:
 org.apache.log4j.Logger

 The code (and the problem) that I'm using resembles the one used here :
 http://stackoverflow.com/questions/29208844/apache-spark-logging-within-scala,
 meaning:

 val log = Logger.getLogger(getClass.getName)

   def doTest() {
val conf = new SparkConf().setMaster(local[4]).setAppName(LogTest)
val spark = new SparkContext(conf)

val someRdd = spark.parallelize(List(1, 2, 3))
someRdd.map {
  element =
*log.info http://log.info(s$element will be processed)*
element + 1
 }
 I'm posting the same problem due to the fact that the one from
 stackoverflow didn't get any answer.
 In this case, can you please tell us what is the best way to use  logging?
 Is any solution that is not using the rdd.forEachPartition?

 I look forward for your answers.
 Regards,
 Florin

Re: [ANNOUNCE] Announcing Apache Spark 2.4.4

[Wesley Peng]

on 2019/9/2 5:54, Dongjoon Hyun wrote:

We are happy to announce the availability of Spark 2.4.4!

Spark 2.4.4 is a maintenance release containing stability fixes. This
release is based on the branch-2.4 maintenance branch of Spark. We strongly
recommend all 2.4 users to upgrade to this stable release.


That's awesome. thanks for the work.

regards.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: unsubscribe

[Wesley Peng]

please send an empty email to: user-unsubscr...@spark.apache.org to 
unsubscribe yourself from the list.



Sri Kris wrote:
Sent from Mail  for 
Windows 10




-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Unsubscribe

[Wesley Peng]

Bhavya Jain wrote:

Unsubscribe


please send an email to: user-unsubscr...@spark.apache.org to 
unsubscribe yourself from the list. thanks.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org