custom generate spark application id

[rtijoriwala]

Hi,

We would like to control how spark generates its application id. Currently,
it changes everytime we restart the job and also hard to correlate. For e.g.
it looks like this - app-20161129054045-0096.

I would like to control how this id gets generated to its easier to track
when jmx metrics are reported to graphite.

Any help appreciated.

Thanks,

Ritesh



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/custom-generate-spark-application-id-tp28160.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

map vs mapPartitions

[rtijoriwala]

Hi All,
I am a newbie to spark and want to know if there is any performance
difference between map vs mapPartitions if I am doing strictly a per item
transformation?

For e.g.

reversedWords = words.map(w => w.reverse()); 
vs.
reversedWords = words.mapPartitions(pwordsIterator => {
List pWordList = new ArrayList();

while(pwordsIterator.hasNext()) {
pWordList.add(pwordsIterator.next().reverse());
}
return pWordList
}

Is the instantiation of "pWordList" for mapPartitions justified for any
performance gain?

Thanks!



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/map-vs-mapPartitions-tp27490.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Writing all values for same key to one file

[rtijoriwala]

Hi Colzer,
Thanks for the response. My main question was about writing one file per
"key" i.e. have a file with all values for a given key. So in the pseudo
code that I have above, am I opening/creating the file in the right place?.
Once the file is created and closed, I cannot append to it.

Thanks,
Ritesh



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Writing-all-values-for-same-key-to-one-file-tp27455p27485.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Writing all values for same key to one file

[rtijoriwala]

Any recommendations? comments?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Writing-all-values-for-same-key-to-one-file-tp27455p27480.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org