Re: Is there anyway to log properties from a Spark application
set spark.logConf as true in spark-default.conf will log the property in driver side. But it would only log the property you set, not including the properties with default value. On Mon, Dec 28, 2015 at 8:18 PM, alvarobrandon <alvarobran...@gmail.com> wrote: > Hello: > > I was wondering if its possible to log properties from Spark Applications > like spark.yarn.am.memory, spark.driver.cores, > spark.reducer.maxSizeInFlight > without having to access the SparkConf object programmatically. I'm trying > to find some kind of log file that has traces of the execution of Spark > apps > and its parameters. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-anyway-to-log-properties-from-a-Spark-application-tp25820.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Best Regards Jeff Zhang
Is there anyway to log properties from a Spark application
Hello: I was wondering if its possible to log properties from Spark Applications like spark.yarn.am.memory, spark.driver.cores, spark.reducer.maxSizeInFlight without having to access the SparkConf object programmatically. I'm trying to find some kind of log file that has traces of the execution of Spark apps and its parameters. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-anyway-to-log-properties-from-a-Spark-application-tp25820.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there anyway to log properties from a Spark application
If you run it as yarn-client mode, it will be client side log. If it is yarn-cluster mode, it will be logged in the AM container (the first container) On Mon, Dec 28, 2015 at 8:30 PM, Alvaro Brandon <alvarobran...@gmail.com> wrote: > Thanks for the swift response. > > I'm launching my applications through YARN. Where will these properties be > logged?. I guess they wont be part of YARN logs > > 2015-12-28 13:22 GMT+01:00 Jeff Zhang <zjf...@gmail.com>: > >> set spark.logConf as true in spark-default.conf will log the property in >> driver side. But it would only log the property you set, not including the >> properties with default value. >> >> >> On Mon, Dec 28, 2015 at 8:18 PM, alvarobrandon <alvarobran...@gmail.com> >> wrote: >> >>> Hello: >>> >>> I was wondering if its possible to log properties from Spark Applications >>> like spark.yarn.am.memory, spark.driver.cores, >>> spark.reducer.maxSizeInFlight >>> without having to access the SparkConf object programmatically. I'm >>> trying >>> to find some kind of log file that has traces of the execution of Spark >>> apps >>> and its parameters. >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-anyway-to-log-properties-from-a-Spark-application-tp25820.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > -- Best Regards Jeff Zhang
Re: Is there anyway to log properties from a Spark application
Thanks for the swift response. I'm launching my applications through YARN. Where will these properties be logged?. I guess they wont be part of YARN logs 2015-12-28 13:22 GMT+01:00 Jeff Zhang <zjf...@gmail.com>: > set spark.logConf as true in spark-default.conf will log the property in > driver side. But it would only log the property you set, not including the > properties with default value. > > > On Mon, Dec 28, 2015 at 8:18 PM, alvarobrandon <alvarobran...@gmail.com> > wrote: > >> Hello: >> >> I was wondering if its possible to log properties from Spark Applications >> like spark.yarn.am.memory, spark.driver.cores, >> spark.reducer.maxSizeInFlight >> without having to access the SparkConf object programmatically. I'm trying >> to find some kind of log file that has traces of the execution of Spark >> apps >> and its parameters. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-anyway-to-log-properties-from-a-Spark-application-tp25820.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > Best Regards > > Jeff Zhang >
