Re: Spark Implicit Functions

Thanks for sharing, Bill. On Fri, Oct 16, 2015 at 2:06 PM, Bill Bejeck wrote: > All, > > I just did a post on adding groupByKeyToList and groupByKeyUnique using > implicit classes. I thought it might be useful to someone. > > http://codingjunkie.net/learning-scala-implicits-with-spark/ > > Tha

Spark Implicit Functions

All, I just did a post on adding groupByKeyToList and groupByKeyUnique using implicit classes. I thought it might be useful to someone. http://codingjunkie.net/learning-scala-implicits-with-spark/ Thanks, Bill

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

JB, I am using spark-env.sh to define the master address instead of using spark-defaults.conf. I understand that that should work, and indeed it does, but only if SPARK_MASTER_IP is set to a DNS name and not an IP address. Perhaps I'm misunderstanding these configuration methods... Nick On Fr

Re: Insight into Spark Packages

[repost to mailing list] I don't know much about packages, but have you heard about the sbt-spark-package plugin? Looking at the code, specifically https://github.com/databricks/sbt-spark-package/blob/master/src/main/scala/sbtsparkpackage/SparkPackagePlugin.scala, might give you insight on the det

Re: Spark Event Listener

The reason for having two separate interfaces is developer API backwards-compatibility, as far as I know. SparkFirehoseListener came later. On Tue, Oct 13, 2015 at 4:36 PM, Jakob Odersky wrote: > the path of the source file defining the event API is > `core/src/main/scala/org/apache/spark/schedu

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

Ah, my bad, I missed it since the GitHub search results preview only showed the first hit from start-master.sh and n

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

It's used in scripts like sbin/start-master.sh On Fri, Oct 16, 2015 at 5:01 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I'd look into tracing a possible bug here, but I'm not sure where to look. > Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not show it > being us

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

if [ "$SPARK_MASTER_IP" = "" ]; then SPARK_MASTER_IP=`hostname` --ip $SPARK_MASTER_IP --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT \ "$sbin"/../tachyon/bin/tachyon bootstrap-conf $SPARK_MASTER_IP ./sbin/start-master.sh if [ "$SPARK_MASTER_IP" = "" ]; then SPARK_MASTER_IP

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

Hi Nick, there's the Spark master defined in conf/spark-defaults.conf and the -h option that you can provide to sbin/start-master.sh script. Did you try: sbin/start-master.sh -h xxx.xxx.xxx.xxx and then use the IP when you start the slaves: sbin/start-slave.sh spark://xxx.xxx.xxx.xxx.7077

Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

I'd look into tracing a possible bug here, but I'm not sure where to look. Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not show it being used in any place directly by Spark . Clearly, Spark is using this env

Re: Building Spark

Hi Annabel, with the user that you use to lunch mvn, do you have write permission in C:\Users\Annabel\git\spark\network\common folder ? Regards JB On 10/16/2015 05:19 PM, Annabel Melongo wrote: Can someone please provide insight why I get an access denied when I do the build according to the

Re: Building Spark

Can someone please provide insight why I get an access denied when I do the build according to the documentation? Ted said I have to provide the credentials but there's nothing mention about that in the build documentation. On Thursday, October 15, 2015 8:39 PM, Annabel Melongo wrote: