Re: Publishing of the Spectral LDA model on Spark Packages

This is very cool ! Thanks a lot for making this more accessible ! Best, -- FG On Wed, Dec 7, 2016 at 11:46 PM Jencir Lee <jenc...@uci.edu> wrote: > Hello, > > We just published the Spectral LDA model on Spark Packages. It’s an > alternative approach to the LDA modelli

Re: Spark-packages

I think there is a bit more life in the connector side of things for spark-packages, but there seem to be some outstanding issues with Python support that are waiting on progress (see https://github.com/databricks/sbt-spark-package/issues/26 ). It's possible others are just distributing on maven

Spark-packages

What is the state of the spark-packages project(s) ? When running a query for machine learning algorithms the results are not encouraging. https://spark-packages.org/?q=tags%3A%22Machine%20Learning%22 There are 62 packages. Only a few have actual releases - and even less with dates in the past

accessing spark packages through proxy

Dear Spark users, I am trying to use spark packages, however I get the ivy error listed below. I checked JIRA and stackoverflow and it might be a proxy error. However, neither of proposed solutions did not work for me. Could you suggest how to solve this issue? https://issues.apache.org/jira

Error publishing to spark-packages

Hi guys, I built a spark package but couldn't publish them with sbt-spark-package plugin. Any idea why these are failing? http://spark-packages.org/staging?id=1179 http://spark-packages.org/staging?id=1168 Repo: https://github.com/spotify/spark-bigquery Jars are published to Maven:

Re: Building Spark packages with SBTor Maven

Thanks the maven structure is identical to sbt. just sbt file I will have to replace with pom.xml I will use your pom.xml to start with it. Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Building Spark packages with SBTor Maven

Yes, sbt uses the same structure as maven for source files. > On Mar 15, 2016, at 1:53 PM, Mich Talebzadeh > wrote: > > Thanks the maven structure is identical to sbt. just sbt file I will have to > replace with pom.xml > > I will use your pom.xml to start with it.

Re: Building Spark packages with SBTor Maven

You can build using maven from the command line as well. This layout should give you an idea and here are some resources - http://www.scala-lang.org/old/node/345 project/ pom.xml - Defines the project src/ main/ java/ - Contains

Re: Building Spark packages with SBTor Maven

sounds like the layout is basically the same as sbt layout, the sbt file is replaced by pom.xml? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Building Spark packages with SBTor Maven

Thanks again Is there anyway one can set this one up without eclipse much like what I did with sbt? I need to know the directory structure foe MVN project. Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Building Spark packages with SBTor Maven

Do you have the Eclipse Maven plugin setup? http://www.eclipse.org/m2e/ Once you have it setup, File -> New -> Other -> MavenProject -> Next / Finish. You’ll see a default POM.xml which you can modify / replace. Here is some documentation that should help:

Re: Building Spark packages with SBTor Maven

Great Chandeep. I also have Eclipse Scala IDE below scala IDE build of Eclipse SDK Build id: 4.3.0-vfinal-2015-12-01T15:55:22Z-Typesafe I am no expert on Eclipse so if I create project called ImportCSV where do I need to put the pom file or how do I reference it please. My Eclipse runs on a

Re: Building Spark packages with SBTor Maven

Btw, just to add to the confusion ;) I use Maven as well since I moved from Java to Scala but everyone I talk to has been recommending SBT for Scala. I use the Eclipse Scala IDE to build. http://scala-ide.org/ Here is my sample PoM. You can add dependancies based on

Re: Building Spark packages with SBTor Maven

Ok. Sounds like opinion is divided :) I will try to build a scala app with Maven. When I build with SBT I follow this directory structure High level directory the package name like ImportCSV under ImportCSV I have a directory src and the sbt file ImportCSV.sbt in directory src I have main

Re: Building Spark packages with SBTor Maven

FWIW, I strongly prefer Maven over SBT even for Scala projects. The Spark build of reference is Maven. On Tue, Mar 15, 2016 at 10:45 AM, Chandeep Singh wrote: > For Scala, SBT is recommended. > > On Mar 15, 2016, at 10:42 AM, Mich Talebzadeh >

Re: Building Spark packages with SBTor Maven

Puled this from stack overflow: We're using Maven to build Scala projects at work because it integrates well with our CI server. We could just run a shell script to kick off a build, of course, but we've got a bunch of other information coming out of Maven that we want to go into CI. That's

Re: Building Spark packages with SBTor Maven

There're build jobs for both on Jenkins: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/ https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/ You can choose either one. I use mvn. On Tue, Mar 15, 2016 at 3:42 AM, Mich Talebzadeh

Re: Building Spark packages with SBTor Maven

For Scala, SBT is recommended. > On Mar 15, 2016, at 10:42 AM, Mich Talebzadeh > wrote: > > Hi, > > I build my Spark/Scala packages using SBT that works fine. I have created > generic shell scripts to build and submit it. > > Yesterday I noticed that some use

Building Spark packages with SBTor Maven

Hi, I build my Spark/Scala packages using SBT that works fine. I have created generic shell scripts to build and submit it. Yesterday I noticed that some use Maven and Pom for this purpose. Which approach is recommended? Thanks, Dr Mich Talebzadeh LinkedIn *

Re: Using SPARK packages in Spark Cluster

Monday, February 15, 2016 at 10:03 To: Jorge Machado <jom...@me.com> Cc: Spark Group <user@spark.apache.org> Subject: Re: Using SPARK packages in Spark Cluster Hi Jorge/ All, Please please please go through this link http://spark.apache.org/docs/latest/spark-standalone.html. The

Re: Using SPARK packages in Spark Cluster

Hi Jorge/ All, Please please please go through this link http://spark.apache.org/docs/latest/spark-standalone.html. The link tells you how to start a SPARK cluster in local mode. If you have not started or worked in SPARK cluster in

Re: Using SPARK packages in Spark Cluster

Hi Gourav, I did not unterstand your problem… the - - packages command should not make any difference if you are running standalone or in YARN for example. Give us an example what packages are you trying to load, and what error are you getting… If you want to use the libraries in

Re: Using SPARK packages in Spark Cluster

Hi, I am grateful for everyone's response, but sadly no one here actually has read the question before responding. Has anyone yet tried starting a SPARK cluster as mentioned in the link in my email? :) Regards, Gourav On Mon, Feb 15, 2016 at 11:16 AM, Jorge Machado wrote: >

Re: Using SPARK packages in Spark Cluster

$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.3.0 It will download everything for you and register into your JVM. If you want to use it in your Prod just package it with maven. > On 15/02/2016, at 12:14, Gourav Sengupta wrote: > > Hi, >

Re: Using SPARK packages in Spark Cluster

Hi, How to we include the following package: https://github.com/databricks/spark-csv while starting a SPARK standalone cluster as mentioned here: http://spark.apache.org/docs/latest/spark-standalone.html Thanks and Regards, Gourav Sengupta On Mon, Feb 15, 2016 at 10:32 AM, Ramanathan R

Re: Using SPARK packages in Spark Cluster

Hi Gourav, If your question is how to distribute python package dependencies across the Spark cluster programmatically? ...here is an example - $ export PYTHONPATH='path/to/thrift.zip:path/to/happybase.zip:path/to/your/py/application' And in code:

Re: Using SPARK packages in Spark Cluster

Hi, So far no one is able to get my question at all. I know what it takes to load packages via SPARK shell or SPARK submit. How do I load packages when starting a SPARK cluster, as mentioned here http://spark.apache.org/docs/latest/spark-standalone.html ? Regards, Gourav Sengupta On Mon,

Re: Using SPARK packages in Spark Cluster

Hi, I was interested in knowing how to load the packages into SPARK cluster started locally. Can someone pass me on the links to set the conf file so that the packages can be loaded? Regards, Gourav On Fri, Feb 12, 2016 at 6:52 PM, Burak Yavuz wrote: > Hello Gourav, > > The

Using SPARK packages in Spark Cluster

Hi, I am creating sparkcontext in a SPARK standalone cluster as mentioned here: http://spark.apache.org/docs/latest/spark-standalone.html using the following code: -- sc.stop()

Re: Using SPARK packages in Spark Cluster

Hello Gourav, The packages need to be loaded BEFORE you start the JVM, therefore you won't be able to add packages dynamically in code. You should use the --packages with pyspark before you start your application. One option is to add a `conf` that will load some packages if you are constantly

Re: Guidelines for writing SPARK packages

park.apache.org> > Date: 01/02/2016 07:03 pm > Subject:Re: Guidelines for writing SPARK packages > Sent by:marchoffo...@gmail.com > -- > > > > Hi Praveen, > > The basic requirements for releasing a Spark package on > s

Re: Guidelines for writing SPARK packages

share with me...else debugging through would be the way :-) Thanking You Praveen Devarao From: David Russell <themarchoffo...@gmail.com> To: Praveen Devarao/India/IBM@IBMIN Cc: user <user@spark.apache.org> Date: 01/02/2016 07:03 pm Subject:Re: Guidelines for w

Re: Guidelines for writing SPARK packages

//github.com/databricks/sbt-spark-package) to help with building/publishing if you plan to use Scala in your package. You could of course use Maven as well, but we don't have a maven plugin for Spark Packages. Best, Burak

Re: Guidelines for writing SPARK packages

Hi Praveen, The basic requirements for releasing a Spark package on spark-packages.org are as follows: 1. The package content must be hosted by GitHub in a public repo under the owner's account. 2. The repo name must match the package name. 3. The master branch of the repo must contain

Guidelines for writing SPARK packages

Hi, Is there any guidelines or specs to write a Spark package? I would like to implement a spark package and would like to know the way it needs to be structured (implement some interfaces etc) so that it can plug into Spark for extended functionality. Could any one help me

Re: Spark Packages Configuration Not Found

As another, general question, are spark packages the go-to way of extending spark functionality? In my specific use-case I would like to start spark (be it spark-shell or other) and hook into the listener API. Since I wasn't able to find much documentation about spark packages, I was wondering

Re: Spark Packages Configuration Not Found

Hi Jakob, > As another, general question, are spark packages the go-to way of extending spark functionality? Definitely. There are ~150 Spark Packages out there in spark-packages.org. I use a lot of them in every day Spark work. The number of released packages have steadily increased rate o

Re: Spark Packages Configuration Not Found

or trying to run spark-shell with an external package > that I built and published locally > using the spark-package sbt plugin ( > https://github.com/databricks/sbt-spark-package). > > To my understanding, spark packages can be published simply as maven > artifacts, yet after runn

Spark Packages Configuration Not Found

Hi, I ran into in error trying to run spark-shell with an external package that I built and published locally using the spark-package sbt plugin ( https://github.com/databricks/sbt-spark-package). To my understanding, spark packages can be published simply as maven artifacts, yet after running

Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

Hi, I have raised a JIRA ( https://issues.apache.org/jira/browse/SPARK-11045) to track the discussion but also mailing user group . This Kafka consumer is around for a while in spark-packages ( http://spark-packages.org/package/dibbhatt/kafka-spark-consumer ) and I see many started using it , I

Just Released V1.0.4 Low Level Receiver Based Kafka-Spark-Consumer in Spark Packages having built-in Back Pressure Controller

Dear All, Just now released the 1.0.4 version of Low Level Receiver based Kafka-Spark-Consumer in spark-packages.org . You can find the latest release here : http://spark-packages.org/package/dibbhatt/kafka-spark-consumer Here is github location :

Announcing Spark Packages

Dear Spark users and developers, I’m happy to announce Spark Packages (http://spark-packages.org), a community package index to track the growing number of open source packages and libraries that work with Apache Spark. Spark Packages makes it easy for users to find, discuss, rate, and install

Re: Announcing Spark Packages

...@gmail.com wrote: Dear Spark users and developers, I’m happy to announce Spark Packages (http://spark-packages.org), a community package index to track the growing number of open source packages and libraries that work with Apache Spark. Spark Packages makes it easy for users to find

Re: Announcing Spark Packages

and developers, I’m happy to announce Spark Packages (http://spark-packages.org), a community package index to track the growing number of open source packages and libraries that work with Apache Spark. Spark Packages makes it easy for users to find, discuss, rate, and install packages for any

Re: Announcing Spark Packages

website, without written approval of the VP, Apache Brand Management or designee. The title on the packages website is “A community index of packages for Apache Spark.” Furthermore, the footnote of the website reads “Spark Packages is a community site hosting modules that are not part of Apache Spark

Re: Announcing Spark Packages

for Apache Spark. Furthermore, the footnote of the website reads Spark Packages is a community site hosting modules that are not part of Apache Spark. I think there's nothing on there that would confuse a relevant consumer about the source of software. It's pretty clear that the Spark Packages