Re: [Spark ML] Positive-Only Training Classification in Scala

[hosur narahari]

You can make use of probability vector from spark classification.
When you run spark classification model for prediction, along with
classifying into its class spark also gives probability vector(what's the
probability that this could belong to each individual class) . So just take
the probability corresponding to the donor class. And it'll be same as
what's the probability the a person will become donor.

Best Regards,
Hari

On 15 Jan 2018 11:51 p.m., "Matt Hicks"  wrote:

> I'm attempting to create a training classification, but only have positive
> information.  Specifically in this case it is a donor list of users, but I
> want to use it as training in order to determine classification for new
> contacts to give probabilities that they will donate.
>
> Any insights or links are appreciated. I've gone through the documentation
> but have been unable to find any references to how I might do this.
>
> Thanks
>
> ---*Matt Hicks*
>
> *Chief Technology Officer*
>
> 405.283.6887 | http://outr.com
>
> [image: logo 2 small.png]
>
>

RE: Do we anything for Deep Learning in Spark?

[hosur narahari]

Thank you.

Best Regards,
Hari

On 7 Jul 2017 3:59 a.m., "Roope Astala"  wrote:

> You can use an attached GPU VM for DNN training, and do other processing
> on regular CPU nodes. You can even deallocate the GPU VM to save costs when
> not using it. The GPU branch has instructions how to set up such compute
> environment:
>
>
>
> https://github.com/Azure/mmlspark/tree/gpu#gpu-vm-setup
>
>
>
> Cheers,
>
> Roope – Microsoft Cloud AI Team
>
>
>
> *From:* hosur narahari [mailto:hnr1...@gmail.com]
> *Sent:* Thursday, July 6, 2017 1:54 AM
> *To:* Gaurav1809 
> *Cc:* user 
> *Subject:* Re: Do we anything for Deep Learning in Spark?
>
>
>
> Hi Roope,
>
>
>
> Does this mmlspark project uses GPGPU for processing and just CPU cores
> since DL models are computationally very intensive.
>
>
>
> Best Regards,
>
> Hari
>
>
>
> On 6 Jul 2017 9:33 a.m., "Gaurav1809"  wrote:
>
> Thanks Roope for the inputs.
>
>
>
> On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=28826&i=0>> wrote:
>
> Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep
> learning models on Spark.
>
> https://github.com/Azure/mmlspark
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fmmlspark&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=1FqnrRsLY6JrPb%2BdkG4AlLBN%2F8f2O6w3Ehi9DYvu6lc%3D&reserved=0>
>
> The library APIs are focused on image processing scenarios, and are
> compatible with SparkML Pipelines.
>
> Cheers,
> Roope - Microsoft Cloud AI Team
> --
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-
> anything-for-Deep-Learning-in-Spark-tp28772p28824.html
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=zFajYYH16xwzDRwgP1se9m%2BAA5fIufUS7FFxHjzTOl0%3D&reserved=0>
>
> To start a new topic under Apache Spark User List, email [hidden email]
> <http://user/SendEmail.jtp?type=node&node=28826&i=1>
> To unsubscribe from Do we anything for Deep Learning in Spark?, click here.
> NAML
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=wzDT2AoJRfwqxlumkdoRlGq6b0PaeoypZtvvLgFlR1M%3D&reserved=0>
>
>
>
>
> --
>
> View this message in context: Re: Do we anything for Deep Learning in
> Spark?
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=dDprNB1TSm9WkAY1iuN1NjkG7vzx6JaiXB42DcXqUt4%3D&reserved=0>
> Sent from the Apache Spark User List mailing list archive
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2F&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=bI5METH7hpg6xCjUQF0%2BiIKY7qFy709YGC4g0n9MjW0%3D&reserved=0>
> at Nabble.com.
>
>

Re: Do we anything for Deep Learning in Spark?

[hosur narahari]

Hi Roope,

Does this mmlspark project uses GPGPU for processing and just CPU cores
since DL models are computationally very intensive.

Best Regards,
Hari

On 6 Jul 2017 9:33 a.m., "Gaurav1809"  wrote:

> Thanks Roope for the inputs.
>
> On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden
> email] > wrote:
>
>> Microsoft Machine Learning Library for Apache Spark lets you run CNTK
>> deep learning models on Spark.
>>
>> https://github.com/Azure/mmlspark
>>
>> The library APIs are focused on image processing scenarios, and are
>> compatible with SparkML Pipelines.
>>
>> Cheers,
>> Roope - Microsoft Cloud AI Team
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-an
>> ything-for-Deep-Learning-in-Spark-tp28772p28824.html
>> To start a new topic under Apache Spark User List, email [hidden email]
>> 
>> To unsubscribe from Do we anything for Deep Learning in Spark?, click
>> here.
>> NAML
>> 
>>
>
>
> --
> View this message in context: Re: Do we anything for Deep Learning in
> Spark?
> 
> Sent from the Apache Spark User List mailing list archive
>  at Nabble.com.
>

RE: Any NLP library for sentiment analysis in Spark?

[hosur narahari]

Tensorflow provides NLP implementation which uses deep learning technology.
But it's not distributed. So you can try to integrate spark with
Tensorflow.

Best Regards,
Hari

On 11 Apr 2017 11:44 p.m., "Gabriel James" 
wrote:

> Me too. Experiences and recommendations please.
>
>
>
> Gabriel
>
>
>
> *From:* Kevin Wang [mailto:buz...@gmail.com]
> *Sent:* Wednesday, April 12, 2017 6:11 AM
> *To:* Alonso Isidoro Roman 
> *Cc:* Gaurav1809 ; user@spark.apache.org
> *Subject:* Re: Any NLP library for sentiment analysis in Spark?
>
>
>
> I am also interested in this topic.  Anything else anyone can recommend?
> Thanks.
>
>
>
> Best,
>
>
>
> Kevin
>
>
>
> On Tue, Apr 11, 2017 at 5:00 AM, Alonso Isidoro Roman 
> wrote:
>
> i did not use it yet, but this library looks promising:
>
>
>
> https://github.com/databricks/spark-corenlp
>
>
>
>
> *Alonso Isidoro Roman*
>
> about.me/alonso.isidoro.roman
>
>
>
> 2017-04-11 11:02 GMT+02:00 Gaurav1809 :
>
> Hi All,
>
> I need to determine sentiment for given document (statement, paragraph
> etc.)
> Is there any NLP library available with Apache Spark that I can use here?
>
> Any other pointers towards this would be highly appreciated.
>
> Thanks in advance.
> Gaurav Pandya
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Any-NLP-library-for-sentiment-
> analysis-in-Spark-tp28586.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
>
>
>

Re: convert JavaRDD> to JavaRDD

[hosur narahari]

Use flatmap function on JavaRDD

On 5 Apr 2017 3:13 p.m., "Hamza HACHANI"  wrote:

> I want to convert a JavaRDD> to JavaRDD. For example
> if there is 3 elment in List 3 Object would be created in my new
> JavaRDD.
>
> Does any one have an idea ?
>

Re: dataframe filter, unable to bind variable

[hosur narahari]

Try lit(fromDate) and lit(toDate). You've to import
org.apache.spark.sql.functions.lit to use it

On 31 Mar 2017 7:45 a.m., "shyla deshpande" 
wrote:

The following works

df.filter($"createdate".between("2017-03-20", "2017-03-22"))


I would like to pass variables fromdate and todate to the filter

 instead of constants. Unable to get the syntax right. Please help.


Thanks