Re: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-03-04 Thread Nick Pentreath
ghran <ste...@hortonworks.com>; Shouheng Yi < > sho...@microsoft.com.invalid>; Apache Spark Dev <dev@spark.apache.org>; > Markus Weimer <mwei...@microsoft.com>; Rogan Carr <roc...@microsoft.com>; > Pei Jiang <pej...@microsoft.com>; Miruna Oprescu <mopr

RE: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-03-03 Thread Shouheng Yi
oc...@microsoft.com>>; Pei Jiang <pej...@microsoft.com<mailto:pej...@microsoft.com>>; Miruna Oprescu <mopre...@microsoft.com<mailto:mopre...@microsoft.com>> Subject: Re: [Spark Namespace]: Expanding Spark ML under Different Namespace? Regarding logging, Graphframes makes a sim

RE: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-03-03 Thread Shouheng Yi
t;; Markus Weimer <mwei...@microsoft.com>; Rogan Carr <roc...@microsoft.com>; Pei Jiang <pej...@microsoft.com>; Miruna Oprescu <mopre...@microsoft.com> Subject: Re: [Spark Namespace]: Expanding Spark ML under Different Namespace? Regarding logging, Graphframes make

Re: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-02-24 Thread Tim Hunter
Regarding logging, Graphframes makes a simple wrapper this way: https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/ graphframes/Logging.scala Regarding the UDTs, they have been hidden to be reworked for Datasets, the reasons being detailed here [1]. Can you describe your

Re: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-02-23 Thread Joseph Bradley
+1 for Nick's comment about discussing APIs which need to be made public in https://issues.apache.org/jira/browse/SPARK-19498 ! On Thu, Feb 23, 2017 at 2:36 AM, Steve Loughran wrote: > > On 22 Feb 2017, at 20:51, Shouheng Yi > wrote: > > Hi

Re: [Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-02-23 Thread Steve Loughran
On 22 Feb 2017, at 20:51, Shouheng Yi > wrote: Hi Spark developers, Currently my team at Microsoft is extending Spark’s machine learning functionalities to include new learners and transformers. We would like users to use

[Spark Namespace]: Expanding Spark ML under Different Namespace?

2017-02-22 Thread Shouheng Yi
Hi Spark developers, Currently my team at Microsoft is extending Spark's machine learning functionalities to include new learners and transformers. We would like users to use these within spark pipelines so that they can mix and match with existing Spark learners/transformers, and overall have