[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/19527
  
Thank you for all the work in this PR!  Here's the follow-up: 
https://github.com/apache/spark/pull/20132


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
@jkbradley Thanks for reviewing and merging this. Thanks for all helping 
this too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/19527
  
Thanks for the updates!  I still think there's some confusion, but since I 
think this code is correct & it doesn't affect APIs, I'll go ahead and merge 
this.  I'll ping you on a follow-up PR to show what I had in mind.

LGTM
Merging with master


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85551/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85551 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85551/testReport)**
 for PR 19527 at commit 
[`e94496a`](https://github.com/apache/spark/commit/e94496a5c8b08fdc437e9623dfba2b0d80998263).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85551 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85551/testReport)**
 for PR 19527 at commit 
[`e94496a`](https://github.com/apache/spark/commit/e94496a5c8b08fdc437e9623dfba2b0d80998263).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
@jkbradley Ok. I will update this today.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/19527
  
@viirya Will you have time to update this soon?  I'd like to get it in 2.3, 
which will mean merging it by Jan. 1.  If you're busy, I can merge it as is and 
do a follow-up myself too.  Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85391/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85391 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85391/testReport)**
 for PR 19527 at commit 
[`587ad42`](https://github.com/apache/spark/commit/587ad427a6682e98e1fefe592ecf278c674767f3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
Unit tests are reformatted too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85389/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85389 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85389/testReport)**
 for PR 19527 at commit 
[`144f07d`](https://github.com/apache/spark/commit/144f07d5e92bf5cbc10cb2dc990fc32f15405977).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85391 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85391/testReport)**
 for PR 19527 at commit 
[`587ad42`](https://github.com/apache/spark/commit/587ad427a6682e98e1fefe592ecf278c674767f3).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #85389 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85389/testReport)**
 for PR 19527 at commit 
[`144f07d`](https://github.com/apache/spark/commit/144f07d5e92bf5cbc10cb2dc990fc32f15405977).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[MLnick]

Github user MLnick commented on the issue:

https://github.com/apache/spark/pull/19527
  
Agree on keeping the new OneHotEncoderEstimator as an alias for 3.0

On Fri, 1 Dec 2017 at 23:29, jkbradley  wrote:

> *@jkbradley* commented on this pull request.
> --
>
> In mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala
> :
>
> > @@ -41,8 +41,12 @@ import org.apache.spark.sql.types.{DoubleType, 
NumericType, StructType}
>   * The output vectors are sparse.
>   *
>   * @see `StringIndexer` for converting categorical values into category 
indices
> + * @deprecated `OneHotEncoderEstimator` will be renamed `OneHotEncoder` 
and this `OneHotEncoder`
>
> Note for the future: For 3.0, it'd be nice to do what you're describing
> here but also leave OneHotEncoderEstimator as a deprecated alias. That 
way,
> user code won't break but will have deprecation warnings when upgrading to
> 3.0.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
@jkbradley Can you review this again? Thanks.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84690/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #84690 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84690/testReport)**
 for PR 19527 at commit 
[`32318fa`](https://github.com/apache/spark/commit/32318faebd118509bdd0c0100e84c4755182ea27).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #84690 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84690/testReport)**
 for PR 19527 at commit 
[`32318fa`](https://github.com/apache/spark/commit/32318faebd118509bdd0c0100e84c4755182ea27).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
Ok, I understood. In other words, the extra category is added as the last 
category and `dropLast` option works as before. It makes sense to me.






---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/19527
  
> For example with 5 categories, we don't know [0.0, 0.0, 0.0, 0.0, 0.0] 
means last category or invalid value.

For the semantics I described ("OPTION 1"), I think it is clear:
* Last category would lead to [0.0, 0.0, 0.0, 0.0, 1.0]
* Invalid value would lead to [0.0, 0.0, 0.0, 0.0, 0.0]

For OPTION 1, I figured:
* ```keep=true``` adds 1 extra "category" indicating an invalid value
* ```dropLast=true``` removes the last category
* Invalid values ("keep") are handled before removing the last category 
("dropLast").

I realize now that it's the ordering of operations which is unclear here.  
If we handled "dropLast" before "keep" then we would have OPTION 2:
* ```keep=true, dropLast=true``` ==> vector size n (all-0 for the last 
category; _has a 1 at the end for invalid values_)
  * These semantics seem weird to me, so I'd prefer we handle "keep" before 
"dropLast".
* ```keep=true, dropLast=false``` ==> (same regardless of the order of 
operations)
* ```keep=false, dropLast=true``` ==> (same regardless of the order of 
operations)
* ```keep=false, dropLast=false``` ==> (same regardless of the order of 
operations)

OPTION 1 is more flexible than disallowing ```keep=true, dropLast=true```.  
With OPTION 1, we can match sklearn's behavior with ```keep=true, 
dropLast=true```.

What do you think?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
The behavior I thought is:

* keep=true, dropLast=true ==> error option
* keep=true, dropLast=false ==> vector size n (all-0 only for invalid value)

For the cases of `dropLast = false`, it behaves similarly as 
`sklearn.preprocessing.OneHotEncoder`.

If we make it behave as:

* keep=true, dropLast=true ==> vector size n (all-0 only if there was an 
invalid value)

For example with 5 categories, we don't know `[0.0, 0.0, 0.0, 0.0, 0.0]` 
means last category or invalid value.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/19527
  
Question about this PR description comment:
> Note that keep can't be used at the same time with dropLast as true. 
Because they will conflict in encoded vector by producing a vector of zeros.

Why is this necessary?  With ```n``` categories found in fitting, shouldn't 
the behavior be the following?
* ```keep=true, dropLast=true``` ==> vector size n
* ```keep=true, dropLast=false``` ==> vector size n+1
* ```keep=false, dropLast=true``` ==> vector size n-1
* ```keep=false, dropLast=false``` ==> vector size n



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83738/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83738 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83738/testReport)**
 for PR 19527 at commit 
[`3b339b7`](https://github.com/apache/spark/commit/3b339b74c8faefc6a79f503125d9ef4880ec60df).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83738 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83738/testReport)**
 for PR 19527 at commit 
[`3b339b7`](https://github.com/apache/spark/commit/3b339b74c8faefc6a79f503125d9ef4880ec60df).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83258/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83258 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83258/testReport)**
 for PR 19527 at commit 
[`4c6cc57`](https://github.com/apache/spark/commit/4c6cc57136a60c577dd0ba2ae90521f7c18ae5d1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83258 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83258/testReport)**
 for PR 19527 at commit 
[`4c6cc57`](https://github.com/apache/spark/commit/4c6cc57136a60c577dd0ba2ae90521f7c18ae5d1).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
Thanks for reviewing @zhengruifeng. I will update it later.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83070/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83070 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83070/testReport)**
 for PR 19527 at commit 
[`ae2ac82`](https://github.com/apache/spark/commit/ae2ac82b10e457b8beede9dc4a33ce0a578f007d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83070 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83070/testReport)**
 for PR 19527 at commit 
[`ae2ac82`](https://github.com/apache/spark/commit/ae2ac82b10e457b8beede9dc4a33ce0a578f007d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
@huaxingao Good catch! Thanks.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83000/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83000 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83000/testReport)**
 for PR 19527 at commit 
[`adc4107`](https://github.com/apache/spark/commit/adc410770528c6c95a3c35de64548362c1b46643).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #83000 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83000/testReport)**
 for PR 19527 at commit 
[`adc4107`](https://github.com/apache/spark/commit/adc410770528c6c95a3c35de64548362c1b46643).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
@BryanCutler @MLnick @WeichenXu123 Thanks for reviewing. Your comments 
should be all addressed now. Please take a look again when you have more time.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82934/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82934 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82934/testReport)**
 for PR 19527 at commit 
[`e024120`](https://github.com/apache/spark/commit/e0241200c58a5ec201a0f1abdebc1660878ed49f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82933/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82933 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82933/testReport)**
 for PR 19527 at commit 
[`fe80e98`](https://github.com/apache/spark/commit/fe80e98712f52a4b5795c96a20e8f92e65849cb4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82932/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82932 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82932/testReport)**
 for PR 19527 at commit 
[`a9e9262`](https://github.com/apache/spark/commit/a9e9262c2a05174f019cddb8a1ae14d48a92ffba).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82934 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82934/testReport)**
 for PR 19527 at commit 
[`e024120`](https://github.com/apache/spark/commit/e0241200c58a5ec201a0f1abdebc1660878ed49f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82933 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82933/testReport)**
 for PR 19527 at commit 
[`fe80e98`](https://github.com/apache/spark/commit/fe80e98712f52a4b5795c96a20e8f92e65849cb4).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82932 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82932/testReport)**
 for PR 19527 at commit 
[`a9e9262`](https://github.com/apache/spark/commit/a9e9262c2a05174f019cddb8a1ae14d48a92ffba).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82930/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82930 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82930/testReport)**
 for PR 19527 at commit 
[`66d46ac`](https://github.com/apache/spark/commit/66d46acaa58ca0e6304878504e94117bbee59d24).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82930 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82930/testReport)**
 for PR 19527 at commit 
[`66d46ac`](https://github.com/apache/spark/commit/66d46acaa58ca0e6304878504e94117bbee59d24).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82917/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82917 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82917/testReport)**
 for PR 19527 at commit 
[`b42d175`](https://github.com/apache/spark/commit/b42d175ddc4928ec36718177702059ccf0bfbfea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82917 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82917/testReport)**
 for PR 19527 at commit 
[`b42d175`](https://github.com/apache/spark/commit/b42d175ddc4928ec36718177702059ccf0bfbfea).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
Benchmark against existing one hot encoder.

Because existing encoder only needs to run `transform`, there is no fitting 
time.


Transforming:

numColums | Existing one hot encoder
-- | -- 
1 | 0.2516055188
100 | 20.29175892115
1000 | 26242.039411932*

* Because ten iterations take too long to finish, I just ran one iteration 
for 1000 columns. But it shows the scale already.

Benchmark codes:

```scala
import org.apache.spark.ml.feature._
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
import spark.implicits._
import scala.util.Random

val seed = 123l
val random = new Random(seed)
val n = 1
val m = 1000
val rows = sc.parallelize(1 to n).map(i=> 
Row(Array.fill(m)(random.nextInt(1000)): _*))
val struct = new StructType(Array.range(0,m,1).map(i => 
StructField(s"c$i",IntegerType,true)))
val df = spark.createDataFrame(rows, struct)
df.persist()
df.count()

val inputCols = Array.range(0,m,1).map(i => s"c$i")
val outputCols = Array.range(0,m,1).map(i => s"c${i}_encoded")

val encoders = Array.range(0,m,1).map(i => new 
OneHotEncoder().setInputCol(s"c$i").setOutputCol(s"c${i}_encoded"))
var duration = 0.0
for (i <- 0 until 10) {
  var encoded = df
  val start = System.nanoTime()
  encoders.foreach { encoder =>
encoded = encoder.transform(encoded)
  }
  encoded.count
  val end = System.nanoTime()
  duration += (end - start) / 1e9
}
println(s"duration: ${duration / 10}")
```



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  

Benchmark against multi-column one hot encoder.

Multi-Col, Multiple run: The first commit. Run multiple `treeAggregate` on 
columns.
Multi-Col, Single Run: Run one `treeAggregate` on all columns, see 
suggestion at https://github.com/apache/spark/pull/19527#discussion_r145457081.

Fitting:

numColums | Multi-Col, Multiple run | Multi-Col, Single Run
-- | -- | --
1 | 0.1100363843003 | 0.1296882409998
100 | 3.687933463507 | 0.3643889783995
1000 | 90.3695017947 | 2.4687475008

Transforming:

numColums | Multi-Col, Multiple run | Multi-Col, Single Run
-- | -- | --
1 | 0.1408046101999 | 0.1434849307
100 | 0.3636357813 | 0.4145960696996
1000 | 3.1933874685 | 2.8026313985


Benchmark codes:
```scala
import org.apache.spark.ml.feature._
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
import spark.implicits._
import scala.util.Random

val seed = 123l
val random = new Random(seed)
val n = 1
val m = 1000
val rows = sc.parallelize(1 to n).map(i=> 
Row(Array.fill(m)(random.nextInt(1000)): _*))
val struct = new StructType(Array.range(0,m,1).map(i => 
StructField(s"c$i",IntegerType,true)))
val df = spark.createDataFrame(rows, struct)
df.persist()
df.count()

val inputCols = Array.range(0,m,1).map(i => s"c$i")
val outputCols = Array.range(0,m,1).map(i => s"c${i}_encoded")

val encoder = new 
OneHotEncoderEstimator().setInputCols(inputCols).setOutputCols(outputCols)
var durationFitting = 0.0
var durationTransforming = 0.0
for (i <- 0 until 10) {
  val startFitting = System.nanoTime()
  val model = encoder.fit(df)
  val endFitting = System.nanoTime()
  durationFitting += (endFitting - startFitting) / 1e9

  val startTransforming = System.nanoTime()
  model.transform(df).count
  val endTransforming = System.nanoTime()
  durationTransforming += (endTransforming - startTransforming) / 1e9
}
println(s"fitting: ${durationFitting / 10}")
println(s"transforming: ${durationTransforming / 10}")





---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
cc @MLnick @WeichenXu123 @jkbradley This adds a new class 
`OneHotEncoderEstimator` which extends `Estimator`. Please review this when you 
can. Thanks.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82879/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82879 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82879/testReport)**
 for PR 19527 at commit 
[`8fd4677`](https://github.com/apache/spark/commit/8fd4677fd0e729d99d8777010e78bb5cfea3cf86).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class OneHotEncoderEstimator @Since(\"2.3.0\") (@Since(\"2.3.0\") 
override val uid: String)`
  * `  class OneHotEncoderModelWriter(instance: OneHotEncoderModel) extends 
MLWriter `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #82879 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82879/testReport)**
 for PR 19527 at commit 
[`8fd4677`](https://github.com/apache/spark/commit/8fd4677fd0e729d99d8777010e78bb5cfea3cf86).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org