[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133075802 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133079087 --- Diff: mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala --- @@ -374,6 +380,22 @@ private[ann] trait TopologyModel extends Serializable

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133079226 --- Diff: mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala --- @@ -374,6 +380,22 @@ private[ann] trait TopologyModel extends Serializable

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133076755 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133075752 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133080046 --- Diff: mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala --- @@ -361,9 +361,15 @@ private[ann] trait TopologyModel extends Serializable

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133076328 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133082415 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133080187 --- Diff: mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala --- @@ -527,9 +550,21 @@ private[ml] class FeedForwardModel private

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133082607 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala --- @@ -82,6 +83,49 @@ class

[GitHub] spark pull request #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17373#discussion_r133079322 --- Diff: mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala --- @@ -374,6 +380,22 @@ private[ann] trait TopologyModel extends Serializable

[GitHub] spark issue #17373: [SPARK-12664] Expose probability in mlp model

2017-08-14 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/17373 @WeichenXu123 Can you please add "[ML]" to the PR title? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

spark git commit: [SPARK-17025][ML][PYTHON] Persistence for Pipelines with Python-only Stages

2017-08-12 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master b0bdfce9c -> 35db3b9fe [SPARK-17025][ML][PYTHON] Persistence for Pipelines with Python-only Stages ## What changes were proposed in this pull request? Implemented a Python-only persistence framework for pipelines containing stages that

[GitHub] spark issue #18888: [Spark-17025][ML][Python] Persistence for Pipelines with...

2017-08-11 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/1 LGTM pending tests! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132744793 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132744462 --- Diff: python/pyspark/ml/pipeline.py --- @@ -16,14 +16,15 @@ # import sys +import os if sys.version >

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132599249 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132742064 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132744343 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132743731 --- Diff: python/pyspark/ml/tests.py --- @@ -1142,6 +1142,35 @@ def test_nested_pipeline_persistence(self): except OSError

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132599646 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132744158 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +327,65 @@ def _to_java(self): JavaParams._new_java_obj

[GitHub] spark pull request #18888: [Spark-17025][ML][Python] Persistence for Pipelin...

2017-08-11 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/1#discussion_r132599624 --- Diff: python/pyspark/ml/pipeline.py --- @@ -242,3 +319,74 @@ def _to_java(self): JavaParams._new_java_obj

spark git commit: [SPARK-21542][ML][PYTHON] Python persistence helper functions

2017-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master baf5cac0f -> fdcee028a [SPARK-21542][ML][PYTHON] Python persistence helper functions ## What changes were proposed in this pull request? Added DefaultParamsWriteable, DefaultParamsReadable, DefaultParamsWriter, and DefaultParamsReader to

[GitHub] spark issue #18742: [Spark-21542][ML][Python]Python persistence helper funct...

2017-08-07 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18742 Merging with master Thanks @ajaysaini725 and @WeichenXu123 ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #18742: [Spark-21542][ML][Python]Python persistence helper funct...

2017-08-07 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18742 LGTM pending tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-07 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131782033 --- Diff: python/pyspark/ml/util.py --- @@ -61,33 +66,86 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-07 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131780734 --- Diff: python/pyspark/ml/tests.py --- @@ -1158,6 +1165,33 @@ def test_decisiontree_regressor(self): except OSError: pass

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-07 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131749278 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +333,201 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-07 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131749037 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +333,201 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-05 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131516889 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +333,204 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-05 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131516809 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +333,204 @@ def numFeatures(self): Returns the number of features the model was trained

spark git commit: [SPARK-21633][ML][PYTHON] UnaryTransformer in Python

2017-08-04 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 25826c77d -> 1347b2a69 [SPARK-21633][ML][PYTHON] UnaryTransformer in Python ## What changes were proposed in this pull request? Implemented UnaryTransformer in Python. ## How was this patch tested? This patch was tested by creating a

[GitHub] spark issue #18746: [SPARK-21633][ML][Python] UnaryTransformer in Python

2017-08-04 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18746 LGTM Merging with master Thanks @ajaysaini725 ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-04 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131332345 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +341,198 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-04 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131331807 --- Diff: python/pyspark/ml/util.py --- @@ -156,28 +218,23 @@ def write(self): @inherit_doc -class MLReader(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-04 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131331793 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,89 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-04 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131332176 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +341,198 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131293810 --- Diff: python/pyspark/ml/tests.py --- @@ -1957,6 +1988,40 @@ def test_chisquaretest(self): self.assertTrue(all(field in fieldNames

[GitHub] spark issue #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18746 @ajaysaini725 Is there a JIRA for this PR? Please tag this PR in the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131284503 --- Diff: python/pyspark/ml/tests.py --- @@ -1957,6 +1964,46 @@ def test_chisquaretest(self): self.assertTrue(all(field in fieldNames

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131285744 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala --- @@ -471,3 +471,24 @@ private[ml] object MetaAlgorithmReadWrite { List

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131286314 --- Diff: python/pyspark/ml/util.py --- @@ -61,20 +66,74 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131287820 --- Diff: python/pyspark/ml/util.py --- @@ -237,6 +300,13 @@ def _load_java_obj(cls, clazz): java_obj = getattr(java_obj, name

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288629 --- Diff: python/pyspark/ml/util.py --- @@ -237,6 +300,13 @@ def _load_java_obj(cls, clazz): java_obj = getattr(java_obj, name

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288896 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288360 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131287028 --- Diff: python/pyspark/ml/util.py --- @@ -61,20 +66,74 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288786 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288351 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131290424 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131288910 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +353,143 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131287269 --- Diff: python/pyspark/ml/util.py --- @@ -86,7 +145,7 @@ def context(self, sqlContext): def session(self, sparkSession): --- End diff

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131286586 --- Diff: python/pyspark/ml/util.py --- @@ -61,20 +66,74 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131265094 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala --- @@ -471,3 +471,26 @@ private[ml] object MetaAlgorithmReadWrite { List

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131265462 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala --- @@ -471,3 +471,26 @@ private[ml] object MetaAlgorithmReadWrite { List

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131262008 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131262239 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131261798 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131260736 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131264975 --- Diff: python/pyspark/ml/util.py --- @@ -283,3 +289,124 @@ def numFeatures(self): Returns the number of features the model was trained

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131261087 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131223795 --- Diff: python/pyspark/ml/base.py --- @@ -116,3 +121,53 @@ class Model(Transformer): """ __metacl

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131222861 --- Diff: python/pyspark/ml/base.py --- @@ -116,3 +121,53 @@ class Model(Transformer): """ __metacl

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131258120 --- Diff: python/pyspark/ml/tests.py --- @@ -1957,6 +1987,24 @@ def test_chisquaretest(self): self.assertTrue(all(field in fieldNames

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r13190 --- Diff: python/pyspark/ml/base.py --- @@ -116,3 +121,53 @@ class Model(Transformer): """ __metacl

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r13122 --- Diff: python/pyspark/ml/base.py --- @@ -116,3 +121,53 @@ class Model(Transformer): """ __metacl

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131258476 --- Diff: python/pyspark/ml/tests.py --- @@ -1957,6 +1987,24 @@ def test_chisquaretest(self): self.assertTrue(all(field in fieldNames

[GitHub] spark pull request #18746: [ML][Python] UnaryTransformer in Python

2017-08-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18746#discussion_r131257864 --- Diff: python/pyspark/ml/tests.py --- @@ -1957,6 +1987,24 @@ def test_chisquaretest(self): self.assertTrue(all(field in fieldNames

[GitHub] spark issue #18746: [ML][Python] Implemented UnaryTransformer in Python

2017-08-02 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18746 Also, you can remove "implemented" from the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark issue #18746: [ML][Python] Implemented UnaryTransformer in Python

2017-08-02 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18746 @ajaysaini725 Is there a JIRA for this PR? Please tag this PR in the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131001072 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r130996252 --- Diff: python/pyspark/ml/param/__init__.py --- @@ -375,6 +375,18 @@ def copy(self, extra=None): that._defaultParamMap

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131001862 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131000706 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131001546 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131000859 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark pull request #18742: [Spark-21542][ML][Python]Python persistence helpe...

2017-08-02 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18742#discussion_r131000608 --- Diff: python/pyspark/ml/util.py --- @@ -61,32 +66,82 @@ def _randomUID(cls): @inherit_doc -class MLWriter(object): +class

[GitHub] spark issue #18313: [SPARK-21087] [ML] CrossValidator, TrainValidationSplit ...

2017-08-01 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18313 Oh, you're right; I overlooked that it only holds all of the models for a single split. In that case, I agree it could be problematic to keep all in memory by default. How does this sound

[GitHub] spark issue #18618: [SPARK-20090][PYTHON] Add StructType.fieldNames in PySpa...

2017-07-31 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18618 @holdenk Thanks for merging it! Just wondering: Why is the "pushed a commit" notification from hubot? Did you use the `dev/merge_spark_pr.py` script? --- If your project is set up f

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r129989469 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala --- @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r129989379 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -294,6 +296,18 @@ final class OneVsRest @Since("

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r129989332 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -294,6 +296,18 @@ final class OneVsRest @Since("

[GitHub] spark issue #18281: [SPARK-21027][ML][PYTHON] Added tunable parallelism to o...

2017-07-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18281 @holdenk Some of those improvements on handling parallelism sounds useful, but I'd prefer we merge this and then add more improvements. This PR should be a strict improvement there (moving from

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r129982672 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,45 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18313: [SPARK-21087] [ML] CrossValidator, TrainValidatio...

2017-07-24 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18313#discussion_r129178830 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala --- @@ -113,15 +122,28 @@ class CrossValidator @Since("1.2.0"

[GitHub] spark issue #13794: [SPARK-15574][ML][PySpark] Python meta-algorithms in Sca...

2017-07-17 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/13794 @yinxusen Thanks for this PR! I still think this seems like a very cool feature, but I've become less convinced that it's worth the engineering and maintenance effort. The alternative

spark git commit: [SPARK-21221][ML] CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest

2017-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 4ce735eed -> 7047f49f4 [SPARK-21221][ML] CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest ## What changes were proposed in this pull request? Added functionality for CrossValidator and

[GitHub] spark issue #18428: [Spark-21221][ML] CrossValidator and TrainValidationSpli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18428 LGTM Merging with master Thanks @ajaysaini725 for the patch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127753960 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,50 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127753831 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,50 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127752302 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127752074 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-17 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127753088 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,50 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127552088 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,50 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127552778 --- Diff: python/pyspark/ml/classification.py --- @@ -1511,27 +1512,47 @@ class OneVsRest(Estimator, OneVsRestParams, MLReadable, MLWritable

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127552478 --- Diff: python/pyspark/ml/classification.py --- @@ -1511,27 +1512,47 @@ class OneVsRest(Estimator, OneVsRestParams, MLReadable, MLWritable

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127552072 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,50 @@ class OneVsRestSuite extends

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127552824 --- Diff: python/pyspark/ml/classification.py --- @@ -1511,27 +1512,47 @@ class OneVsRest(Estimator, OneVsRestParams, MLReadable, MLWritable

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r127550679 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/HasParallelism.scala --- @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

<    4   5   6   7   8   9   10   11   12   13   >