[
https://issues.apache.org/jira/browse/SPARK-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Cutler updated SPARK-15018:
---------------------------------
Description:
When fitting a PySpark Pipeline with no stages, it should work as an identity
transformer. Instead the following error is raised:
{noformat}
Traceback (most recent call last):
File "./spark/python/pyspark/ml/base.py", line 64, in fit
return self._fit(dataset)
File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit
for stage in stages:
TypeError: 'NoneType' object is not iterable
{noformat}
The param {{stages}} needs to be an empty list and {{getStages}} should call
{{getOrDefault}}.
Also, since the default value is {{None}} is then changed to and empty list
{{[]}}, this never changes the value if passed in as a keyword argument.
Instead, the {{kwargs}} value should be changed directly if {{stages is None}}.
For example
{noformat}
if stages is None:
stages = []
{noformat}
should be this
{noformat}
if stages is None:
kwargs['stages'] = []
{noformat}
However, since there is no default value in the Scala implementation, assigning
a default here is not needed and should be cleaned up. The pydocs should
better indicate that stages is required to be a list.
was:
When fitting a PySpark Pipeline with no stages, it should work as an identity
transformer. Instead the following error is raised:
{noformat}
Traceback (most recent call last):
File "./spark/python/pyspark/ml/base.py", line 64, in fit
return self._fit(dataset)
File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit
for stage in stages:
TypeError: 'NoneType' object is not iterable
{noformat}
The param {{stages}} needs to be an empty list and {{getStages}} should call
{{getOrDefault}}.
Also, since the default value is {{None}} is then changed to and empty list
{{[]}}, this never changes the value if passed in as a keyword argument.
Instead, the {{kwargs}} value should be changed directly if {{stages is None}}.
For example
{noformat}
if stages is None:
stages = []
{noformat}
should be this
{noformat}
if stages is None:
kwargs['stages'] = []
{noformat}
However, since there is no default value in the Scala implementation, assigning
a default here is not needed and should be cleaned up.
> PySpark ML Pipeline raises unclear error when no stages set
> -----------------------------------------------------------
>
> Key: SPARK-15018
> URL: https://issues.apache.org/jira/browse/SPARK-15018
> Project: Spark
> Issue Type: Improvement
> Components: ML, PySpark
> Reporter: Bryan Cutler
> Assignee: Bryan Cutler
> Priority: Minor
>
> When fitting a PySpark Pipeline with no stages, it should work as an identity
> transformer. Instead the following error is raised:
> {noformat}
> Traceback (most recent call last):
> File "./spark/python/pyspark/ml/base.py", line 64, in fit
> return self._fit(dataset)
> File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit
> for stage in stages:
> TypeError: 'NoneType' object is not iterable
> {noformat}
> The param {{stages}} needs to be an empty list and {{getStages}} should call
> {{getOrDefault}}.
> Also, since the default value is {{None}} is then changed to and empty list
> {{[]}}, this never changes the value if passed in as a keyword argument.
> Instead, the {{kwargs}} value should be changed directly if {{stages is
> None}}.
> For example
> {noformat}
> if stages is None:
> stages = []
> {noformat}
> should be this
> {noformat}
> if stages is None:
> kwargs['stages'] = []
> {noformat}
> However, since there is no default value in the Scala implementation,
> assigning a default here is not needed and should be cleaned up. The pydocs
> should better indicate that stages is required to be a list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
