[ https://issues.apache.org/jira/browse/SPARK-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Cutler updated SPARK-15018: --------------------------------- Description: When fitting a PySpark Pipeline with no stages, it should work as an identity transformer. Instead the following error is raised: {noformat} Traceback (most recent call last): File "./spark/python/pyspark/ml/base.py", line 64, in fit return self._fit(dataset) File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit for stage in stages: TypeError: 'NoneType' object is not iterable {noformat} The param {{stages}} needs to be an empty list and {{getStages}} should call {{getOrDefault}}. Also, since the default value is {{None}} is then changed to and empty list {{[]}}, this never changes the value if passed in as a keyword argument. Instead, the {{kwargs}} value should be changed directly if {{stages is None}}. For example {noformat} if stages is None: stages = [] {noformat} should be this {noformat} if stages is None: kwargs['stages'] = [] {noformat} However, since there is no default value in the Scala implementation, assigning a default here is not needed and should be cleaned up. The pydocs should better indicate that stages is required to be a list. was: When fitting a PySpark Pipeline with no stages, it should work as an identity transformer. Instead the following error is raised: {noformat} Traceback (most recent call last): File "./spark/python/pyspark/ml/base.py", line 64, in fit return self._fit(dataset) File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit for stage in stages: TypeError: 'NoneType' object is not iterable {noformat} The param {{stages}} needs to be an empty list and {{getStages}} should call {{getOrDefault}}. Also, since the default value is {{None}} is then changed to and empty list {{[]}}, this never changes the value if passed in as a keyword argument. Instead, the {{kwargs}} value should be changed directly if {{stages is None}}. For example {noformat} if stages is None: stages = [] {noformat} should be this {noformat} if stages is None: kwargs['stages'] = [] {noformat} However, since there is no default value in the Scala implementation, assigning a default here is not needed and should be cleaned up. > PySpark ML Pipeline raises unclear error when no stages set > ----------------------------------------------------------- > > Key: SPARK-15018 > URL: https://issues.apache.org/jira/browse/SPARK-15018 > Project: Spark > Issue Type: Improvement > Components: ML, PySpark > Reporter: Bryan Cutler > Assignee: Bryan Cutler > Priority: Minor > > When fitting a PySpark Pipeline with no stages, it should work as an identity > transformer. Instead the following error is raised: > {noformat} > Traceback (most recent call last): > File "./spark/python/pyspark/ml/base.py", line 64, in fit > return self._fit(dataset) > File "./spark/python/pyspark/ml/pipeline.py", line 99, in _fit > for stage in stages: > TypeError: 'NoneType' object is not iterable > {noformat} > The param {{stages}} needs to be an empty list and {{getStages}} should call > {{getOrDefault}}. > Also, since the default value is {{None}} is then changed to and empty list > {{[]}}, this never changes the value if passed in as a keyword argument. > Instead, the {{kwargs}} value should be changed directly if {{stages is > None}}. > For example > {noformat} > if stages is None: > stages = [] > {noformat} > should be this > {noformat} > if stages is None: > kwargs['stages'] = [] > {noformat} > However, since there is no default value in the Scala implementation, > assigning a default here is not needed and should be cleaned up. The pydocs > should better indicate that stages is required to be a list. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org