Repository: spark Updated Branches: refs/heads/branch-2.4 f2d502223 -> 3682d29f4
[SPARK-25072][PYSPARK] Forbid extra value for custom Row ## What changes were proposed in this pull request? Add value length check in `_create_row`, forbid extra value for custom Row in PySpark. ## How was this patch tested? New UT in pyspark-sql Closes #22140 from xuanyuanking/SPARK-25072. Lead-authored-by: liyuanjian <liyuanj...@baidu.com> Co-authored-by: Yuanjian Li <xyliyuanj...@gmail.com> Signed-off-by: Bryan Cutler <cutl...@gmail.com> (cherry picked from commit c84bc40d7f33c71eca1c08f122cd60517f34c1f8) Signed-off-by: Bryan Cutler <cutl...@gmail.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3682d29f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3682d29f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3682d29f Branch: refs/heads/branch-2.4 Commit: 3682d29f45870031d9dc4e812accbfbb583cc52a Parents: f2d5022 Author: liyuanjian <liyuanj...@baidu.com> Authored: Thu Sep 6 10:17:29 2018 -0700 Committer: Bryan Cutler <cutl...@gmail.com> Committed: Thu Sep 6 10:17:46 2018 -0700 ---------------------------------------------------------------------- python/pyspark/sql/tests.py | 4 ++++ python/pyspark/sql/types.py | 3 +++ 2 files changed, 7 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/3682d29f/python/pyspark/sql/tests.py ---------------------------------------------------------------------- diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py index 81c0af0..6d9d636 100644 --- a/python/pyspark/sql/tests.py +++ b/python/pyspark/sql/tests.py @@ -277,6 +277,10 @@ class DataTypeTests(unittest.TestCase): struct_field = StructField("a", IntegerType()) self.assertRaises(TypeError, struct_field.typeName) + def test_invalid_create_row(self): + row_class = Row("c1", "c2") + self.assertRaises(ValueError, lambda: row_class(1, 2, 3)) + class SQLTests(ReusedSQLTestCase): http://git-wip-us.apache.org/repos/asf/spark/blob/3682d29f/python/pyspark/sql/types.py ---------------------------------------------------------------------- diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py index 0b61707..ce1d004 100644 --- a/python/pyspark/sql/types.py +++ b/python/pyspark/sql/types.py @@ -1500,6 +1500,9 @@ class Row(tuple): # let object acts like class def __call__(self, *args): """create new Row object""" + if len(args) > len(self): + raise ValueError("Can not create Row with fields %s, expected %d values " + "but got %s" % (self, len(self), args)) return _create_row(self, args) def __getitem__(self, item): --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
