[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-116848080 Merging this in. Thanks @ilganeli! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6686 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115029251 @ilganeli Once you correct the parameter (make it consistent), this looks good to me. Unfortunately, this PR cannot catch 1.4.1 release, which should only include bugfix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115039570 LGTM, waiting for tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r33203825 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,53 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name, data_type=None, nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema. The method accepts +either: +a) A single parameter which is a StructField object. +b) Between 2 and 4 parameters as (name, data_type, nullable (optional), + metadata(optional). The data_type parameter may be either a String or a DataType object + + struct1 = StructType().add(f1, StringType(), True).add(f2, StringType(), True, None) + struct2 = StructType([StructField(f1, StringType(), True),\ + StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + struct1 = StructType().add(StructField(f1, StringType(), True)) + struct2 = StructType([StructField(f1, StringType(), True)]) + struct1 == struct2 +True + struct1 = StructType().add(f1, string, True) + struct2 = StructType([StructField(f1, StringType(), True)]) + struct1 == struct2 +True + +:param nameOrStructField: Either the name of the field or a StructField object --- End diff -- This is different than `name`. I'd like to use `field`, which could be FieldType or string of name. How do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115032140 [Test build #35728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35728/consoleFull) for PR 6686 at commit [`27c1de1`](https://github.com/apache/spark/commit/27c1de1a4905b1e4b6bc315f688f900f3b55332a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115030576 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115030558 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115053843 [Test build #35728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35728/console) for PR 6686 at commit [`27c1de1`](https://github.com/apache/spark/commit/27c1de1a4905b1e4b6bc315f688f900f3b55332a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-115053967 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114973836 @davies @rxin Are there any further suggestions? Would love to get this into 1.4.1, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114186900 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6686#discussion_r32959777
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
---
@@ -43,24 +43,6 @@ private[r] object SQLUtils {
StructType(fields)
}
- def getSQLDataType(dataType: String): DataType = {
-dataType match {
- case byte = org.apache.spark.sql.types.ByteType
- case integer = org.apache.spark.sql.types.IntegerType
- case double = org.apache.spark.sql.types.DoubleType
- case numeric = org.apache.spark.sql.types.DoubleType
- case character = org.apache.spark.sql.types.StringType
--- End diff --
These are R types, cc @shivaram
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114187161 [Test build #35460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35460/consoleFull) for PR 6686 at commit [`5fef5a4`](https://github.com/apache/spark/commit/5fef5a479ef3638ce481887b24bf43afa61d8f1d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114189600 [Test build #35460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35460/console) for PR 6686 at commit [`5fef5a4`](https://github.com/apache/spark/commit/5fef5a479ef3638ce481887b24bf43afa61d8f1d). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32959687 --- Diff: python/pyspark/sql/types.py --- @@ -56,6 +56,32 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other) +def from_string(self, data_type): --- End diff -- The alias (`integer`, `raw`, `logical`) are only used for R, so we could should `_parse_datatype_json_value ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114189619 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114186871 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32958201 --- Diff: python/pyspark/sql/types.py --- @@ -56,6 +56,32 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other) +def from_string(self, data_type): --- End diff -- Is it ok to leave it as is to be specific? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114243273 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114285248 [Test build #35482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35482/console) for PR 6686 at commit [`467d836`](https://github.com/apache/spark/commit/467d8367341f2e53dbc1575c6322b8695801519d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114285290 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114243212 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-114243827 [Test build #35482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35482/consoleFull) for PR 6686 at commit [`467d836`](https://github.com/apache/spark/commit/467d8367341f2e53dbc1575c6322b8695801519d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32806503 --- Diff: python/pyspark/sql/types.py --- @@ -56,6 +56,32 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other) +def from_string(self, data_type): --- End diff -- I think we could use `_parse_datatype_json_value` here, but lose some type alias. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32806501 --- Diff: python/pyspark/sql/types.py --- @@ -56,6 +56,32 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other) +def from_string(self, data_type): +assert(isinstance(data_type, str)) + +if data_type == byte: +return ByteType +elif data_type == long: +return LongType +elif data_type == float: +return FloatType +elif data_type == int or data_type == integer: +return IntegerType +elif data_type == double or data_type == numeric: +return DoubleType +elif data_type == string or data_type == character: --- End diff -- This is Python, we don't need to have R types here (numeric and character) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6686#discussion_r32806472
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
---
@@ -43,26 +43,8 @@ private[r] object SQLUtils {
StructType(fields)
}
- def getSQLDataType(dataType: String): DataType = {
-dataType match {
- case byte = org.apache.spark.sql.types.ByteType
- case integer = org.apache.spark.sql.types.IntegerType
- case double = org.apache.spark.sql.types.DoubleType
- case numeric = org.apache.spark.sql.types.DoubleType
- case character = org.apache.spark.sql.types.StringType
- case string = org.apache.spark.sql.types.StringType
- case binary = org.apache.spark.sql.types.BinaryType
- case raw = org.apache.spark.sql.types.BinaryType
- case logical = org.apache.spark.sql.types.BooleanType
- case boolean = org.apache.spark.sql.types.BooleanType
- case timestamp = org.apache.spark.sql.types.TimestampType
- case date = org.apache.spark.sql.types.DateType
- case _ = throw new IllegalArgumentException(sInvaid type
$dataType)
-}
- }
-
def createStructField(name: String, dataType: String, nullable:
Boolean): StructField = {
-val dtObj = getSQLDataType(dataType)
+val dtObj = DataTypeParser.parse(dataType)
--- End diff --
These two are different, `getSQLDataType` is used for R.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32804117 --- Diff: python/pyspark/sql/types.py --- @@ -56,6 +56,32 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other) +def from_string(self, data_type): --- End diff -- @davies why don't we just call the jvm parse type string here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32804096 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +393,47 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=None, nullable=True, metadata=None): --- End diff -- can we change name_or_struct_field to just name, and just explain we support passing in a struct type in the docstring? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113189491 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113189528 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113190209 [Test build #35137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35137/consoleFull) for PR 6686 at commit [`8109e00`](https://github.com/apache/spark/commit/8109e0067b3abce6f4eec937b39c6d7db2eb6b71). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32751352 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,43 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=None, nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema + + struct1 = StructType().add(f1, StringType(), True).add(f2, StringType(), True, None) + struct2 = StructType([StructField(f1, StringType(), True),\ + StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + struct1 = (StructType().add(StructField(f1, StringType(), True)) +... .add(StructField(f2, StringType(), True, None))) + struct2 = StructType([StructField(f1, StringType(), True), +... StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + +:param nameOrStructField: Either the name of the field or a StructField object +:param data_type: If present, the DataType of the StructField to create +:param nullable: Whether the field to add should be nullable (default True) +:param metadata: Any additional metadata (default None) +:return: a new updated StructType + +if isinstance(name_or_struct_field, StructField): +self.fields.append(name_or_struct_field) +return self --- End diff -- Nit: you can save a line by put this out of `if` (at last). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32751787 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,43 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=None, nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema + + struct1 = StructType().add(f1, StringType(), True).add(f2, StringType(), True, None) + struct2 = StructType([StructField(f1, StringType(), True),\ + StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + struct1 = (StructType().add(StructField(f1, StringType(), True)) +... .add(StructField(f2, StringType(), True, None))) + struct2 = StructType([StructField(f1, StringType(), True), +... StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + +:param nameOrStructField: Either the name of the field or a StructField object +:param data_type: If present, the DataType of the StructField to create +:param nullable: Whether the field to add should be nullable (default True) +:param metadata: Any additional metadata (default None) +:return: a new updated StructType + +if isinstance(name_or_struct_field, StructField): +self.fields.append(name_or_struct_field) +return self +else: +if isinstance(name_or_struct_field, str) and data_type is None: +raise ValueError(Must specify DataType if passing name of struct_field to create.) +self.fields.append(StructField(name_or_struct_field, data_type, nullable, metadata)) --- End diff -- Do we need to check the type of `data_type`? In scala, we support that `data_type` as a string, so it will be good that we also support it here. At least, we should check the type. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113354797 [Test build #35201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35201/console) for PR 6686 at commit [`4085489`](https://github.com/apache/spark/commit/4085489efc22de12e9b695392872ee24ad9cd04c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113354814 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113331823 [Test build #35201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35201/consoleFull) for PR 6686 at commit [`4085489`](https://github.com/apache/spark/commit/4085489efc22de12e9b695392872ee24ad9cd04c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113331787 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113331773 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113232083 [Test build #35137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35137/console) for PR 6686 at commit [`8109e00`](https://github.com/apache/spark/commit/8109e0067b3abce6f4eec937b39c6d7db2eb6b71). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113232120 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32785670 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,43 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=None, nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema + + struct1 = StructType().add(f1, StringType(), True).add(f2, StringType(), True, None) + struct2 = StructType([StructField(f1, StringType(), True),\ + StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + struct1 = (StructType().add(StructField(f1, StringType(), True)) +... .add(StructField(f2, StringType(), True, None))) + struct2 = StructType([StructField(f1, StringType(), True), +... StructField(f2, StringType(), True, None)]) + struct1 == struct2 +True + +:param nameOrStructField: Either the name of the field or a StructField object +:param data_type: If present, the DataType of the StructField to create +:param nullable: Whether the field to add should be nullable (default True) +:param metadata: Any additional metadata (default None) +:return: a new updated StructType + +if isinstance(name_or_struct_field, StructField): +self.fields.append(name_or_struct_field) +return self +else: +if isinstance(name_or_struct_field, str) and data_type is None: +raise ValueError(Must specify DataType if passing name of struct_field to create.) +self.fields.append(StructField(name_or_struct_field, data_type, nullable, metadata)) --- End diff -- @davies Is there an equivalent of the DataTypeParser in Python? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113307230 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113307245 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113307627 [Test build #35184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35184/consoleFull) for PR 6686 at commit [`3670cf5`](https://github.com/apache/spark/commit/3670cf5be7d8430b7d661a1f765bc7103b05d21b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113307870 [Test build #35184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35184/console) for PR 6686 at commit [`3670cf5`](https://github.com/apache/spark/commit/3670cf5be7d8430b7d661a1f765bc7103b05d21b). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113307873 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113063566 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113063564 [Test build #35109 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35109/console) for PR 6686 at commit [`e7ba7e0`](https://github.com/apache/spark/commit/e7ba7e03cb11b68ef990945f4c2fa53fbe9ea78d). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113079072 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113079089 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113062905 @davies Made the changes you suggested. Only thing I didn't know how to do is to intercept an exception in Python. Is there a cleaner way of doing it than I added? The way I'm doing it right now seems hacky but I didn't see a better example in the other cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113063328 [Test build #35109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35109/consoleFull) for PR 6686 at commit [`e7ba7e0`](https://github.com/apache/spark/commit/e7ba7e03cb11b68ef990945f4c2fa53fbe9ea78d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113063174 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113063190 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113079303 [Test build #35117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35117/consoleFull) for PR 6686 at commit [`41ab686`](https://github.com/apache/spark/commit/41ab6867faad559902d7c8c92e300959096cd59a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113102560 [Test build #35117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35117/console) for PR 6686 at commit [`41ab686`](https://github.com/apache/spark/commit/41ab6867faad559902d7c8c92e300959096cd59a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-113102620 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32656127 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): --- End diff -- Davies - totally agree. This was changed specifically to consolidate to a single method as suggested by Reynold. I initially had separate add methods - one which accepted a StructField and one which accepted the 4 parameters, the first two of which were defined. What would you suggest? My preference is to break this out into two methods for clarity and to avoid the problem you mention. Thank you, Ilya Ganelin -Original Message- From: Davies Liu [notificati...@github.commailto:notificati...@github.com] Sent: Wednesday, June 17, 2015 01:18 PM Eastern Standard Time To: apache/spark Cc: Ganelin, Ilya Subject: Re: [spark] [SPARK-8056][SQL] Design an easier way to construct schema for both Scala and Python (#6686) In python/pyspark/sql/types.pyhttps://github.com/apache/spark/pull/6686#discussion_r32650869: @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): What's the use cases that we should have StructType without specifying the dataType of each column? In createDataFrame, if a schema of StructType is provided, it will not try to infer the data types, so it does not work with StructType with NoneType in it. — Reply to this email directly or view it on GitHubhttps://github.com/apache/spark/pull/6686/files#r32650869. The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32656949 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): --- End diff -- I think you could use `None` as default value of `dataType`, and raise an exception if `name_or_struct_field ` is string and `dataType` is None --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32650599 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema + struct1 = StructType().add(f1, StringType(), True).add(f2, StringType(), True, None) + struct2 = StructType([StructField(f1, StringType(), True),\ --- End diff -- The doctests will be taken as examples in API docs, should be simple. We could put other test cases into sql/tests.py --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32650869 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): --- End diff -- What's the use cases that we should have StructType without specifying the dataType of each column? In `createDataFrame`, if a schema of StructType is provided, it will not try to infer the data types, so it does not work with StructType with NoneType in it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6686#discussion_r32650363 --- Diff: python/pyspark/sql/types.py --- @@ -368,8 +367,49 @@ def __init__(self, fields): struct1 == struct2 False -assert all(isinstance(f, DataType) for f in fields), fields should be a list of DataType -self.fields = fields +if not fields: +self.fields = [] +else: +self.fields = fields +assert all(isinstance(f, StructField) for f in fields),\ +fields should be a list of StructField + +def add(self, name_or_struct_field, data_type=NullType(), nullable=True, metadata=None): + +Construct a StructType by adding new elements to it to define the schema --- End diff -- add new line here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112480475 @rxin Is this good to go? Looking forward to getting this merged! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112110192 [Test build #34944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34944/consoleFull) for PR 6686 at commit [`e240040`](https://github.com/apache/spark/commit/e240040cc6d4c206498e8cfc27429f898ba5dab7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112099273 [Test build #34943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34943/consoleFull) for PR 6686 at commit [`bab7823`](https://github.com/apache/spark/commit/bab78239afc4c3148f38e7f79b440ed5235a287e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112098983 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112098935 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112099705 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112099702 [Test build #34943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34943/console) for PR 6686 at commit [`bab7823`](https://github.com/apache/spark/commit/bab78239afc4c3148f38e7f79b440ed5235a287e). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112108881 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112108953 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112160908 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112160926 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112161079 [Test build #34954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34954/consoleFull) for PR 6686 at commit [`15868fa`](https://github.com/apache/spark/commit/15868fa5d071de4a5c2fd554455e04c4c904b688). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112141366 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112141325 [Test build #34944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34944/console) for PR 6686 at commit [`e240040`](https://github.com/apache/spark/commit/e240040cc6d4c206498e8cfc27429f898ba5dab7). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112151316 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112201653 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112201545 [Test build #34954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34954/console) for PR 6686 at commit [`15868fa`](https://github.com/apache/spark/commit/15868fa5d071de4a5c2fd554455e04c4c904b688). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112182686 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-112182645 [Test build #34952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34952/console) for PR 6686 at commit [`b79b992`](https://github.com/apache/spark/commit/b79b992ecf5e8ffab34eb12090b9e4b0deab36da). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111980790 [Test build #34918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34918/console) for PR 6686 at commit [`73d4677`](https://github.com/apache/spark/commit/73d46774b91175e20194739d61e627c848fec4a3). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111980837 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111956199 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111957224 [Test build #34918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34918/consoleFull) for PR 6686 at commit [`73d4677`](https://github.com/apache/spark/commit/73d46774b91175e20194739d61e627c848fec4a3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111956606 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111956577 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111805854 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111805844 [Test build #34878 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34878/console) for PR 6686 at commit [`0484d7a`](https://github.com/apache/spark/commit/0484d7ac6f479a6da91f8b8b7f8e1a60576e00d0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111805171 [Test build #34879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34879/console) for PR 6686 at commit [`04cbf0c`](https://github.com/apache/spark/commit/04cbf0c3610c1dd21ac5b27b08b89621e49409e7). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111805174 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111796022 [Test build #34879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34879/consoleFull) for PR 6686 at commit [`04cbf0c`](https://github.com/apache/spark/commit/04cbf0c3610c1dd21ac5b27b08b89621e49409e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111794364 [Test build #34870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34870/console) for PR 6686 at commit [`689e54d`](https://github.com/apache/spark/commit/689e54dc9992ca9218442729cb8b22e12151cc70). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111794368 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111791362 [Test build #34872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34872/console) for PR 6686 at commit [`6aeb740`](https://github.com/apache/spark/commit/6aeb74018473c5850b7838a0784494ea079c3346). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111791373 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111794023 [Test build #34878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34878/consoleFull) for PR 6686 at commit [`0484d7a`](https://github.com/apache/spark/commit/0484d7ac6f479a6da91f8b8b7f8e1a60576e00d0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111795884 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8056][SQL] Design an easier way to cons...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6686#issuecomment-111795912 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
