[ https://issues.apache.org/jira/browse/SPARK-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543403#comment-14543403 ]
Santiago M. Mola commented on SPARK-7012: ----------------------------------------- In Spark SQL, every expression can be nullable or not (i.e. values can be null or not). All Spark SQL and Catalyst internals support specifying this. See, for example, StructField, which is the relevant class for schemas: https://github.com/apache/spark/blob/2d6612cc8b98f767d73c4d15e4065bf3d6c12ea7/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructField.scala#L31 Or AttributeReference: https://github.com/apache/spark/blob/c1080b6fddb22d84694da2453e46a03fbc041576/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L166 However, when creating a temporary table through a SQL statement (CREATE TEMPORARY TABLE), there is no way of specifying if a column is nullable or not (it will be always nullable by default). Standard SQL supports a constraint called "NOT NULL" to specify that a column is not nullable. See: http://www.w3schools.com/sql/sql_notnull.asp In order to implement this, the parser for "CREATE TEMPORARY TABLE", that is, DDLParser, should be modifyed to allow "NOT NULL" and set nullable = false accordingly in StructField. See: https://github.com/apache/spark/blob/0595b6de8f1da04baceda082553c2aa1aa2cb006/sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala#L176 > Add support for NOT NULL modifier for column definitions on DDLParser > --------------------------------------------------------------------- > > Key: SPARK-7012 > URL: https://issues.apache.org/jira/browse/SPARK-7012 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.3.0 > Reporter: Santiago M. Mola > Priority: Minor > Labels: easyfix > > Add support for NOT NULL modifier for column definitions on DDLParser. This > would add support for the following syntax: > CREATE TEMPORARY TABLE (field INTEGER NOT NULL) ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org