[ https://issues.apache.org/jira/browse/SPARK-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567061#comment-14567061 ]
Reynold Xin commented on SPARK-5302: ------------------------------------ [~btiernay] is this resolved now SPARK-5182 is resolved? > Add support for SQLContext "partition" columns > ---------------------------------------------- > > Key: SPARK-5302 > URL: https://issues.apache.org/jira/browse/SPARK-5302 > Project: Spark > Issue Type: New Feature > Components: SQL > Reporter: Bob Tiernay > > For {{SQLContext}} (not {{HiveContext}}) it would be very convenient to > support a virtual column that maps to part of the the file path, similar to > what is done in Hive for partitions (e.g. {{/data/clicks/dt=2015-01-01/}} > where {{dt}} is a column of type {{TEXT}}). > The API could allow the user to type the column using an appropriate > {{DataType}} instance. This new field could be addressed in SQL statements > much the same as is done in Hive. > As a consequence, pruning of partitions could be possible when executing a > query and also remove the need to materialize a column in each logical > partition that is already encoded in the path name. Furthermore, this would > provide an nice interop and migration strategy for Hive users who may one day > use {{SQLContext}} directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org