[ https://issues.apache.org/jira/browse/SPARK-18086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630027#comment-15630027 ]
Apache Spark commented on SPARK-18086: -------------------------------------- User 'rdblue' has created a pull request for this issue: https://github.com/apache/spark/pull/15738 > Regression: Hive variables no longer work in Spark 2.0 > ------------------------------------------------------ > > Key: SPARK-18086 > URL: https://issues.apache.org/jira/browse/SPARK-18086 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Reporter: Ryan Blue > > The behavior of variables in the SQL shell has changed from 1.6 to 2.0. > Specifically, --hivevar name=value and {{SET hivevar:name=value}} no longer > work. Queries that worked correctly in 1.6 will either fail or produce > unexpected results in 2.0 so I think this is a regression that should be > addressed. > Hive and Spark 1.6 work like this: > 1. Command-line args --hiveconf and --hivevar can be used to set session > properties. --hiveconf properties are added to the Hadoop Configuration. > 2. {{SET}} adds a Hive Configuration property, {{SET hivevar:<name>=<value>}} > adds a Hive var. > 3. Hive vars can be substituted into queries by name, and Configuration > properties can be substituted using {{hiveconf:name}}. > In 2.0, hiveconf, sparkconf, and conf variable prefixes are all removed, then > the value in SQLConf for the rest of the key is returned. SET adds properties > to the session config and (according to [a > comment|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala#L28]) > the Hadoop configuration "during I/O". > {code:title=Hive and Spark 1.6.1 behavior} > [user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2 > spark-sql> select "${hiveconf:test.conf}"; > 1 > spark-sql> select "${test.conf}"; > ${test.conf} > spark-sql> select "${hivevar:test.var}"; > 2 > spark-sql> select "${test.var}"; > 2 > spark-sql> set test.set=3; > SET test.set=3 > spark-sql> select "${test.set}" > "${test.set}" > spark-sql> select "${hivevar:test.set}" > "${hivevar:test.set}" > spark-sql> select "${hiveconf:test.set}" > 3 > spark-sql> set hivevar:test.setvar=4; > SET hivevar:test.setvar=4 > spark-sql> select "${hivevar:test.setvar}"; > 4 > spark-sql> select "${test.setvar}"; > 4 > {code} > {code:title=Spark 2.0.0 behavior} > [user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2 > spark-sql> select "${hiveconf:test.conf}"; > 1 > spark-sql> select "${test.conf}"; > 1 > spark-sql> select "${hivevar:test.var}"; > ${hivevar:test.var} > spark-sql> select "${test.var}"; > ${test.var} > spark-sql> set test.set=3; > test.set 3 > spark-sql> select "${test.set}"; > 3 > spark-sql> set hivevar:test.setvar=4; > hivevar:test.setvar 4 > spark-sql> select "${hivevar:test.setvar}"; > 4 > spark-sql> select "${test.setvar}"; > ${test.setvar} > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org