[2/2] spark git commit: [SPARK-16621][SQL] Generate stable SQLs in SQLBuilder
[SPARK-16621][SQL] Generate stable SQLs in SQLBuilder
Currently, the generated SQLs have not-stable IDs for generated attributes.
The stable generated SQL will give more benefit for understanding or testing
the queries.
This PR provides stable SQL generation by the followings.
- Provide unique ids for generated subqueries, `gen_subquery_xxx`.
- Provide unique and stable ids for generated attributes, `gen_attr_xxx`.
**Before**
```scala
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res0: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_4` AS `1` FROM (SELECT 1 AS `gen_attr_4`) AS
gen_subquery_0
```
**After**
```scala
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res2: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
```
Pass the existing Jenkins tests.
Author: Dongjoon Hyun
Closes #14257 from dongjoon-hyun/SPARK-16621.
(cherry picked from commit 5b8e848bbfbc0c99a5faf758e40b188b0bbebb7b)
Signed-off-by: Reynold Xin
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/be9965b0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/be9965b0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/be9965b0
Branch: refs/heads/branch-2.0
Commit: be9965b077cded3d30a2d35342f3440f4708c357
Parents: 44234b1
Author: Dongjoon Hyun
Authored: Wed Jul 27 13:23:59 2016 +0800
Committer: Reynold Xin
Committed: Tue Jul 26 23:51:51 2016 -0700
--
.../apache/spark/sql/catalyst/SQLBuilder.scala | 23 +++-
sql/hive/src/test/resources/sqlgen/agg1.sql | 2 +-
sql/hive/src/test/resources/sqlgen/agg2.sql | 2 +-
sql/hive/src/test/resources/sqlgen/agg3.sql | 2 +-
.../sqlgen/aggregate_functions_and_window.sql | 2 +-
sql/hive/src/test/resources/sqlgen/case.sql | 2 +-
.../test/resources/sqlgen/case_with_else.sql| 2 +-
.../src/test/resources/sqlgen/case_with_key.sql | 2 +-
.../resources/sqlgen/case_with_key_and_else.sql | 2 +-
.../src/test/resources/sqlgen/cluster_by.sql| 2 +-
.../sqlgen/data_source_json_parquet_t0.sql | 2 +-
.../sqlgen/data_source_orc_parquet_t0.sql | 2 +-
.../sqlgen/data_source_parquet_parquet_t0.sql | 2 +-
.../resources/sqlgen/distinct_aggregation.sql | 2 +-
.../src/test/resources/sqlgen/distribute_by.sql | 2 +-
.../sqlgen/distribute_by_with_sort_by.sql | 2 +-
sql/hive/src/test/resources/sqlgen/except.sql | 2 +-
.../resources/sqlgen/filter_after_subquery.sql | 2 +-
.../resources/sqlgen/generate_with_other_1.sql | 2 +-
.../resources/sqlgen/generate_with_other_2.sql | 2 +-
.../sqlgen/generator_in_lateral_view_1.sql | 2 +-
.../sqlgen/generator_in_lateral_view_2.sql | 2 +-
.../sqlgen/generator_non_referenced_table_1.sql | 2 +-
.../sqlgen/generator_non_referenced_table_2.sql | 2 +-
.../resources/sqlgen/generator_non_udtf_1.sql | 2 +-
.../resources/sqlgen/generator_non_udtf_2.sql | 2 +-
.../sqlgen/generator_referenced_table_1.sql | 2 +-
.../sqlgen/generator_referenced_table_2.sql | 2 +-
.../sqlgen/generator_with_ambiguous_names_1.sql | 2 +-
.../sqlgen/generator_with_ambiguous_names_2.sql | 2 +-
.../sqlgen/generator_without_from_1.sql | 2 +-
.../sqlgen/generator_without_from_2.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_1.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_1.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_2.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_3.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_4.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_5.sql | 2 +-
sql/hive/src/test/resources/sqlgen/in.sql | 2 +-
.../src/test/resources/sqlgen/intersect.sql | 2 +-
.../src/test/resources/sqlgen/join_2_tables.sql | 2 +-
.../resources/sqlgen/json_tuple_generator_1.sql | 2 +-
.../resources/sqlgen/json_tuple_generator_2.sql | 2 +-
.../test/resources/sqlgen/multi_distinct.sql| 2 +-
.../nested_generator_in_lateral_view_1.sql | 2 +-
.../nested_generator_in_lateral_view_2.sql | 2 +-
sql/hive/src/test/resources/sqlgen/not_in.sql | 2 +-
sql/hive/src/test/resources/sqlgen/not_like.sql | 2 +-
.../resources/sqlgen/predicate_subquery.sql | 2 +-
.../sqlgen/regular_expressions_and_window.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_1_1.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_1_2.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_2_1.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_2_2.sql | 2
[2/2] spark git commit: [SPARK-16621][SQL] Generate stable SQLs in SQLBuilder
[SPARK-16621][SQL] Generate stable SQLs in SQLBuilder
## What changes were proposed in this pull request?
Currently, the generated SQLs have not-stable IDs for generated attributes.
The stable generated SQL will give more benefit for understanding or testing
the queries.
This PR provides stable SQL generation by the followings.
- Provide unique ids for generated subqueries, `gen_subquery_xxx`.
- Provide unique and stable ids for generated attributes, `gen_attr_xxx`.
**Before**
```scala
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res0: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_4` AS `1` FROM (SELECT 1 AS `gen_attr_4`) AS
gen_subquery_0
```
**After**
```scala
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res2: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS
gen_subquery_0
```
## How was this patch tested?
Pass the existing Jenkins tests.
Author: Dongjoon Hyun
Closes #14257 from dongjoon-hyun/SPARK-16621.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5b8e848b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5b8e848b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5b8e848b
Branch: refs/heads/master
Commit: 5b8e848bbfbc0c99a5faf758e40b188b0bbebb7b
Parents: 738b4cc
Author: Dongjoon Hyun
Authored: Wed Jul 27 13:23:59 2016 +0800
Committer: Cheng Lian
Committed: Wed Jul 27 13:23:59 2016 +0800
--
.../apache/spark/sql/catalyst/SQLBuilder.scala | 23 +++-
sql/hive/src/test/resources/sqlgen/agg1.sql | 2 +-
sql/hive/src/test/resources/sqlgen/agg2.sql | 2 +-
sql/hive/src/test/resources/sqlgen/agg3.sql | 2 +-
.../sqlgen/aggregate_functions_and_window.sql | 2 +-
sql/hive/src/test/resources/sqlgen/case.sql | 2 +-
.../test/resources/sqlgen/case_with_else.sql| 2 +-
.../src/test/resources/sqlgen/case_with_key.sql | 2 +-
.../resources/sqlgen/case_with_key_and_else.sql | 2 +-
.../src/test/resources/sqlgen/cluster_by.sql| 2 +-
.../sqlgen/data_source_json_parquet_t0.sql | 2 +-
.../sqlgen/data_source_orc_parquet_t0.sql | 2 +-
.../sqlgen/data_source_parquet_parquet_t0.sql | 2 +-
.../resources/sqlgen/distinct_aggregation.sql | 2 +-
.../src/test/resources/sqlgen/distribute_by.sql | 2 +-
.../sqlgen/distribute_by_with_sort_by.sql | 2 +-
sql/hive/src/test/resources/sqlgen/except.sql | 2 +-
.../resources/sqlgen/filter_after_subquery.sql | 2 +-
.../resources/sqlgen/generate_with_other_1.sql | 2 +-
.../resources/sqlgen/generate_with_other_2.sql | 2 +-
.../sqlgen/generator_in_lateral_view_1.sql | 2 +-
.../sqlgen/generator_in_lateral_view_2.sql | 2 +-
.../sqlgen/generator_non_referenced_table_1.sql | 2 +-
.../sqlgen/generator_non_referenced_table_2.sql | 2 +-
.../resources/sqlgen/generator_non_udtf_1.sql | 2 +-
.../resources/sqlgen/generator_non_udtf_2.sql | 2 +-
.../sqlgen/generator_referenced_table_1.sql | 2 +-
.../sqlgen/generator_referenced_table_2.sql | 2 +-
.../sqlgen/generator_with_ambiguous_names_1.sql | 2 +-
.../sqlgen/generator_with_ambiguous_names_2.sql | 2 +-
.../sqlgen/generator_without_from_1.sql | 2 +-
.../sqlgen/generator_without_from_2.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_1.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_1.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_2.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_3.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_4.sql | 2 +-
.../test/resources/sqlgen/grouping_sets_2_5.sql | 2 +-
sql/hive/src/test/resources/sqlgen/in.sql | 2 +-
.../src/test/resources/sqlgen/intersect.sql | 2 +-
.../src/test/resources/sqlgen/join_2_tables.sql | 2 +-
.../resources/sqlgen/json_tuple_generator_1.sql | 2 +-
.../resources/sqlgen/json_tuple_generator_2.sql | 2 +-
.../test/resources/sqlgen/multi_distinct.sql| 2 +-
.../nested_generator_in_lateral_view_1.sql | 2 +-
.../nested_generator_in_lateral_view_2.sql | 2 +-
sql/hive/src/test/resources/sqlgen/not_in.sql | 2 +-
sql/hive/src/test/resources/sqlgen/not_like.sql | 2 +-
.../resources/sqlgen/predicate_subquery.sql | 2 +-
.../sqlgen/regular_expressions_and_window.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_1_1.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_1_2.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_2_1.sql | 2 +-
.../test/resources/sqlgen/rollup_cube_2_2.sql | 2 +-
.../test/resou
