[ https://issues.apache.org/jira/browse/SPARK-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guru Prabhakar Reddy Marthala updated SPARK-20592: -------------------------------------------------- Attachment: (was: Chaitanya Duvvuru.doc) > Alter table concatenate is not working as expected. > --------------------------------------------------- > > Key: SPARK-20592 > URL: https://issues.apache.org/jira/browse/SPARK-20592 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.1.0 > Reporter: Guru Prabhakar Reddy Marthala > Labels: hive, pyspark > > Created a table using CTAS from csv to parquet.Parquet table generated > numerous small files.tried alter table concatenate but it's not working as > expected. > spark.sql("CREATE TABLE flight.flight_data(year INT, month INT, day INT, > day_of_week INT, dep_time INT, crs_dep_time INT, arr_time INT, > crs_arr_time INT, unique_carrier STRING, flight_num INT, tail_num > STRING, actual_elapsed_time INT, crs_elapsed_time INT, air_time INT, > arr_delay INT, dep_delay INT, origin STRING, dest STRING, distance > INT, taxi_in INT, taxi_out INT, cancelled INT, cancellation_code > STRING, diverted INT, carrier_delay STRING, weather_delay STRING, > nas_delay STRING, security_delay STRING, late_aircraft_delay STRING) ROW > FORMAT DELIMITED FIELDS TERMINATED BY ',' stored as textfile") > spark.sql("load data local INPATH 'i:/2008/2008.csv' INTO TABLE > flight.flight_data") > spark.sql("create table flight.flight_data_pq stored as parquet as select * > from flight.flight_data") > spark.sql("create table flight.flight_data_orc stored as orc as select * from > flight.flight_data") > pyspark.sql.utils.ParseException: u'\nOperation not allowed: alter table > concatenate(line 1, pos 0)\n\n== SQL ==\nalter table > flight_data.flight_data_pq concatenate\n^^^\n' > Tried on both orc and parquet format.It's not working. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org