Re: insert overwrite to hive orc table in aws

2018-05-01 Thread Gopal Vijayaraghavan
> delta_000_000 ... > I am using Glue data catalog as metastore, so should there be any link up to > these tables from hive? That would be why transactions are returning as 0 (there is never a transaction 0), because it is not using a Hive standard metastore. You might not be able to u

Re: insert overwrite to hive orc table in aws

2018-05-01 Thread Sowjanya Kakarala
Ya I did try the select statement but it did not return anything. In S3 location I saw that it created buckets path as "delta_000_000" but there is nothing in it. I am using Glue data catalog as metastore, so should there be any link up to these tables from hive? or to see if data catalog h

Re: insert overwrite to hive orc table in aws

2018-05-01 Thread Furcy Pin
I suggest that you do a DESCRIBE FORMATTED orc and to check directly on s3 the location of your data. Did you also try a SELECT * FROM orc LIMIT 10; ? Did it return something? On Fri, 27 Apr 2018, 21:30 Sowjanya Kakarala, wrote: > > > > >> Hi Guys, >> >> I am new to aws, I am not sure how t

Re: Exploding arrays of structs

2018-05-01 Thread Furcy Pin
Hi Tom, High level manipulation with arrays of structural is generally where SQL reaches its limit in terms of expressivity and where Spark starts to get my preference. Unfortunately I don't know any public UDF that can do exactly what you want, but coding one is not that hard. If you use scala, s

ApacheCon North America 2018 schedule is now live.

2018-05-01 Thread Rich Bowen
Dear Apache Enthusiast, We are pleased to announce our schedule for ApacheCon North America 2018. ApacheCon will be held September 23-27 at the Montreal Marriott Chateau Champlain in Montreal, Canada. Registration is open! The early bird rate of $575 lasts until July 21, at which time it goe

Re: Hive External Table with Zero Bytes files

2018-05-01 Thread Mahender Sarangam
Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325 (Folder) and same has Filename. actually in our upstream, our source table is partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now when we create external table with partitioned by columns and fire