Re: Read Hive ACID tables in Spark or Pig

2019-03-06 Thread Alan Gates
Pig is in the same place as Spark, that the tables need to be compacted first. The issue is that outside readers don't understand which records in the delta files are valid and which are not. Theoretically all this is possible, as outside clients could get the valid transaction list from the meta

Read Hive ACID tables in Spark or Pig

2019-03-06 Thread Abhishek Gupta
Hi, Does Hive ACID tables for Hive version 1.2 posses the capability of being read into Apache Pig using HCatLoader or Spark using SQLContext. For Spark, it seems it is only possible to read ACID tables if the table is fully compacted i.e no delta folders exist in any partition. Details in the fol

4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las Vegas, and Berlin!

2019-03-06 Thread Rich Bowen
Dear Apache Enthusiast, (You’re receiving this because you are subscribed to one or more user mailing lists for an Apache Software Foundation project.) TL;DR: * Apache Roadshow DC is in 3 weeks. Register now at https://apachecon.com/usroadshowdc19/ * Registration for Apache Roadshow Chicago is