x27;t mean it will trigger Impala's catalogd to
pull in the latest metadata which is cached on catalogd.
This is probably not a Parquet related answers but more of the background how
Impala works with Hive, and how Spark updates data into Hive?
AL
Date: Sat, 2 Aug 2014 10:30:27 +0200
Subject
Hi Michael,
Thanks for your reply. Is this the correct way to load data from Spark
into Parquet? Somehow it doesn't feel right. When we followed the steps
described for storing the data into Hive tables everything was smooth, we
used HiveContext and the table is automatically recognised by Hive
So is the only issue that impala does not see changes until you refresh the
table? This sounds like a configuration that needs to be changed on the
impala side.
On Fri, Aug 1, 2014 at 7:20 AM, Patrick McGloin
wrote:
> Sorry, sent early, wasn't finished typing.
>
> CREATE EXTERNAL TABLE
>
Sorry, sent early, wasn't finished typing.
CREATE EXTERNAL TABLE
Then we can select the data using Impala. But this is registered as an
external table and must be refreshed if new data is inserted.
Obviously this doesn't seem good and doesn't seem like the correct solution.
How should we
Hi,
We would like to use Spark SQL to store data in Parquet format and then
query that data using Impala.
We've tried to come up with a solution and it is working but it doesn't
seem good. So I was wondering if you guys could tell us what is the
correct way to do this. We are using Spark 1.0 an