Apologies I missed your two points
My question:
#1 If there are 10 tables or more tables, do I need to read each table into
memory though Spark bases on memory compution?
Every table will be read as I described above. It is lazy read by Spark.
The computation happens when there is an action on
Well, Spark to BigQuery API is very efficient in doing what it needs to do.
Personally I have never found a JDBC connection to BigQuery that works
under all circumstances
.
In a typical environment you need to set-up your connection variable to
BigQuery from Spark.
These are my recommended ones
It's not possible to load everything into memory. We should use a big query
connector (should be existing already?) and register table B and C and temp
views in Spark.
On Fri, May 14, 2021 at 8:50 AM bo zhao wrote:
> Hi Team,
>
> I've followed Spark community for several years. This is my first
Hi Team,
I've followed Spark community for several years. This is my first time for
asking help. I hope you guys can give some experience.
I want to develop a spark application with processing a sqlscript file. The
data is on BigQuery.
For example, the sqlscript is:
delete from tableA;
insert
