Apologies I missed your two points
My question:
#1 If there are 10 tables or more tables, do I need to read each table into
memory though Spark bases on memory compution?
Every table will be read as I described above. It is lazy read by Spark.
The computation happens when there is an action on t
Well, Spark to BigQuery API is very efficient in doing what it needs to do.
Personally I have never found a JDBC connection to BigQuery that works
under all circumstances
.
In a typical environment you need to set-up your connection variable to
BigQuery from Spark.
These are my recommended ones
It's not possible to load everything into memory. We should use a big query
connector (should be existing already?) and register table B and C and temp
views in Spark.
On Fri, May 14, 2021 at 8:50 AM bo zhao wrote:
> Hi Team,
>
> I've followed Spark community for several years. This is my first
Hi Team,
I've followed Spark community for several years. This is my first time for
asking help. I hope you guys can give some experience.
I want to develop a spark application with processing a sqlscript file. The
data is on BigQuery.
For example, the sqlscript is:
delete from tableA;
insert in
