Yes, trigger (once=True) set to all streaming sources and it will treat as
a batch mode. Then you can use any scheduler (e.g airflow) to run it
whatever time window. With checkpointing, in the next run it will start
processing files from the last checkpoint.
On Fri, Apr 23, 2021 at 8:13 AM Mich
Hi,
Currently the user-facing Catalog API doesn't support backup/restore
metadata. Our customers are asking for such functionalities. Here is a
usage example:
1. Read all metadata of one Spark cluster
2. Save them into a Parquet file on DFS
3. Read the Parquet file and restore all metadata in
Hi Farhan,
I have used it successfully and it works. The only thing that potentially
can cause this issue is the jdbc driver itself. Have you tried another
jdbc driver like progress direct etc. Most of these defects are related to
jdbc driver itself!
HHT,
Mich
On Fri, 30 Apr 2021 at 13:49,
Hi Mich,
I have tried this already. I am using the same methods you are using in my
Java code. I see the same error, where 'dbtable' or 'query' gets added as a
connection property in the JDBC connection string for the source db, which
is AAS in my case.
Thanks,
Farhan.
On Fri, Apr 30, 2021
Right, yes it did not continue. It's not in Spark.
On Fri, Apr 30, 2021 at 7:07 AM jonnysettle
wrote:
> I remeber back in 2019 reading about Cypher language for graph queries been
> introduced to spark 3.X. But I don't see it in the latest version. Has
> the
> project been abandoned (issues
I remeber back in 2019 reading about Cypher language for graph queries been
introduced to spark 3.X. But I don't see it in the latest version. Has the
project been abandoned (issues 25994). I know there was a the Morpheus
project but that is not been maintained any more.
Have I missed some
Hi all,
I implemented a recursive UDF, that tries to find a document number in a long
list of predecessor documents. This can be a multi-level hierarchy:
C is successor of B is successor of A (but many more levels are possible)
As input to that UDF I prepare a dict that contains the complete
Hi,
If you are using Spark JDBC connection then you can do the following
generic JDBC from PySpark say. that tablename could be sql query as well
(select col1, col2 from )
## load from database
def loadTableFromJDBC(spark, url, tableName, user, password, driver,
fetchsize):
try:
Indeed adding public constructors solved the problem...
Thanks a lot!
> Am 29.04.2021 um 18:53 schrieb Rico Bergmann :
>
>
> It didn’t have it. So I added public no args and all args constructors. But I
> still get the same error
>
>
>
>>> Am 29.04.2021 um 17:47 schrieb Sean Owen :
