Hi Chris, Could you show the code you are using? When you mention "I like to use a static datasource (JDBC) in the state function" do you refer to a DataFrame from a JDBC source or an independent JDBC connection?
The key point to consider is that the flatMapGroupsWithState function must be serializable. Its execution happens in the workers of a Spark job. If you are using a JDBC connection, you need to make sure the connection is made in the context of the function. JDBC connections are not serializable. Likewise, Dataset/DataFrames only function in the driver where they are defined. They are bound to the Spark Session in the driver and it does not make sense to access them in a remote executor. Make sure you check the executor logs as well. There might be a NullPointerException lurking somewhere in your logs. met vriendelijke groeten, Gerard. PS: spark-dev (d...@spark.apache.org) is for discussions about open source development of the Spark project. For general questions like this, use the user's mailing list ( user@spark.apache.org) (note that I changed that address in the to: ) On Thu, Jul 19, 2018 at 12:51 PM Christiaan Ras < christiaan....@semmelwise.nl> wrote: > I use the state function flatmapgroupswithstate to track state of a kafka > stream. To further customize the state function I like to use a static > datasource (JDBC) in the state function. This datasource contains data I > like to join with the stream (as Iterator) within flatmapgroupswithstate. > > > > When I try to access the JDBC source within flatmapgroupswithstate Spark > execution freezes without any Exceptions or logging. > > To verify the JDBC connection works, I also tried to access the source > outside the state function and that works. So now I join the static source > with streaming source before feeding it to flatmapgroupswithstate. It seems > to work so far… > > > > Any ideas why accessing the JDBC source within flatmapgroupswithstate > could fail (freezes Spark execution)? Is it wise to use external > datasources within flatmapgroupswithstate? > > > > Thanks, > > Chris > > > > >
