[
https://issues.apache.org/jira/browse/SPARK-27234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798677#comment-16798677
]
Jungtaek Lim commented on SPARK-27234:
--------------------------------------
Could you please provide reproducer for Apache vanilla version of Spark? Is it
still broken if you remove "makeReply" from the code?
If it's specific to "makeReply", you may have to contact Azure / Databricks
support.
> Continuous Streaming does not support python UDFs
> -------------------------------------------------
>
> Key: SPARK-27234
> URL: https://issues.apache.org/jira/browse/SPARK-27234
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Environment: Azure Databricks 5.1
> Reporter: Mark Hamilton
> Priority: Major
>
> Heres a repro
> get_p_eng = udf(lambda p: "foo")
> df = spark \
> .readStream \
> .format("rate") \
> .load()\
> .withColumn("p_eng", get_p_eng(col("value")))\
> .makeReply("p_eng")
> df.isStreaming # Returns True for DataFrames that have streaming sources
> df.printSchema
> # Read all the csv files written atomically in a directory
> df.writeStream.format("console").trigger(continuous="1 second").start()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
