Mark Hamilton created SPARK-27234: ------------------------------------- Summary: Continuous Streaming does not support python UDFs Key: SPARK-27234 URL: https://issues.apache.org/jira/browse/SPARK-27234 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.4.0 Environment: Azure Databricks 5.1 Reporter: Mark Hamilton
Heres a repro get_p_eng = udf(lambda p: "foo") df = spark \ .readStream \ .format("rate") \ .load()\ .withColumn("p_eng", get_p_eng(col("value")))\ .makeReply("p_eng") df.isStreaming # Returns True for DataFrames that have streaming sources df.printSchema # Read all the csv files written atomically in a directory df.writeStream.format("console").trigger(continuous="1 second").start() -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org