[Announcement] Analytics Zoo 0.11.0 release

Hi Everyone, I’m happy to announce the 0.11.0 release of Analytics Zoo (distributed TensorFlow and PyTorch on Apache Spark & Ray); the highlights of this release include:

Re: Append to an existing Delta Lake using structured streaming

I will try to provide a stripped down example of what I am doing The initial delta lake is built from a dataframe like this from within a notebook val hourly_new = events.select(window('timestamp, "1 hour"), 'serial, 'type) .select($"window.start".as("start"), 'serialno, 'eventType)

Re: Append to an existing Delta Lake using structured streaming

Including the Delta Lake Users and Developers DL to help out. Saying this, could you clarify how data is not being added? By any chance do you have any code samples to recreate this? Sent via Superhuman On Wed, Jul 21, 2021 at 2:49 AM, wrote: >

This is a blog post explaining how to use a new Spark library, datafu-spark

https://medium.com/paypal-tech/introducing-datafu-spark-ba67faf1933a [https://miro.medium.com/max/1200/0*koSzBO7KqbmIpiPl] Introducing DataFu-Spark. DataFu-Spark is a new addition to… | by Eyal Allweil | Technology at PayPal

Append to an existing Delta Lake using structured streaming

Hi all,   I stumbled upon an interessting problem. I have an existing Deltalake with data recovered from a backup and would like to append to this Deltalake using Spark structured streaming. This does not work. Although the streaming job is running no data is appended. If I created the original