HiveThrift2 ACID Transactions?

2021-11-09 Thread Bode, Meikel, NMA-CFD
Hi all, We want to use apply INSERTS, UPDATE, and DELETE operations on tables based on parquet or ORC files served by thrift2. Actually its unclear whether we can enable them and where. At the moment, when executing UPDATE or DELETE operations those are getting blocked. Anyone out who uses

DataFrame.mapInArrow

2021-11-09 Thread Hyukjin Kwon
Hi dev, I proposed DataFrame.mapInArrow (https://github.com/apache/spark/pull/34505) which allows users to directly leverage Arrow batch to plug in other external systems easily. I would like to make sure this design of API covers most use cases, and would like to know if there is other feedback

Re: ASF board report draft for November

2021-11-09 Thread Mich Talebzadeh
Hi, Just a minor modification Under Description: Apache Spark is a fast and general engine for large-scale data processing. It should read Apache Spark is a fast and general purpose engine for large-scale data processing. HTH view my Linkedin profile

ASF board report draft for November

2021-11-09 Thread Matei Zaharia
Hi all, Our ASF board report needs to be submitted again this Wednesday (November 10). I wrote a draft with the major things that happened in the past three months — let me know if I missed something. === Description: Apache Spark is a fast and general engine for large-scale data