[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-864207085 Thanks @HeartSaVioR @xuanyuanking! I've updated the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-858953384 @HeartSaVioR @xuanyuanking Can we move forward with this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-854377685 Oh, not need to apologize. I've not updated this yet. :) This is a SQL config now. Please help review if you find some time. Thanks! -- This is an automated message from the Ap

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-854377685 Oh, not need to apologize. I've not updated this yet. :) This is a SQL config now. Please help review if you find some time. Thanks! -- This is an automated message from the Ap

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851802006 Okay, sounds good. Let me change to using a source option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851618733 @HeartSaVioR Does it sound okay for you? If okay, still prefer an option over config? If so, please let me know so I can change to use option. -- This is an automated message f

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851107724 > Which steps end users require to do to resolve such case with your PR? Deleting metadata directory and letting read path to ignore the metadata? Currently in the use-case

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851067116 > What's the solution of this? Doesn't it mean you want to make the directory be writable from multiple queries? The use-case looks like this. The user wants to write to sa

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851065018 > And one more, I think let file stream sink to ignore metadata directory on reading existing metadata but write to the metadata directory is odd and error-prone. The metadata is

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851061991 > This was already proposed before from a part of #31638, though I'm not sure you've indicated this. Oh, this comes from internal customer request. It seems hard to do wor

[GitHub] [spark] viirya commented on pull request #32702: [SPARK-35565][SS] Add config for ignoring metadata directory of FileStreamSink

viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851028415 cc @HeartSaVioR @xuanyuanking -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the