[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469749#comment-17469749 ] Fabian Paul commented on FLINK-25021: - Thanks for the detailed proposal. I have one last question. The Kafka connector you have posted first creates if I understand correctly first creates local files and only periodically submits the gzip of it. Do you want to do the same with the Flink connector? This approach sounds like a special case of our currently existing FileSink + FLIP-191 which is currently under development. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468066#comment-17468066 ] Xue Wang commented on FLINK-25021: -- [~fpaul] if things LGTY, then [Azure Data Explorer Kafka Connect Kusto Sink Connector|https://github.com/Azure/kafka-sink-azure-kusto] provides a good starting point or reference from my perspective. As I mentioned previously, I'm thinking of wrapping underlying synchronous ingestion with CompletableFuture as a temporary solution until ADX provides a real async API and implementation (or maybe I'll help them do so...). Looking forward to your reply :) > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468056#comment-17468056 ] Xue Wang commented on FLINK-25021: -- Hi [~fpaul], happy new year! One potential scenario would be to use Flink as a streaming ETL solution and then ingest results into ADX for final time-series or user behavior analytics, supporting both interactive queries and data visualizations at low latency. Some common scenarios as depicted in [ADX solutions|https://docs.microsoft.com/en-us/azure/data-explorer/solution-architectures] often suggest using ADX to directly consume events from message queues such as Kafka or Azure EventHub. However in actual development this is usually a sub-optimal solution due to various reasons. Furthermore, you may also find some docs describing other scenarios like [Azure Data Explorer and Stream Analytics for anomaly detection|https://azure.microsoft.com/en-us/blog/azure-data-explorer-and-stream-analytics-for-anomaly-detection/] and [When to use Azure Data Explorer or/and Azure Stream Analytics|https://docs.microsoft.com/en-us/azure/stream-analytics/azure-database-explorer-output#when-to-use-azure-data-explorer-orand-azure-stream-analytics] (Azure Stream Analytics is an Azure native offering for streaming processing). I personally often treat ADX as an close alternative to ElasticSearch, since feature-wise they are close enough (though by design they are essentially different in many ways). I hope this can also help you understand why I'd like to use Flink with ADX. I believe AWS should have similar offering via Kinesis, however I cannot comment on this since I'm not familiar with AWS . > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467961#comment-17467961 ] Fabian Paul commented on FLINK-25021: - [~xwang51] sorry for the late response. I just came back to work. Before we can discuss the technical details I would like to understand how a Source/Sink for Azure Data Explorer should look like from a Flink perspective. Can you elaborate more about your use case or the potential scenario? When looking through the documentation I had a hard time picturing using Flink with Azure Data Explorer. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463427#comment-17463427 ] Martijn Visser commented on FLINK-25021: [~xwang51] I have to defer to [~arvid] [~fpaul] for that, but I can imagine that the upcoming holidays might delay a response. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461169#comment-17461169 ] Xue Wang commented on FLINK-25021: -- Hi [~MartijnVisser], I checked the Kusto Java client in detail today and found they haven't implemented an Async client as in C#. What if I wrap the synchronous APIs with CompletableFuture like [this|https://github.com/Azure/azure-kusto-java/blob/b3d73b8bb1334c2565e551b7d1d2853d14f3fd13/samples/src/main/java/FileIngestionCompletableFuture.java#L77]? Any concerns? > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454912#comment-17454912 ] Xue Wang commented on FLINK-25021: -- [~MartijnVisser] No problem, Flink has a very active ML which should keep you busy. Really appreciate your guide here. I'll follow your suggestions and work on the async sink. Will let you know in this thread if I get more questions. Thanks! > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454860#comment-17454860 ] Martijn Visser commented on FLINK-25021: [~xwang51] Sorry for the late reply. 1. For Async sink, definitely use FLIP-171. 2. There's currently work being done on the Kinesis Data Streams sink. An example PR could be https://github.com/apache/flink/pull/17345. CC [~arvid] [~fpaul] in case they have other suggestions 3. We are currently working on the so called 'external connector repository' for Flink connectors indeed. I think its probably best to start from your own fork for now. We could also work together with you while we're setting up this repository to test if everything works as expected etc. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17453114#comment-17453114 ] Xue Wang commented on FLINK-25021: -- Hi [~MartijnVisser] , I took some time to go through these FLIPs you mentioned and some recordings of FF21. To start with, I'd like to implement a sink for ADX using [azure-kusto-java|[https://github.com/Azure/azure-kusto-java].] The SDK supports both sync and async ingest APIs. Currently, I'm planning to use async ingestion APIs. Here are some questions I have so far: # For async sink, what's the preferred API? SinkWriter ([FLIP-177|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API])] or AsyncSinkWriter ([FLIP-171|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink])] ? I see the AsyncSyncWriter is still marked as @PublicEvolving. # For SyncWriter, I'd like to reference Kafka and ElasticSearch implementations. For AsyncSinkWriter, I can only see an example in the FLIP. Would like to hear more from you on which implementations should I reference. # I learnt from FF21 that you're planning to have separate projects for connectors. So should I start this work in my own fork of Flink or should I start a new project instead? > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Assignee: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449223#comment-17449223 ] Martijn Visser commented on FLINK-25021: [~xwang51] Sounds good if you want to take on the issue! The most important part is that you use the new Source or Sink API (for the latter, you might also want to check out the ASync API). They are described in FLIPs: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink If you would like any example implementations, let me know. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448317#comment-17448317 ] Xue Wang commented on FLINK-25021: -- Thank you [~MartijnVisser] for the guidance. Glad to see these opportunities to contribute to the community and expand its ecosystem. If this looks good to you, shall I take on this issue? I'd love to also take the issues you mentioned once I finish this one. Also, before I start, is there any guidance or best practice I should be aware of (I committed only one bug fix before this :))? > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25021) Source/Sink for Azure Data Explorer (ADX)
[ https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448158#comment-17448158 ] Martijn Visser commented on FLINK-25021: Hi [~xwang51]! Thanks for opening this ticket. There are a couple of open tickets with regards to Azure sources/sinks, particularly https://issues.apache.org/jira/browse/FLINK-18568, https://issues.apache.org/jira/browse/FLINK-18567, https://issues.apache.org/jira/browse/FLINK-18566, https://issues.apache.org/jira/browse/FLINK-18565, https://issues.apache.org/jira/browse/FLINK-18564 and https://issues.apache.org/jira/browse/FLINK-18563. [~iekpo] can probably tell more about those. I don't think there's currently an ADX connector and we would welcome contributions for those. One small note, there's no such thing as an 'official connector'. Connectors in Flink are maintained by various open source contributors and the community, but there are also many connectors which aren't bundled with Flink itself (for example, https://github.com/ververica/flink-cdc-connectors) which are also commonly used. > Source/Sink for Azure Data Explorer (ADX) > - > > Key: FLINK-25021 > URL: https://issues.apache.org/jira/browse/FLINK-25021 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Xue Wang >Priority: Major > > Hi all, > I'm considering implementing source/sink for Azure Data Explorer (ADX). But > first I'd like to check with the community if this is already a supported > scenario. If not, will you consider adding it to the list of official > connectors? > FWIW, Azure Data Explorer is a widely used analytic service on Azure well > suited for ad-hoc and time series analysis over large volume of structured, > semi-structured, and unstructured data. And it can integrate with a wide > range of data sources and visualization tools. > References: > [General Intro by ADX > PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto] > [ADX > Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview] > [Ingest data using the Azure Data Explorer Java > SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data] -- This message was sent by Atlassian Jira (v8.20.1#820001)
