[jira] [Updated] (FLINK-25696) Introduce metadataConsumer to InitContext in Sink
[
https://issues.apache.org/jira/browse/FLINK-25696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-25696:
-
Summary: Introduce metadataConsumer to InitContext in Sink (was: Introduce
MetadataPublisher interface to SinkWriter)
> Introduce metadataConsumer to InitContext in Sink
> -
>
> Key: FLINK-25696
> URL: https://issues.apache.org/jira/browse/FLINK-25696
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream, Connectors / Kafka, Table Store
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> In Table Store, we want to get the offsets of kafka writer, only the offset
> returned by the callback inside the KafkaWriter is accurate, so we need this
> callback mechanism.
> This ticket wants to add a interface MetadataPublisher:
> {code:java}
> public interface MetadataPublisher {
> void subscribe(Consumer consumer);
> } {code}
> SinkWriter can implement this interface, so that table store can subscribe
> metadata from SinkWriter.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zjureel commented on pull request #18490: [FLINK-25794][sql-runtime] Clean cache after memory segments in it after they are released to MemoryManager
zjureel commented on pull request #18490: URL: https://github.com/apache/flink/pull/18490#issuecomment-1020749306 Hello @KarmaGYZ @JingsongLi , could you help to review this issue when you're free? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25794) Memory pages in LazyMemorySegmentPool should be clear after they are released to MemoryManager
[ https://issues.apache.org/jira/browse/FLINK-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25794: --- Labels: pull-request-available (was: ) > Memory pages in LazyMemorySegmentPool should be clear after they are released > to MemoryManager > -- > > Key: FLINK-25794 > URL: https://issues.apache.org/jira/browse/FLINK-25794 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Affects Versions: 1.11.6, 1.13.5, 1.14.3 >Reporter: Shammon >Priority: Major > Labels: pull-request-available > > `LazyMemorySegmentPool` manages memory segments cache for join, agg, sort and > etc. operators. These segments in the cache will be released to > `MemoryManager` after some specify operations such as join operator finishes > to build data in `LazyMemorySegmentPool.cleanCache` method. But these > segments are still in `LazyMemorySegmentPool.cachePages`, it may cause memory > fault if the `MemoryManager` has deallocated these segments -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] xintongsong commented on a change in pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
xintongsong commented on a change in pull request #18489:
URL: https://github.com/apache/flink/pull/18489#discussion_r791304149
##
File path:
flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystem.java
##
@@ -39,17 +45,67 @@
GSFileSystem(GoogleHadoopFileSystem googleHadoopFileSystem,
GSFileSystemOptions options) {
super(Preconditions.checkNotNull(googleHadoopFileSystem));
-LOGGER.info("Creating GSFileSystem with options {}", options);
-
this.options = Preconditions.checkNotNull(options);
+LOGGER.info("Creating GSFileSystem with options {}", options);
}
@Override
-public RecoverableWriter createRecoverableWriter() {
-LOGGER.info("Creating recoverable writer with options {}", options);
+public RecoverableWriter createRecoverableWriter() throws IOException {
+
+// follow the same rules as for the Hadoop connector, i.e.
+// 1) only use service credentials at all if Hadoop
+// "google.cloud.auth.service.account.enable" is true (default: true)
+// 2) use GOOGLE_APPLICATION_CREDENTIALS as location of credentials,
if supplied
+// 3) use Hadoop "google.cloud.auth.service.account.json.keyfile" as
location of
+// credentials, if supplied
+// 4) use no credentials
+
+// store any credentials we are to use, here
+Optional credentialsPath = Optional.empty();
+
+// only look for credentials if service account support is enabled
+Configuration hadoopConfig = getHadoopFileSystem().getConf();
+boolean enableServiceAccount =
+
hadoopConfig.getBoolean("google.cloud.auth.service.account.enable", true);
+if (enableServiceAccount) {
+
+// load google application credentials, and then fall back to
+// "google.cloud.auth.service.account.json.keyfile" from Hadoop
+credentialsPath =
Optional.ofNullable(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"));
+if (credentialsPath.isPresent()) {
+LOGGER.info(
+"Recoverable writer is using
GOOGLE_APPLICATION_CREDENTIALS at {}",
+credentialsPath.get());
+} else {
+credentialsPath =
+Optional.ofNullable(
+
hadoopConfig.get("google.cloud.auth.service.account.json.keyfile"));
+credentialsPath.ifPresent(
+path ->
+LOGGER.info(
+"Recoverable writer is using
credentials from Hadoop at {}",
+path));
+}
+}
-// create the Google storage service instance
-Storage storage = StorageOptions.getDefaultInstance().getService();
+// construct the storage instance, using credentials if provided
+Storage storage;
+if (credentialsPath.isPresent()) {
+LOGGER.info(
+"Creating GSRecoverableWriter using credentials from {}",
+credentialsPath.get());
+try (FileInputStream credentialsStream = new
FileInputStream(credentialsPath.get())) {
+GoogleCredentials credentials =
GoogleCredentials.fromStream(credentialsStream);
+storage =
+StorageOptions.newBuilder()
+.setCredentials(credentials)
+.build()
+.getService();
+}
+} else {
+LOGGER.info("Creating GSRecoverableWriter using no credentials");
+storage = StorageOptions.newBuilder().build().getService();
+}
Review comment:
I'd suggest to minimize things we do in the `if-else` branches as follow:
```
// construct the storage instance, using credentials if provided
StorageOptions.Builder storageOptionBuilder =
StorageOptions.newBuilder();
if (credentialsPath.isPresent()) {
LOGGER.info(
"Creating GSRecoverableWriter using credentials from {}",
credentialsPath.get());
try (FileInputStream credentialsStream = new
FileInputStream(credentialsPath.get())) {
GoogleCredentials credentials =
GoogleCredentials.fromStream(credentialsStream);
storageOptionBuilder.setCredentials(credentials);
}
} else {
LOGGER.info("Creating GSRecoverableWriter using no credentials");
}
// create the GS blob storage wrapper
GSBlobStorageImpl blobStorage =
new
GSBlobStorageImpl(storageOptionBuilder.build().getService());
```
##
File path:
flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystem.jav
[GitHub] [flink] flinkbot edited a comment on pull request #18432: [FLINK-25739][dist] Include Changelog to flink-dist jar
flinkbot edited a comment on pull request #18432: URL: https://github.com/apache/flink/pull/18432#issuecomment-1017970404 ## CI report: * eed34f330aff2dae3d4ea3429cbec0fa83fa4a34 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30073) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zjureel opened a new pull request #18490: [FLINK-25794][sql-runtime] Clean cache after memory segments in it after they are released to MemoryManager
zjureel opened a new pull request #18490: URL: https://github.com/apache/flink/pull/18490 ## What is the purpose of the change This pr aims to clean cache in `LazyMemorySegmentPool` after memory segments in it are released to `MemoryManager` ## Verifying this change This change added tests and can be verified as follows: - *Added test `ResettableExternalBufferTest.testSegmentPoolCleanCache` that validates the cache is cleared in `LazyMemorySegmentPool.cleanCache` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) no - The serializers: (yes / no / don't know) no - The runtime per-record code paths (performance sensitive): (yes / no / don't know) no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no - The S3 file system connector: (yes / no / don't know) no ## Documentation - Does this pull request introduce a new feature? (yes / no) no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24947) Support host network for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang closed FLINK-24947. - Resolution: Fixed > Support host network for native K8s integration > --- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Assignee: liuzhuo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] paul8263 commented on pull request #18460: [FLINK-25767][doc] Totally translated state.md into Chinese
paul8263 commented on pull request #18460: URL: https://github.com/apache/flink/pull/18460#issuecomment-1020748254 Hi @Myasuka , Thank you for valuable suggestion. I've corrected some translations and please perform an another review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24947) Support host network for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481502#comment-17481502 ] Yang Wang commented on FLINK-24947: --- Thanks [~spoon-lz] for working on this ticket and make it happen in the upcoming release-1.15. > Support host network for native K8s integration > --- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Assignee: liuzhuo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24947) Support host network for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481501#comment-17481501 ] Yang Wang commented on FLINK-24947: --- Fixed via: master: c7219f8827016d1067f008e2324c0194dad65cb1 36e98b33408cff77886ecd2280fa0e5f79033191 > Support host network for native K8s integration > --- > > Key: FLINK-24947 > URL: https://issues.apache.org/jira/browse/FLINK-24947 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: liuzhuo >Assignee: liuzhuo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > For the use of flink on k8s, for performance considerations, it is important > to choose a CNI plug-in. Usually we have two environments: Managed and > UnManaged. > Managed: Cloud vendors usually provide very efficient CNI plug-ins, we > don’t need to care about network performance issues > UnManaged: On self-built K8s clusters, CNI plug-ins are usually optional, > similar to Flannel and Calcico, but such software network cards usually lose > some performance or require some additional network strategies. > For an unmanaged environment, if we also want to achieve the best network > performance, should we support the *HostNetWork* model? > Use the host network to achieve the best performance -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wangyang0918 merged pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
wangyang0918 merged pull request #18119: URL: https://github.com/apache/flink/pull/18119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
wangyang0918 commented on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-1020746626 Merging this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25795) Support Pulsar sink connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ada Wong updated FLINK-25795: - Description: https://issues.apache.org/jira/browse/FLINK-20732 the PR of this ticket is reviewd, we could develop Python Pulsar sink. (was: [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=knaufk] ) > Support Pulsar sink connector in Python DataStream API. > --- > > Key: FLINK-25795 > URL: https://issues.apache.org/jira/browse/FLINK-25795 > Project: Flink > Issue Type: New Feature > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.3 >Reporter: Ada Wong >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-20732 the PR of this ticket is > reviewd, we could develop Python Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25795) Support Pulsar sink connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481499#comment-17481499 ] Ada Wong commented on FLINK-25795: -- [~dianfu] When the PR ([https://github.com/apache/flink/pull/17452)] merge master, I would like take this ticket. > Support Pulsar sink connector in Python DataStream API. > --- > > Key: FLINK-25795 > URL: https://issues.apache.org/jira/browse/FLINK-25795 > Project: Flink > Issue Type: New Feature > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.3 >Reporter: Ada Wong >Priority: Major > > [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=knaufk] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25795) Support Pulsar sink connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ada Wong updated FLINK-25795: - External issue ID: (was: 20726) > Support Pulsar sink connector in Python DataStream API. > --- > > Key: FLINK-25795 > URL: https://issues.apache.org/jira/browse/FLINK-25795 > Project: Flink > Issue Type: New Feature > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.3 >Reporter: Ada Wong >Priority: Major > > [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=knaufk] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25795) Support Pulsar sink connector in Python DataStream API.
Ada Wong created FLINK-25795: Summary: Support Pulsar sink connector in Python DataStream API. Key: FLINK-25795 URL: https://issues.apache.org/jira/browse/FLINK-25795 Project: Flink Issue Type: New Feature Components: API / Python, Connectors / Pulsar Affects Versions: 1.14.3 Reporter: Ada Wong [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=knaufk] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] JingsongLi edited a comment on pull request #18412: [FLINK-25696][datastream] Introduce MetadataPublisher interface to SinkWriter
JingsongLi edited a comment on pull request #18412: URL: https://github.com/apache/flink/pull/18412#issuecomment-1020739383 > From a Kafka connector side, the subscriber is not updated on every record, and just when the KafkaProducer is flushed it is only updated for a bulk of records (either during a checkpoint or if the internal buffer size is reached). I would definitely prefer to handle the consumer in the mailbox and not by the Kafka thread. The Kafka threads might have surprising effects on the overall pipeline stability i.e. the shutdown is blocked because the producer cannot be stopped because it is executing the metadata consumer. Thanks for the information. I share my concern here: - Performance problem: If we put every meta into mailbox (we don't know bulk in the callback), this performance will be very poor, you can look at `TaskMailboxImpl.put`, every data is locked will lead to very poor throughput. - Consistency issues: for example, the data flushed at the time of transaction commit, then re-stuff their meta into the mailbox, these data belong to the next checkpoint, no longer the current checkpoint, and TableStore wants the metas of the current checkpoint. - Block problem: I think from the protocol Callback similar interfaces should be executed quickly, we can add comments, you can look at `org.apache.kafka.clients.producer.Callback`, it is also required without very heavy logic. > Regarding adding a method to the InitContext I think that is okay. Do you think there will be ever multiple Subscribers? Maybe it is safer to already add a list instead of an optional. I think we can let the implementer assemble it himself, if there is a need for list. > I am still a bit surprised that the TableStoreSink reads all metadata offsets nevertheless if they are committed in Kafka or not. `TableStoreSink` will only read these offsets at preSnapshot time, which is used to synchronize full (file) and incremental (log) data, the records must be flushed at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #18412: [FLINK-25696][datastream] Introduce MetadataPublisher interface to SinkWriter
JingsongLi commented on pull request #18412: URL: https://github.com/apache/flink/pull/18412#issuecomment-1020739383 > From a Kafka connector side, the subscriber is not updated on every record, and just when the KafkaProducer is flushed it is only updated for a bulk of records (either during a checkpoint or if the internal buffer size is reached). I would definitely prefer to handle the consumer in the mailbox and not by the Kafka thread. The Kafka threads might have surprising effects on the overall pipeline stability i.e. the shutdown is blocked because the producer cannot be stopped because it is executing the metadata consumer. Thanks for the information. I share my concern here: - Performance problem: If we put every meta into mailbox, this performance will be very poor, you can look at `TaskMailboxImpl.put`, every data is locked will lead to very poor throughput. - Consistency issues: for example, the data flushed at the time of transaction commit, then re-stuff their meta into the mailbox, these data belong to the next checkpoint, no longer the current checkpoint, and TableStore wants the metas of the current checkpoint. - Block problem: I think from the protocol Callback similar interfaces should be executed quickly, we can add comments, you can look at `org.apache.kafka.clients.producer.Callback`, it is also required without very heavy logic. > Regarding adding a method to the InitContext I think that is okay. Do you think there will be ever multiple Subscribers? Maybe it is safer to already add a list instead of an optional. I think we can let the implementer assemble it himself, if there is a need for list. > I am still a bit surprised that the TableStoreSink reads all metadata offsets nevertheless if they are committed in Kafka or not. `TableStoreSink` will only read these offsets at preSnapshot time, which is used to synchronize full (file) and incremental (log) data, the records must be flushed at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18460: [FLINK-25767][doc] Totally translated state.md into Chinese
flinkbot edited a comment on pull request #18460: URL: https://github.com/apache/flink/pull/18460#issuecomment-1019844798 ## CI report: * 6385885a06a970cc31f450251e9f6b0d4150a0dd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30002) * 0a08d1b660690c6c98f73066e1a54bfff10c20be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30094) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25530) Support Pulsar source connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481493#comment-17481493 ] Ada Wong commented on FLINK-25530: -- [~syhily] Thanks, I create a pulsar python sink ticket. > Support Pulsar source connector in Python DataStream API. > - > > Key: FLINK-25530 > URL: https://issues.apache.org/jira/browse/FLINK-25530 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.2 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > > Flink have supported Pulsar source connector. > https://issues.apache.org/jira/browse/FLINK-20726 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18460: [FLINK-25767][doc] Totally translated state.md into Chinese
flinkbot edited a comment on pull request #18460: URL: https://github.com/apache/flink/pull/18460#issuecomment-1019844798 ## CI report: * 6385885a06a970cc31f450251e9f6b0d4150a0dd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30002) * 0a08d1b660690c6c98f73066e1a54bfff10c20be UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25794) Memory pages in LazyMemorySegmentPool should be clear after they are released to MemoryManager
Shammon created FLINK-25794: --- Summary: Memory pages in LazyMemorySegmentPool should be clear after they are released to MemoryManager Key: FLINK-25794 URL: https://issues.apache.org/jira/browse/FLINK-25794 Project: Flink Issue Type: Sub-task Components: Table SQL / Runtime Affects Versions: 1.14.3, 1.13.5, 1.11.6 Reporter: Shammon `LazyMemorySegmentPool` manages memory segments cache for join, agg, sort and etc. operators. These segments in the cache will be released to `MemoryManager` after some specify operations such as join operator finishes to build data in `LazyMemorySegmentPool.cleanCache` method. But these segments are still in `LazyMemorySegmentPool.cachePages`, it may cause memory fault if the `MemoryManager` has deallocated these segments -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] deadwind4 commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
deadwind4 commented on a change in pull request #18388: URL: https://github.com/apache/flink/pull/18388#discussion_r791298184 ## File path: tools/ci/java-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/JarFileChecker.java ## @@ -210,6 +210,11 @@ private static int findNonBinaryFilesContainingText( // dual-licensed under GPL 2 and EPL 2.0 // contained in sql-avro-confluent-registry .filter(path -> !pathStartsWith(path, "/org/glassfish/jersey/internal")) +// contained in sql-connector-pulsar +.filter( +path -> +!pathStartsWith( +path, "/org/apache/pulsar/shade/org/glassfish/jersey/")) Review comment: This code is to skip checking license. Only `org/glassfish/jersey/` I find needs to be skipped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18483: [FLINK-24041][connectors] Removed public setter for elementConverter in As…
flinkbot edited a comment on pull request #18483: URL: https://github.com/apache/flink/pull/18483#issuecomment-1020286441 ## CI report: * eec2f51db2ed6c4862959336f6edf6bdfc16b251 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30071) * 813bf185c6d59236e078f25146340c53d3d88b98 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30079) * ddf6b397383be3096f061ee5f292c6bf0b204fa6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30084) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18482: [FLINK-25744] Support native savepoints
flinkbot edited a comment on pull request #18482: URL: https://github.com/apache/flink/pull/18482#issuecomment-1020278176 ## CI report: * 91675405ae82d0e945f46caaa685d65763b296c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30070) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25770) Delete file is not correct in MergeTreeWriter
[ https://issues.apache.org/jira/browse/FLINK-25770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-25770. Resolution: Fixed master: eb9fdfb05cf68ed641124c6b5fe5164d7fcf927f > Delete file is not correct in MergeTreeWriter > - > > Key: FLINK-25770 > URL: https://issues.apache.org/jira/browse/FLINK-25770 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.1.0 > > > The deletion in MergeTreeWriter.updateCompactResult dose not consider upgrade > case, the upgrade file is required by previous snapshot and following > snapshot, we should ensure: > 1. This file is not the output of upgraded. > 2. This file is not the input of upgraded. > Otherwise, the file will be deleted incorrectly. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-table-store] JingsongLi merged pull request #13: [FLINK-25770] Delete file is not correct in MergeTreeWriter
JingsongLi merged pull request #13: URL: https://github.com/apache/flink-table-store/pull/13 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
deadwind4 commented on a change in pull request #18388: URL: https://github.com/apache/flink/pull/18388#discussion_r791295394 ## File path: flink-connectors/pom.xml ## @@ -105,6 +105,7 @@ under the License. flink-sql-connector-hive-3.1.2 flink-sql-connector-kafka flink-sql-connector-kinesis + flink-sql-connector-pulsar Review comment: The `flink-sql-connector-pulsar` is a uber jar is used by PyFlink. Inspired by `flink-connector-rabbitmq` and ` flink-sql-connector-rabbitmq`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18468: [FLINK-24905][connector/kinesis]Adding Kinesis data streams sql uber-jar and end to end test.
flinkbot edited a comment on pull request #18468: URL: https://github.com/apache/flink/pull/18468#issuecomment-1019969068 ## CI report: * 6ac66f07be10e8c320e77b89e41fc6ba17b1571e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30069) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
flinkbot edited a comment on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1015848205 ## CI report: * 3e1198a17fea03035bc621268a39ea3b386809a4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30030) * 1cc4394ed63adc3729cc13c2ef7aebebdf0d4e87 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30080) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
lindong28 commented on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1020716213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25790) Support authentication via core-site.xml in GCS FileSystem plugin
[ https://issues.apache.org/jira/browse/FLINK-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song reassigned FLINK-25790: Assignee: Galen Warren > Support authentication via core-site.xml in GCS FileSystem plugin > - > > Key: FLINK-25790 > URL: https://issues.apache.org/jira/browse/FLINK-25790 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.15.0 >Reporter: Galen Warren >Assignee: Galen Warren >Priority: Major > Labels: pull-request-available > > Add support for authentication via core-site.xml to the new GCS FileSystem > connector, recently added via [FLINK-11838] Create RecoverableWriter for GCS > - ASF JIRA (apache.org). > Specifically, make the RecoverableWriter use explicit credentials supplied in > core-site.xml in the "google.cloud.auth.service.account.json.keyfile" > property. Otherwise, it should use implicit credentials, as it already does. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18479: [FLINK-25387] Introduce ExecNodeMetadata
flinkbot edited a comment on pull request #18479: URL: https://github.com/apache/flink/pull/18479#issuecomment-1020172062 ## CI report: * 26c2d84f06d6605f38a20311466137b07152dac5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30067) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a change in pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
lindong28 commented on a change in pull request #18397:
URL: https://github.com/apache/flink/pull/18397#discussion_r791272806
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaValueOnlyDeserializerWrapper.java
##
@@ -36,10 +36,15 @@
/** A package private class to wrap {@link Deserializer}. */
class KafkaValueOnlyDeserializerWrapper implements
KafkaRecordDeserializationSchema {
+
private static final long serialVersionUID = 5409547407386004054L;
+
private static final Logger LOG =
LoggerFactory.getLogger(KafkaValueOnlyDeserializerWrapper.class);
+
private final Class> deserializerClass;
+// always be false since this Deserializer is only used for value.
Review comment:
Thank you Jing for digging into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a change in pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
lindong28 commented on a change in pull request #18397:
URL: https://github.com/apache/flink/pull/18397#discussion_r791270225
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaRecordSerializationSchemaBuilderTest.java
##
@@ -160,6 +165,37 @@ public void testSerializeRecordWithKey() {
assertArrayEquals(record.value(), serializationSchema.serialize("a"));
}
+@Test
+public void testNoConfigurableKafkaKeySerializer() throws Exception {
+final Map config = ImmutableMap.of("simpleKey",
"simpleValue");
+final KafkaRecordSerializationSchema schema =
+KafkaRecordSerializationSchema.builder()
+.setTopic(DEFAULT_TOPIC)
+// use StringSerializer as dummy Serializer, since
ValueSerializer is
Review comment:
Hmm.. It looks like all Javadoc (e.g. method, API) starts with first
letter being capital. But many non-Javadoc comments (e.g. comments in the
function body) starts with first letter being lower case. You are right that
this seems inconsistent...
I didn't find a rule for this on
https://flink.apache.org/contributing/code-style-and-quality-common.html. I was
primarily following Javadoc code style which almost always starts with an
upper-case letter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18481: [FLINK-25307][test] Change e2e nodename used to 127.0.0.1 and add debug information
flinkbot edited a comment on pull request #18481: URL: https://github.com/apache/flink/pull/18481#issuecomment-1020251442 ## CI report: * ce4954e2d80790b2bcbf2259cf994a5faeed50ff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30066) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 496911178b06d4b404ba74cfac0c90b4ca121dd6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30065) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18382: [FLINK-25524] Fix ChangelogStateBackend.notifyCheckpointComplete
flinkbot edited a comment on pull request #18382: URL: https://github.com/apache/flink/pull/18382#issuecomment-1014709963 ## CI report: * dee3a8cf9a282bc17924ccb9393d0592e4cd2011 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30062) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25577) Update GCS documentation
[ https://issues.apache.org/jira/browse/FLINK-25577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481473#comment-17481473 ] Galen Warren commented on FLINK-25577: -- Here's a second try at a release note: The `flink-gs-fs-hadoop` FileSystem plugin for Google Cloud Storage (GCS) has been introduced. This allows Flink to read data from and write data to GCS via paths with the `gs:://' scheme, and it provides similar functionality for GCS as, for example, the `flink-s3-fs-hadoop` provides for Amazon S3. In particular, this plugin supports the `RecoverableWriter` interface, which allows it to be used with file sinks. Under the hood, the `flink-gs-fs-hadoop` uses Google's `gcs-connector` Hadoop library for basic read/write operations, and it uses Google's `google-cloud-storage` library to implement `RecoverableWriter` functionality. > Update GCS documentation > > > Key: FLINK-25577 > URL: https://issues.apache.org/jira/browse/FLINK-25577 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem, Documentation >Reporter: David Morávek >Assignee: Galen Warren >Priority: Blocker > Labels: pull-request-available > Fix For: 1.15.0 > > > Update GCS documentation with respect to: > - flink-shaded-hadoop artifacts and hadoop version 2.8.3 that are no longer > supported as of 1.15 > - The recoverable writer new feature > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/gcs/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18430: [FLINK-25577][docs] Update GCS documentation
flinkbot edited a comment on pull request #18430: URL: https://github.com/apache/flink/pull/18430#issuecomment-1017872223 ## CI report: * 4e165800625c7265f2ddc075e70ce1aec445ad14 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29959) * 4d4b6ef9bf6c75d3dbbe3d64b58eb5ffcc218ebf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30092) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18430: [FLINK-25577][docs] Update GCS documentation
flinkbot edited a comment on pull request #18430: URL: https://github.com/apache/flink/pull/18430#issuecomment-1017872223 ## CI report: * 4e165800625c7265f2ddc075e70ce1aec445ad14 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29959) * 4d4b6ef9bf6c75d3dbbe3d64b58eb5ffcc218ebf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] galenwarren commented on pull request #18430: [FLINK-25577][docs] Update GCS documentation
galenwarren commented on pull request #18430: URL: https://github.com/apache/flink/pull/18430#issuecomment-1020682450 @xintongsong @MartijnVisser I also added back in the documentation of how to use `core-site.xml` config properties; the code for this is in the in-progress PR: https://github.com/apache/flink/pull/18489. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] galenwarren commented on a change in pull request #18430: [FLINK-25577][docs] Update GCS documentation
galenwarren commented on a change in pull request #18430:
URL: https://github.com/apache/flink/pull/18430#discussion_r791256263
##
File path: docs/content/docs/deployment/filesystems/gcs.md
##
@@ -50,57 +48,40 @@
env.getCheckpointConfig().setCheckpointStorage("gs:///");
```
-### Libraries
+Note that these examples are *not* exhaustive and you can use GCS in other
places as well, including your [high availability setup]({{< ref
"docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref
"docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that
Flink expects a FileSystem URI.
-You must include the following jars in Flink's `lib` directory to connect
Flink with gcs:
+### GCS File System plugin
-```xml
-
- org.apache.flink
- flink-shaded-hadoop2-uber
- ${flink.shared_hadoop_latest_version}
-
+Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
+This implementation is self-contained with no dependency footprint, so there
is no need to add Hadoop to the classpath to use it.
-
- com.google.cloud.bigdataoss
- gcs-connector
- hadoop2-2.2.0
-
-```
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage)
library to provide `RecoverableWriter` support.
-We have tested with `flink-shared-hadoop2-uber` version >= `2.8.5-1.8.3`.
-You can track the latest version of the [gcs-connector hadoop
2](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar).
+This file system supports the [StreamingFileSink]({{< ref
"docs/connectors/datastream/streamfile_sink" >}}) and the [FileSink]({{< ref
"docs/connectors/datastream/file_sink" >}}).
-### Authentication to access GCS
+To use `flink-gs-fs-hadoop`, copy the JAR file from the `opt` directory to the
`plugins` directory of your Flink distribution before starting Flink, i.e.
-Most operations on GCS require authentication. Please see [the documentation
on Google Cloud Storage
authentication](https://cloud.google.com/storage/docs/authentication) for more
information.
+```bash
+mkdir ./plugins/gs-fs-hadoop
+cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
+```
+
+### Configuration
-You can use the following method for authentication
-* Configure via core-site.xml
- You would need to add the following properties to `core-site.xml`
+The underlying Hadoop file system can be [configured using Hadoop's gs
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md)
by adding the configurations to your `flink-conf.yaml`.
- ```xml
-
-
- google.cloud.auth.service.account.enable
- true
-
-
- google.cloud.auth.service.account.json.keyfile
-
-
-
- ```
+For example, Hadoop has a `fs.gs.http.connect-timeout` configuration key. If
you want to change it, you need to set `gs.http.connect-timeout: xyz` in
`flink-conf.yaml`. Flink will internally translate this back to
`fs.gs.http.connect-timeout`. There is no need to pass configuration parameters
using Hadoop's XML configuration files.
- You would need to add the following to `flink-conf.yaml`
+`flink-gs-fs-hadoop` can also be configured by setting the following options
in `flink-conf.yaml`:
- ```yaml
- flinkConfiguration:
-fs.hdfs.hadoopconf:
- ```
+| Key | Description
|
+|---|---
[GitHub] [flink] galenwarren commented on a change in pull request #18430: [FLINK-25577][docs] Update GCS documentation
galenwarren commented on a change in pull request #18430:
URL: https://github.com/apache/flink/pull/18430#discussion_r791256107
##
File path: docs/content/docs/deployment/filesystems/gcs.md
##
@@ -50,57 +48,40 @@
env.getCheckpointConfig().setCheckpointStorage("gs:///");
```
-### Libraries
+Note that these examples are *not* exhaustive and you can use GCS in other
places as well, including your [high availability setup]({{< ref
"docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref
"docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that
Flink expects a FileSystem URI.
-You must include the following jars in Flink's `lib` directory to connect
Flink with gcs:
+### GCS File System plugin
-```xml
-
- org.apache.flink
- flink-shaded-hadoop2-uber
- ${flink.shared_hadoop_latest_version}
-
+Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
+This implementation is self-contained with no dependency footprint, so there
is no need to add Hadoop to the classpath to use it.
-
- com.google.cloud.bigdataoss
- gcs-connector
- hadoop2-2.2.0
-
-```
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage)
library to provide `RecoverableWriter` support.
-We have tested with `flink-shared-hadoop2-uber` version >= `2.8.5-1.8.3`.
-You can track the latest version of the [gcs-connector hadoop
2](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar).
+This file system supports the [StreamingFileSink]({{< ref
"docs/connectors/datastream/streamfile_sink" >}}) and the [FileSink]({{< ref
"docs/connectors/datastream/file_sink" >}}).
Review comment:
Updated in
https://github.com/apache/flink/pull/18430/commits/4d4b6ef9bf6c75d3dbbe3d64b58eb5ffcc218ebf.
##
File path: docs/content/docs/deployment/filesystems/gcs.md
##
@@ -50,57 +48,40 @@
env.getCheckpointConfig().setCheckpointStorage("gs:///");
```
-### Libraries
+Note that these examples are *not* exhaustive and you can use GCS in other
places as well, including your [high availability setup]({{< ref
"docs/deployment/ha/overview" >}}) or the [EmbeddedRocksDBStateBackend]({{< ref
"docs/ops/state/state_backends" >}}#the-rocksdbstatebackend); everywhere that
Flink expects a FileSystem URI.
-You must include the following jars in Flink's `lib` directory to connect
Flink with gcs:
+### GCS File System plugin
-```xml
-
- org.apache.flink
- flink-shaded-hadoop2-uber
- ${flink.shared_hadoop_latest_version}
-
+Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
+This implementation is self-contained with no dependency footprint, so there
is no need to add Hadoop to the classpath to use it.
-
- com.google.cloud.bigdataoss
- gcs-connector
- hadoop2-2.2.0
-
-```
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the
*gs://* scheme. It uses Google's
[gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector)
Hadoop library to access GCS. It also uses Google's
[google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage)
library to provide `RecoverableWriter` support.
-We have tested with `flink-shared-hadoop2-uber` version >= `2.8.5-1.8.3`.
-You can track the latest version of the [gcs-connector hadoop
2](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar).
+This file system supports the [StreamingFileSink]({{< ref
"docs/connectors/datastream/streamfile_sink" >}}) and the [FileSink]({{< ref
"docs/connectors/datastream/file_sink" >}}).
-### Authentication to access GCS
+To use `flink-gs-fs-hadoop`, copy the JAR file from the `opt` directory to the
`plugins` directory of your Flink distribution before starting Flink, i.e.
-Most operations on GCS require authentication. Please see [the documentation
on Google Cloud Storage
authentication](https://cloud.google.com/storage/docs/authentication) for more
information.
+```bash
+mkdir ./plugins/gs-fs-hadoop
+cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
+```
+
+### Configuration
-You can use the following method for authentication
-* Configure via core-site.xml
- You would need to add the following properties to `core-site.xml`
+The underlying Hadoop file system can be [configured using Hadoop's gs
configuration
keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md)
by adding the configurations to your `flink-conf.yaml`.
Review comment:
Yes, that's better. Good catch. Updated in
https://github.com/apache/flink/pull/18430/commits/4d4b6ef9bf6c75d3dbbe3d6
[GitHub] [flink] flinkbot edited a comment on pull request #18478: [FLINK-25787][flink-connector-gcp-pubsub] Adding endpoint support in Pubsub Source / Sink
flinkbot edited a comment on pull request #18478: URL: https://github.com/apache/flink/pull/18478#issuecomment-1020156897 ## CI report: * 7d269e64204469eb03c1b102f44a5930ceb45d62 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30054) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481468#comment-17481468
]
Yun Gao commented on FLINK-18356:
-
With some more observation, it seems (sorry I still have not got the final
result yet)
# Currently the test mostly failed in flink-table/flink-table-planner module.
The tests of this module contains two parts, the tests and the integration
tests. The failure always happens in the integration tests parts.
# In Azure there are two parallel surefire test processes. Since the
flink-table-planner module has set reuseForks = true, it means the same two
processes would be used to run all the integration tests. Thus if we do not
have correctly cleanup or some cases have memory leaking, the memory used would
keep increasing.
# By add some print statements to the watchdog process:
[https://github.com/apache/flink/pull/18486,] from the result
[https://dev.azure.com/gaoyunhaii/gaoyun-flink/_build/results?buildId=562&view=logs&j=43a593e7-535d-554b-08cc-244368da36b4&t=82d122c0-8bbf-56f3-4c0d-8e3d69630d0f]
it seems the total memory is indeed 7G as Dawid pointed out, and the memory
usage is keeping increasing.
# Fortunately the case could be reproduced locally: by first run _mvn clean
install_ then run {_}mvn -Dflink.forkCount=2 -Dcheckstyle.skip=true verify -pl
flink-table/flink-table-planner{_}, the memory of the two processes are keeping
increasing, the maximum memory required is 4G for each process. This is also
wired since we have limit the heap to 2G. By adding
-XX:NativeMemoryTracking=detail to the surefire plugin JVM options, the memory
tracking result at the end of the tests are as follows.
{code:java}
Native Memory Tracking:Total: reserved=5199583KB +38339KB, committed=3802831KB
+44371KB- Java Heap (reserved=2097152KB, committed=1575936KB)
(mmap: reserved=2097152KB, committed=1575936KB)
- Class (reserved=2342856KB +37546KB, committed=1534368KB
+42666KB)
(classes #193700 +5400)
(malloc=38856KB +682KB #351017 +7653)
(mmap: reserved=2304000KB +36864KB,
committed=1495512KB +41984KB)
- Thread (reserved=48453KB -969KB, committed=48453KB -969KB)
(thread #48 -1)
(stack: reserved=48188KB -1028KB, committed=48188KB
-1028KB)
(malloc=146KB -3KB #250 -5)
(arena=119KB +63 #90 -2)
- Code (reserved=287969KB +111KB, committed=244357KB
+1023KB)
(malloc=38369KB +111KB #69847 +655)
(mmap: reserved=249600KB, committed=205988KB +912KB)
- GC (reserved=146916KB +24KB, committed=127580KB +24KB)
(malloc=36324KB +24KB #148561 +842)
(mmap: reserved=110592KB, committed=91256KB)
- Compiler (reserved=442KB, committed=442KB)
(malloc=312KB #8705 +2)
(arena=131KB #7)
- Internal (reserved=177788KB +890KB, committed=177784KB
+890KB)
(malloc=177752KB +890KB #316112 +8088)
(mmap: reserved=36KB, committed=32KB)
- Symbol (reserved=43952KB +93KB, committed=43952KB +93KB)
(malloc=42122KB +93KB #393043 +1171)
(arena=1830KB #1)
- Native Memory Tracking (reserved=21378KB +675KB, committed=21378KB +675KB)
(malloc=1011KB +314KB #14713 +4635)
(tracking overhead=20367KB +361KB)
- Arena Chunk (reserved=28582KB -31KB, committed=28582KB -31KB)
(malloc=28582KB -31KB)
- Unknown (reserved=4096KB, committed=0KB)
(mmap: reserved=4096KB, committed=0KB)
{code}
It seems the heap part and the classes part contributes to most of the memory
consumption.
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-c
[GitHub] [flink] flinkbot edited a comment on pull request #18368: [FLINK-18356][Azure] Enforce memory limits for docker containers
flinkbot edited a comment on pull request #18368: URL: https://github.com/apache/flink/pull/18368#issuecomment-1013194055 ## CI report: * 74a00fd69ea3321fded40d1b6bafae149523ed75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30060) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
flinkbot edited a comment on pull request #18324: URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905 ## CI report: * 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18480: [FLINK-25789][docs-zh] Translate the formats/hadoop page into Chinese.
flinkbot edited a comment on pull request #18480: URL: https://github.com/apache/flink/pull/18480#issuecomment-1020209467 ## CI report: * f7553fcadb4a768cbfb9742704714280961d0569 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30061) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18475: [FLINK-25728] Protential memory leaks in StreamMultipleInputProcessor
flinkbot edited a comment on pull request #18475: URL: https://github.com/apache/flink/pull/18475#issuecomment-1020104543 ## CI report: * 3ade9719b565beeacf7761ab71d5abb7ba62e62a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30045) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18472: [FLINK-25783][docs-zh] Translate azure_table_storage.md page into Chinese.
flinkbot edited a comment on pull request #18472: URL: https://github.com/apache/flink/pull/18472#issuecomment-1020057651 ## CI report: * f2a102b8715567da4c779398863dc3ee9531fd92 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30063) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18306: [FLINK-25445] No need to create local recovery dirs when disabled loc…
flinkbot edited a comment on pull request #18306: URL: https://github.com/apache/flink/pull/18306#issuecomment-1008234006 ## CI report: * f8c4b9cfcf8dd3fc85472dac9b8165c7c2c846dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30056) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18479: [FLINK-25387] Introduce ExecNodeMetadata
flinkbot edited a comment on pull request #18479: URL: https://github.com/apache/flink/pull/18479#issuecomment-1020172062 ## CI report: * d2b8689dd7de1135b312bdc9cf99a57945c07ace Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30055) * 26c2d84f06d6605f38a20311466137b07152dac5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30067) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
flinkbot edited a comment on pull request #18489: URL: https://github.com/apache/flink/pull/18489#issuecomment-1020643947 ## CI report: * 6d23f67c6515ebfeef972314b5c19bb02fc3d2ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30088) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #18189: [FLINK-25430] Replace RunningJobRegistry by JobResultStore
dmvk commented on a change in pull request #18189:
URL: https://github.com/apache/flink/pull/18189#discussion_r791168425
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/MiniDispatcher.java
##
@@ -55,14 +57,16 @@ public MiniDispatcher(
RpcService rpcService,
DispatcherId fencingToken,
DispatcherServices dispatcherServices,
-JobGraph jobGraph,
+@Nullable JobGraph jobGraph,
+@Nullable JobResult recoveredDirtyJob,
DispatcherBootstrapFactory dispatcherBootstrapFactory,
JobClusterEntrypoint.ExecutionMode executionMode)
throws Exception {
super(
rpcService,
fencingToken,
-Collections.singleton(jobGraph),
+CollectionUtil.fromNullable(jobGraph),
Review comment:
```suggestion
CollectionUtil.ofNullable(jobGraph),
```
to be consistent with other java apis (eg. `Optional.ofNullable`)
##
File path:
flink-clients/src/test/java/org/apache/flink/client/testjar/ErrorHandlingSubmissionJob.java
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.testjar;
+
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.io.DiscardingOutputFormat;
+import org.apache.flink.client.cli.CliFrontendTestUtils;
+import org.apache.flink.client.program.PackagedProgram;
+import org.apache.flink.client.program.ProgramInvocationException;
+import org.apache.flink.util.FlinkException;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.net.MalformedURLException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * {@code ErrorHandlingSubmissionJob} provides a factory method for creating a
{@link
+ * PackagedProgram} that monitors the job submission within the job's {@code
main} method.
+ */
+public class ErrorHandlingSubmissionJob {
+
+private static final AtomicReference SUBMISSION_EXCEPTION = new
AtomicReference<>();
+
+/**
+ * {@code ExpectedFailingInMainException} is the failure being thrown when
executing the {@code
+ * main} method of this {@code ErrorHandlingSubmissionJob}.
+ */
+public static class ExpectedFailingInMainException extends Exception {
Review comment:
unused
##
File path:
flink-clients/src/test/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrapITCase.java
##
@@ -147,6 +157,67 @@ public void
testDispatcherRecoversAfterLosingAndRegainingLeadership() throws Exc
}
}
+@Test
+public void testDirtyJobResultRecoveryInApplicationMode() throws Exception
{
+final Deadline deadline = Deadline.fromNow(TIMEOUT);
+final Configuration configuration = new Configuration();
+configuration.set(HighAvailabilityOptions.HA_MODE,
HighAvailabilityMode.ZOOKEEPER.name());
+configuration.set(DeploymentOptions.TARGET, EmbeddedExecutor.NAME);
+configuration.set(ClientOptions.CLIENT_RETRY_PERIOD,
Duration.ofMillis(100));
+final TestingMiniClusterConfiguration clusterConfiguration =
+TestingMiniClusterConfiguration.newBuilder()
+.setConfiguration(configuration)
+.build();
+
+// having a dirty entry in the JobResultStore should make the
ApplicationDispatcherBootstrap
+// implementation fail to submit the job
+final JobResultStore jobResultStore = new EmbeddedJobResultStore();
+jobResultStore.createDirtyResult(
+new JobResultEntry(
+new JobResult.Builder()
+
.jobId(ApplicationDispatcherBootstrap.ZERO_JOB_ID)
+.applicationStatus(ApplicationStatus.SUCCEEDED)
+.netRuntime(1)
+.build()));
+final EmbeddedHaServicesWithLeadershipControl haServices =
+new
EmbeddedHaServicesWithLeadershipControl(TestingUti
[GitHub] [flink] flinkbot commented on pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
flinkbot commented on pull request #18489: URL: https://github.com/apache/flink/pull/18489#issuecomment-1020643947 ## CI report: * 6d23f67c6515ebfeef972314b5c19bb02fc3d2ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
flinkbot commented on pull request #18489: URL: https://github.com/apache/flink/pull/18489#issuecomment-1020643017 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d23f67c6515ebfeef972314b5c19bb02fc3d2ed (Mon Jan 24 23:11:42 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25790).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25290) Add table source and sink test suite in connector testing framework
[ https://issues.apache.org/jira/browse/FLINK-25290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481454#comment-17481454 ] Yufan Sheng commented on FLINK-25290: - I'm working on the sink test. When would this feature be ready? > Add table source and sink test suite in connector testing framework > --- > > Key: FLINK-25290 > URL: https://issues.apache.org/jira/browse/FLINK-25290 > Project: Flink > Issue Type: Sub-task >Reporter: Qingsheng Ren >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] galenwarren commented on pull request #15599: [FLINK-11838][flink-gs-fs-hadoop] Create Google Storage file system with recoverable writer support
galenwarren commented on pull request #15599: URL: https://github.com/apache/flink/pull/15599#issuecomment-1020640696 PR created: https://github.com/apache/flink/pull/18489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25790) Support authentication via core-site.xml in GCS FileSystem plugin
[ https://issues.apache.org/jira/browse/FLINK-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481453#comment-17481453 ] Galen Warren commented on FLINK-25790: -- Created PR: https://github.com/apache/flink/pull/18489 > Support authentication via core-site.xml in GCS FileSystem plugin > - > > Key: FLINK-25790 > URL: https://issues.apache.org/jira/browse/FLINK-25790 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.15.0 >Reporter: Galen Warren >Priority: Major > Labels: pull-request-available > > Add support for authentication via core-site.xml to the new GCS FileSystem > connector, recently added via [FLINK-11838] Create RecoverableWriter for GCS > - ASF JIRA (apache.org). > Specifically, make the RecoverableWriter use explicit credentials supplied in > core-site.xml in the "google.cloud.auth.service.account.json.keyfile" > property. Otherwise, it should use implicit credentials, as it already does. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25790) Support authentication via core-site.xml in GCS FileSystem plugin
[ https://issues.apache.org/jira/browse/FLINK-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25790: --- Labels: pull-request-available (was: ) > Support authentication via core-site.xml in GCS FileSystem plugin > - > > Key: FLINK-25790 > URL: https://issues.apache.org/jira/browse/FLINK-25790 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.15.0 >Reporter: Galen Warren >Priority: Major > Labels: pull-request-available > > Add support for authentication via core-site.xml to the new GCS FileSystem > connector, recently added via [FLINK-11838] Create RecoverableWriter for GCS > - ASF JIRA (apache.org). > Specifically, make the RecoverableWriter use explicit credentials supplied in > core-site.xml in the "google.cloud.auth.service.account.json.keyfile" > property. Otherwise, it should use implicit credentials, as it already does. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] galenwarren opened a new pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
galenwarren opened a new pull request #18489: URL: https://github.com/apache/flink/pull/18489 ## What is the purpose of the change For the GCS FileSystem plugin, use the same authentication options for the RecoverableWriter portion as is done for the normal FileSystem portion. This means that it will use GOOGLE_APPLICATION_CREDENTIALS, if it exists, but will also use the `google.cloud.auth.service.account.json.keyfile` property from Hadoop config. To have both portions of the plugin use the same rules, each of them will only consider using service credentials if the Hadoop property `google.cloud.auth.service.account.enable` is `true` or unspecified (i.e. the default value is `true`). ## Brief change log - Update `GSFileSystemFactory` to read Hadoop config from the location specified in `CoreOptions.FLINK_HADOOP_CONF_DIR` or in the `HADOOP_CONF_DIR` environment variable and to combine it with Hadoop config values from the Flink config - Update `GSFileSystem` to look for credentials in either `GOOGLE_APPLICATION_CREDENTIALS` or `google.cloud.auth.service.account.json.keyfile`, if `google.cloud.auth.service.account.enable` is not false, when constructing the `Storage` instance for the `RecoverableWriter` - ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) No - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) No - The serializers: (yes / no / don't know) No - The runtime per-record code paths (performance sensitive): (yes / no / don't know) No - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) No - The S3 file system connector: (yes / no / don't know) No ## Documentation - Does this pull request introduce a new feature? (yes / no) Yes - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) Will be documented via https://github.com/apache/flink/pull/18430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18476: [FLINK-25726][streaming] Add Global Committer as post commit topology
flinkbot edited a comment on pull request #18476: URL: https://github.com/apache/flink/pull/18476#issuecomment-1020132926 ## CI report: * 9c24ac788a7b5c48ce690a43212bcd25d68cf667 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30053) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
flinkbot edited a comment on pull request #18431: URL: https://github.com/apache/flink/pull/18431#issuecomment-1017952324 ## CI report: * 939c7760b4809ef0e2d1ea897aa8d1217913669d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30058) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18473: [FLINK-25781][network] Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle
flinkbot edited a comment on pull request #18473: URL: https://github.com/apache/flink/pull/18473#issuecomment-1020072553 ## CI report: * 4011a25da659ed1cfa258705a683ca83e1957c28 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30052) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18432: [FLINK-25739][dist] Include Changelog to flink-dist jar
flinkbot edited a comment on pull request #18432: URL: https://github.com/apache/flink/pull/18432#issuecomment-1017970404 ## CI report: * b14ababb8a5d1433710d69f2184b475c335dac2d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30051) * eed34f330aff2dae3d4ea3429cbec0fa83fa4a34 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30073) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * c8820e335d23677ffb548b0d1fa8771fa7c1bce1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29965) * bf9ffe09e2eec6f03c9c1c5f18cf95b323088549 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30075) * bbf72ae73592bd646c44296a59ef70d3eb614701 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30086) * 59f46a383a5cca968faf82f8c260217649a6f825 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18392: [FLINK-25590][metrics] Introduce RequestedMemoryUsage and log warnings if usage exceeds 100%
flinkbot edited a comment on pull request #18392: URL: https://github.com/apache/flink/pull/18392#issuecomment-1015285195 ## CI report: * 8fcd75bc3466edd79726a95f62aa923c31796b2e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30050) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * c8820e335d23677ffb548b0d1fa8771fa7c1bce1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29965) * bf9ffe09e2eec6f03c9c1c5f18cf95b323088549 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30075) * bbf72ae73592bd646c44296a59ef70d3eb614701 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30086) * 59f46a383a5cca968faf82f8c260217649a6f825 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25277) Introduce explicit shutdown signalling between TaskManager and JobManager
[ https://issues.apache.org/jira/browse/FLINK-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481432#comment-17481432 ] Niklas Semmler commented on FLINK-25277: [~chesnay] Yes, you are right. [~trohrmann] needed the shutdown hook for a different use case, so he included the code already in dd6069fabf8a7ff65fbd9ff8dd7b0c47f492288f. When I saw this, I removed it from the commits above to avoid merge conflicts. Also, I just want to stress, the shutdown code was really just the icing on the cake. All the signaling functionality was already implemented, but was just not called during shutdown. > Introduce explicit shutdown signalling between TaskManager and JobManager > -- > > Key: FLINK-25277 > URL: https://issues.apache.org/jira/browse/FLINK-25277 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.13.0, 1.14.0 >Reporter: Niklas Semmler >Assignee: Niklas Semmler >Priority: Major > Labels: pull-request-available, reactive > Fix For: 1.15.0 > > Original Estimate: 504h > Remaining Estimate: 504h > > We need to introduce shutdown signalling between TaskManager and JobManager > for fast & graceful shutdown in reactive scheduler mode. > In Flink 1.14 and earlier versions, the JobManager tracks the availability of > a TaskManager using a hearbeat. This heartbeat is by default configured with > an interval of 10 seconds and a timeout of 50 seconds [1]. Hence, the > shutdown of a TaskManager is recognized only after about 50-60 seconds. This > works fine for the static scheduling mode, where a TaskManager only > disappears as part of a cluster shutdown or a job failure. However, in the > reactive scheduler mode (FLINK-10407), TaskManagers are regularly added and > removed from a running job. Here, the heartbeat-mechanisms incurs additional > delays. > To remove these delays, we add an explicit shutdown signal from the > TaskManager to the JobManager. > > [1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#heartbeat-timeout -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] syhily commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
syhily commented on a change in pull request #18388: URL: https://github.com/apache/flink/pull/18388#discussion_r791185396 ## File path: tools/ci/java-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/JarFileChecker.java ## @@ -210,6 +210,11 @@ private static int findNonBinaryFilesContainingText( // dual-licensed under GPL 2 and EPL 2.0 // contained in sql-avro-confluent-registry .filter(path -> !pathStartsWith(path, "/org/glassfish/jersey/internal")) +// contained in sql-connector-pulsar +.filter( +path -> +!pathStartsWith( +path, "/org/apache/pulsar/shade/org/glassfish/jersey/")) Review comment: Pulsar client shades a lot of jars, such as the `avro`. Can we just change this check filter to `/org/apache/pulsar/shade`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] syhily commented on a change in pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
syhily commented on a change in pull request #18388: URL: https://github.com/apache/flink/pull/18388#discussion_r791184358 ## File path: flink-connectors/pom.xml ## @@ -105,6 +105,7 @@ under the License. flink-sql-connector-hive-3.1.2 flink-sql-connector-kafka flink-sql-connector-kinesis + flink-sql-connector-pulsar Review comment: Pulsar connector doesn't support the Flink SQL now. Add this module may cause confusion for the end-user. Can we just use `flink-connector-pulsar`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18382: [FLINK-25524] Fix ChangelogStateBackend.notifyCheckpointComplete
flinkbot edited a comment on pull request #18382: URL: https://github.com/apache/flink/pull/18382#issuecomment-1014709963 ## CI report: * 06969c07178c598b4bd39fbf88385b1b0638e3a6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30048) * dee3a8cf9a282bc17924ccb9393d0592e4cd2011 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30062) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25530) Support Pulsar source connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481422#comment-17481422 ] Yufan Sheng commented on FLINK-25530: - https://github.com/apache/flink/pull/17452 The code for Pulsar sink is ready to be reviewed. I think we can port this to python too. > Support Pulsar source connector in Python DataStream API. > - > > Key: FLINK-25530 > URL: https://issues.apache.org/jira/browse/FLINK-25530 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.2 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > > Flink have supported Pulsar source connector. > https://issues.apache.org/jira/browse/FLINK-20726 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18471: [FLINK-25780][network] Reduce the maximum size of data output buffers per result partition for sort-shuffle
flinkbot edited a comment on pull request #18471: URL: https://github.com/apache/flink/pull/18471#issuecomment-1020052968 ## CI report: * 65c779eea512e90b8ec20eec4bbe56716f1b373e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30047) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24246) Bump Pulsar to 2.9.1
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-24246: Summary: Bump Pulsar to 2.9.1 (was: Bump Pulsar to 2.9.0) > Bump Pulsar to 2.9.1 > > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > Pulsar 2.8.1 has been released, the hack for getting TxnId from Pulsar > Transaction could be removed after bump flink-connector-pulsar's > pulsar-client-all to 2.8.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24246) Bump Pulsar to 2.9.0
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481413#comment-17481413 ] Yufan Sheng commented on FLINK-24246: - Yeah, Pulsar 2.9.1 has been released. I would cherry-pick this from the [sink PR|https://github.com/apache/flink/pull/17452]. > Bump Pulsar to 2.9.0 > > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > Pulsar 2.8.1 has been released, the hack for getting TxnId from Pulsar > Transaction could be removed after bump flink-connector-pulsar's > pulsar-client-all to 2.8.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25530) Support Pulsar source connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481412#comment-17481412 ] Yufan Sheng commented on FLINK-25530: - [~ana4] I'm quite glad to see this issue. If you have any questions on this implementation, just PING me. > Support Pulsar source connector in Python DataStream API. > - > > Key: FLINK-25530 > URL: https://issues.apache.org/jira/browse/FLINK-25530 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.2 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > > Flink have supported Pulsar source connector. > https://issues.apache.org/jira/browse/FLINK-20726 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17498: [FLINK-14954][rest] Add OpenAPI spec generator
flinkbot edited a comment on pull request #17498: URL: https://github.com/apache/flink/pull/17498#issuecomment-944209637 ## CI report: * 1d7656b27b98700dc8ed4bad5a7bf4bed00680a6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30046) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * c8820e335d23677ffb548b0d1fa8771fa7c1bce1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29965) * bf9ffe09e2eec6f03c9c1c5f18cf95b323088549 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30075) * bbf72ae73592bd646c44296a59ef70d3eb614701 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30086) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25753) KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481394#comment-17481394
]
Roman Khachatryan commented on FLINK-25753:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30040&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=35167
> KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP
> -
>
> Key: FLINK-25753
> URL: https://issues.apache.org/jira/browse/FLINK-25753
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
>
> The test {{KafkaTableITCase.testStartFromGroupOffsetsLatest}} fails on AZP
> with
> {code}
> 2022-01-21T08:48:26.7044016Z Jan 21 08:48:26 [ERROR]
> KafkaTableITCase.testStartFromGroupOffsetsLatest Time elapsed: 5.308 s <<<
> ERROR!
> 2022-01-21T08:48:26.7044630Z Jan 21 08:48:26
> java.util.concurrent.TimeoutException: Can not get the expected result.
> 2022-01-21T08:48:26.7047268Z Jan 21 08:48:26 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:214)
> 2022-01-21T08:48:26.7048108Z Jan 21 08:48:26 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:230)
> 2022-01-21T08:48:26.7048836Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.waitingExpectedResults(KafkaTableTestUtils.java:82)
> 2022-01-21T08:48:26.7049632Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsets(KafkaTableITCase.java:942)
> 2022-01-21T08:48:26.7050427Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsetsLatest(KafkaTableITCase.java:831)
> 2022-01-21T08:48:26.7051077Z Jan 21 08:48:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-01-21T08:48:26.7051638Z Jan 21 08:48:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-01-21T08:48:26.7052284Z Jan 21 08:48:26 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-01-21T08:48:26.7053008Z Jan 21 08:48:26 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-01-21T08:48:26.7053681Z Jan 21 08:48:26 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-01-21T08:48:26.7054328Z Jan 21 08:48:26 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-01-21T08:48:26.7054976Z Jan 21 08:48:26 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-01-21T08:48:26.7055621Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-01-21T08:48:26.7056251Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-01-21T08:48:26.7056904Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-01-21T08:48:26.7057520Z Jan 21 08:48:26 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-01-21T08:48:26.7058445Z Jan 21 08:48:26 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-01-21T08:48:26.7059155Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-01-21T08:48:26.7059779Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-01-21T08:48:26.7060382Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-01-21T08:48:26.7061036Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-01-21T08:48:26.7061670Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-01-21T08:48:26.7062369Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-01-21T08:48:26.7063456Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-01-21T08:48:26.7064218Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-01-21T08:48:26.7064805Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-01-21T08:48:26.7065375Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-01-21T08:48:26.7065919Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-01-21T08:48:26.7066
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * c8820e335d23677ffb548b0d1fa8771fa7c1bce1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29965) * bf9ffe09e2eec6f03c9c1c5f18cf95b323088549 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30075) * bbf72ae73592bd646c44296a59ef70d3eb614701 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18474: [FLINK-25786][network] Adjust the generation of subpartition data storage order for sort-shuffle from random shuffle to random shift
flinkbot edited a comment on pull request #18474: URL: https://github.com/apache/flink/pull/18474#issuecomment-1020097932 ## CI report: * 0b92227b39fabde5919ab161e12a6f7e272e70f7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30044) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18473: [FLINK-25781][network] Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle
flinkbot edited a comment on pull request #18473: URL: https://github.com/apache/flink/pull/18473#issuecomment-1020072553 ## CI report: * d2d1c3c17d01e1a98e29568cfaba43460726092a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30041) * 4011a25da659ed1cfa258705a683ca83e1957c28 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30052) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
flinkbot edited a comment on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1014266348 ## CI report: * 320757eaf30cae2a2b95c06e559c37e068120ef4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30042) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24229) [FLIP-171] DynamoDB implementation of Async Sink
[
https://issues.apache.org/jira/browse/FLINK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481375#comment-17481375
]
Yuri Gusev commented on FLINK-24229:
Hi [~MartijnVisser] [~CrynetLogistics] ,
I'm sorry for the delay with this. Implementation is ready, we've just finished
integration tests as well. Doing last refinements to add javadoc and rebase and
will open PR hopefully tomorrow.
Hope we are still in time to make it into release.
> [FLIP-171] DynamoDB implementation of Async Sink
>
>
> Key: FLINK-24229
> URL: https://issues.apache.org/jira/browse/FLINK-24229
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Reporter: Zichen Liu
>Assignee: Zichen Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> h2. Motivation
> *User stories:*
> As a Flink user, I’d like to use DynamoDB as sink for my data pipeline.
> *Scope:*
> * Implement an asynchronous sink for DynamoDB by inheriting the
> AsyncSinkBase class. The implementation can for now reside in its own module
> in flink-connectors.
> * Implement an asynchornous sink writer for DynamoDB by extending the
> AsyncSinkWriter. The implementation must deal with failed requests and retry
> them using the {{requeueFailedRequestEntry}} method. If possible, the
> implementation should batch multiple requests (PutRecordsRequestEntry
> objects) to Firehose for increased throughput. The implemented Sink Writer
> will be used by the Sink class that will be created as part of this story.
> * Java / code-level docs.
> * End to end testing: add tests that hits a real AWS instance. (How to best
> donate resources to the Flink project to allow this to happen?)
> h2. References
> More details to be found
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18470: [FLINK-25774][network] Restrict the maximum number of buffers can be used per result partition for sort-shuffle
flinkbot edited a comment on pull request #18470: URL: https://github.com/apache/flink/pull/18470#issuecomment-1020036276 ## CI report: * decf33d57cf16564786d77240287fb1aeb034b39 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30043) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18314: [FLINK-24228][connectors/firehose] - Unified Async Sink for Kinesis Firehose
flinkbot edited a comment on pull request #18314: URL: https://github.com/apache/flink/pull/18314#issuecomment-1008761781 ## CI report: * 826fd84d3f61953a127829be78708c6d54de3244 UNKNOWN * 26aa7ccebd039dfa7f890d9d927e93c22ca58a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29925) * b4803c1d8c0492316000c796a5f37e0427d61563 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30083) * ef2b24b71a3cc3ed707b71184158e9033abdedad Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30085) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25775) Document JavaScript SDK
[ https://issues.apache.org/jira/browse/FLINK-25775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25775: --- Labels: pull-request-available (was: ) > Document JavaScript SDK > --- > > Key: FLINK-25775 > URL: https://issues.apache.org/jira/browse/FLINK-25775 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.2.0 >Reporter: Till Rohrmann >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.2.0 > > > The newly introduced JavaScript SDK needs to be documented similarly to the > other SDKs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-statefun] igalshilman opened a new pull request #284: [FLINK-25775] Add Javascript SDK documentation
igalshilman opened a new pull request #284: URL: https://github.com/apache/flink-statefun/pull/284 This PR adds the JavaScript API to the docs: index: https://user-images.githubusercontent.com/546103/150853862-a48a5747-3951-4d54-8f3e-037013547a09.png";> js: https://user-images.githubusercontent.com/546103/150854026-72267202-f840-46ac-af32-f8cae61a12a8.png";> To build the docs locally use: ``` hugo -b "" serve ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18472: [FLINK-25783][docs-zh] Translate azure_table_storage.md page into Chinese.
flinkbot edited a comment on pull request #18472: URL: https://github.com/apache/flink/pull/18472#issuecomment-1020057651 ## CI report: * 7021a6893c943ab0a6ba24705b606884bfebbe2c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30039) * f2a102b8715567da4c779398863dc3ee9531fd92 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30063) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18314: [FLINK-24228][connectors/firehose] - Unified Async Sink for Kinesis Firehose
flinkbot edited a comment on pull request #18314: URL: https://github.com/apache/flink/pull/18314#issuecomment-1008761781 ## CI report: * 826fd84d3f61953a127829be78708c6d54de3244 UNKNOWN * 26aa7ccebd039dfa7f890d9d927e93c22ca58a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29925) * b4803c1d8c0492316000c796a5f37e0427d61563 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30083) * ef2b24b71a3cc3ed707b71184158e9033abdedad UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25775) Document JavaScript SDK
[ https://issues.apache.org/jira/browse/FLINK-25775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman reassigned FLINK-25775: Assignee: Igal Shilman > Document JavaScript SDK > --- > > Key: FLINK-25775 > URL: https://issues.apache.org/jira/browse/FLINK-25775 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Affects Versions: statefun-3.2.0 >Reporter: Till Rohrmann >Assignee: Igal Shilman >Priority: Major > Fix For: statefun-3.2.0 > > > The newly introduced JavaScript SDK needs to be documented similarly to the > other SDKs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18471: [FLINK-25780][network] Reduce the maximum size of data output buffers per result partition for sort-shuffle
flinkbot edited a comment on pull request #18471: URL: https://github.com/apache/flink/pull/18471#issuecomment-1020052968 ## CI report: * 6e793e2a37fbaf6618913fbf03887fc889c880ab Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30038) * 65c779eea512e90b8ec20eec4bbe56716f1b373e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30047) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18314: [FLINK-24228][connectors/firehose] - Unified Async Sink for Kinesis Firehose
flinkbot edited a comment on pull request #18314: URL: https://github.com/apache/flink/pull/18314#issuecomment-1008761781 ## CI report: * 826fd84d3f61953a127829be78708c6d54de3244 UNKNOWN * 26aa7ccebd039dfa7f890d9d927e93c22ca58a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29925) * b4803c1d8c0492316000c796a5f37e0427d61563 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30083) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18314: [FLINK-24228][connectors/firehose] - Unified Async Sink for Kinesis Firehose
flinkbot edited a comment on pull request #18314: URL: https://github.com/apache/flink/pull/18314#issuecomment-1008761781 ## CI report: * 826fd84d3f61953a127829be78708c6d54de3244 UNKNOWN * 26aa7ccebd039dfa7f890d9d927e93c22ca58a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29925) * b4803c1d8c0492316000c796a5f37e0427d61563 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30083) * ef2b24b71a3cc3ed707b71184158e9033abdedad UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18314: [FLINK-24228][connectors/firehose] - Unified Async Sink for Kinesis Firehose
flinkbot edited a comment on pull request #18314: URL: https://github.com/apache/flink/pull/18314#issuecomment-1008761781 ## CI report: * 826fd84d3f61953a127829be78708c6d54de3244 UNKNOWN * 26aa7ccebd039dfa7f890d9d927e93c22ca58a0f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29925) * b4803c1d8c0492316000c796a5f37e0427d61563 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30083) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
