[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27254) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ZhijieYang commented on a change in pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
ZhijieYang commented on a change in pull request #17902: URL: https://github.com/apache/flink/pull/17902#discussion_r759015168 ## File path: docs/content.zh/docs/ops/metrics.md ## @@ -28,26 +28,26 @@ under the License. # 指标 -Flink exposes a metric system that allows gathering and exposing metrics to external systems. +Flink 公开了一个指标系统，允许收集和公开指标给外部系统。 -## Registering metrics +## 注册指标 Review comment: sorry, I can't find the link here, what's your mean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27254) * e11b9136c39e3ab98011fdb54b5c9e5ea32e18e8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r759012504
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450930#comment-17450930
]
Arvid Heise commented on FLINK-24348:
-
Okay the script "testcontainers_start.sh" is generated in KafkaContainer in
test containers 1.16.0. With
https://github.com/testcontainers/testcontainers-java/pull/2078 that was
changed and either it fixes the issue or should provide us better logs, so I'm
proposing to upgrade to 1.16.2.
> Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container
> startup failed"
> --
>
> Key: FLINK-24348
> URL: https://issues.apache.org/jira/browse/FLINK-24348
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24338=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7140
> {code}
> Sep 21 02:44:33 org.testcontainers.containers.ContainerLaunchException:
> Container startup failed
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
> Sep 21 02:44:33 at
> org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
> Sep 21 02:44:33 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase$1.doStart(KafkaTableTestBase.java:71)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
> Sep 21 02:44:33 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Sep 21 02:44:33 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Sep 21 02:44:33 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
>
[jira] [Assigned] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul reassigned FLINK-18634:
---
Assignee: (was: Fabian Paul)
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Priority: Major
> Labels: auto-unassigned, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-22342) FlinkKafkaProducerITCase fails with producer leak
[
https://issues.apache.org/jira/browse/FLINK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul reassigned FLINK-22342:
---
Assignee: (was: Fabian Paul)
> FlinkKafkaProducerITCase fails with producer leak
> -
>
> Key: FLINK-22342
> URL: https://issues.apache.org/jira/browse/FLINK-22342
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16732=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20=6386
> {code}
> [ERROR]
> testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 8.854 s <<< FAILURE!
> java.lang.AssertionError: Detected producer leak. Thread name:
> kafka-producer-network-thread |
> producer-MockTask-002a002c-11
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.checkProducerLeak(FlinkKafkaProducerITCase.java:728)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint(FlinkKafkaProducerITCase.java:381)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r759008827
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r759007852
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27254) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r759003838
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17949: [hotfix][docs] Fix Scala example for MiniCluster test
flinkbot edited a comment on pull request #17949: URL: https://github.com/apache/flink/pull/17949#issuecomment-982158256 ## CI report: * a787ab271f51e7152894777d4740113b8070c011 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27242) * fce2ff2a8bf7a8cda2f1ace8bab72ceaf47a982b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27253) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17949: [hotfix][docs] Fix Scala example for MiniCluster test
flinkbot edited a comment on pull request #17949: URL: https://github.com/apache/flink/pull/17949#issuecomment-982158256 ## CI report: * a787ab271f51e7152894777d4740113b8070c011 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27242) * fce2ff2a8bf7a8cda2f1ace8bab72ceaf47a982b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #17878: [FLINK-24507][table] Cleanup DateTimeUtils
twalthr commented on a change in pull request #17878:
URL: https://github.com/apache/flink/pull/17878#discussion_r758998441
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/typeutils/serializers/python/DateSerializer.java
##
@@ -39,6 +41,11 @@
public static final DateSerializer INSTANCE = new DateSerializer();
+@SuppressWarnings({"unchecked", "rawtypes"})
+private static final DataStructureConverter converter =
Review comment:
@slinkydeveloper you forgot to drop the static field. This
`DateSerializer` is actually technical debt because the Python API still uses
the old type system and thus has to use `java.sql.` types. Hopefully this will
be fixed at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[
https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450921#comment-17450921
]
Arvid Heise commented on FLINK-22300:
-
Okay I see, so Flink would never evict the data. That is indeed a bug.
What do you think is the proper fix? I have a hard time to come up with proper
semantics beyond saying that users can't use {{TimeEvictor}} with processing
time. For a window in processing time, a user probably wants to retain all
elements before trigger and evict all elements after the trigger in all cases.
For other cases, they would use ingestion or event time.
> Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of
> TimeCharacteristic
>
>
> Key: FLINK-22300
> URL: https://issues.apache.org/jira/browse/FLINK-22300
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.11.3
>Reporter: Bo Huang
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime).
> The data after Windows process can not be evicted by TimeEvictor of Keyed
> Windows Beause TimestampedValue have no timestamp value.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17952: [FLINK-25011] Introduce vertex parallelism decider and the default implementation.
flinkbot edited a comment on pull request #17952: URL: https://github.com/apache/flink/pull/17952#issuecomment-982356774 ## CI report: * 53241c7762943233cafc115a929c73a9e4abd878 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27251) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17952: [FLINK-25011] Introduce vertex parallelism decider and the default implementation.
flinkbot commented on pull request #17952: URL: https://github.com/apache/flink/pull/17952#issuecomment-982357716 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 53241c7762943233cafc115a929c73a9e4abd878 (Tue Nov 30 07:29:09 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
Airblader commented on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-982357283 @shenzhu My PR was merged now. If you rebase I think those violations should not show up anymore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17952: [FLINK-25011] Introduce vertex parallelism decider and the default implementation.
flinkbot commented on pull request #17952: URL: https://github.com/apache/flink/pull/17952#issuecomment-982356774 ## CI report: * 53241c7762943233cafc115a929c73a9e4abd878 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25047) Resolve most (trivial) architectural violations in flink-table
[ https://issues.apache.org/jira/browse/FLINK-25047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-25047. Fix Version/s: 1.15.0 Resolution: Fixed Fixed in master: a09cc4704433cb76b936a51b422d812e1ae57945 > Resolve most (trivial) architectural violations in flink-table > -- > > Key: FLINK-25047 > URL: https://issues.apache.org/jira/browse/FLINK-25047 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Ingo Bürk >Assignee: Ingo Bürk >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25011) Introduce VertexParallelismDecider
[ https://issues.apache.org/jira/browse/FLINK-25011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25011: --- Labels: pull-request-available (was: ) > Introduce VertexParallelismDecider > -- > > Key: FLINK-25011 > URL: https://issues.apache.org/jira/browse/FLINK-25011 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Lijie Wang >Assignee: Lijie Wang >Priority: Major > Labels: pull-request-available > > Introduce VertexParallelismDecider and provide a default implementation. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] twalthr closed pull request #17898: [FLINK-25047][table] Resolve architectural violations
twalthr closed pull request #17898: URL: https://github.com/apache/flink/pull/17898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wanglijie95 opened a new pull request #17952: [FLINK-25011] Introduce vertex parallelism decider and the default implementation.
wanglijie95 opened a new pull request #17952: URL: https://github.com/apache/flink/pull/17952 ## What is the purpose of the change Introduce VertexParallelismDecider and default implementation for adaptive batch scheduler. ## Brief change log Introduce VertexParallelismDecider and default implementation for adaptive batch scheduler. ## Verifying this change Unit tests for the default implementation. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17749: [FLINK-24758][Connectors / FileSystem] filesystem sink: add partitiontime-extractor.formatter-pattern to allow user to speify DateTi
flinkbot edited a comment on pull request #17749: URL: https://github.com/apache/flink/pull/17749#issuecomment-965029957 ## CI report: * 2ce453d313fcea2e1fae39900e6ba97b7a129246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27243) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450912#comment-17450912
]
Arvid Heise commented on FLINK-24348:
-
I have not found out much yesterday. It is a standard Unix error but I'm unsure
if it is a general issue with test containers or with the specific kafka
container.
> Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container
> startup failed"
> --
>
> Key: FLINK-24348
> URL: https://issues.apache.org/jira/browse/FLINK-24348
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24338=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7140
> {code}
> Sep 21 02:44:33 org.testcontainers.containers.ContainerLaunchException:
> Container startup failed
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
> Sep 21 02:44:33 at
> org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
> Sep 21 02:44:33 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase$1.doStart(KafkaTableTestBase.java:71)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
> Sep 21 02:44:33 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Sep 21 02:44:33 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Sep 21 02:44:33 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
> Sep 21 02:44:33 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * 603f6ad9b9f8cf86cf63e7c6dc94dc56e3a2fd76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * c0becb684fd81c109a102859b15308d26be7f69c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ZhijieYang commented on a change in pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
ZhijieYang commented on a change in pull request #17902:
URL: https://github.com/apache/flink/pull/17902#discussion_r758982717
##
File path: docs/content.zh/docs/ops/metrics.md
##
@@ -28,26 +28,26 @@ under the License.
# 指标
-Flink exposes a metric system that allows gathering and exposing metrics to
external systems.
+Flink 公开了一个指标系统，允许收集和公开指标给外部系统。
-## Registering metrics
+## 注册指标
-You can access the metric system from any user function that extends
[RichFunction]({{< ref "docs/dev/datastream/user_defined_functions"
>}}#rich-functions) by calling `getRuntimeContext().getMetricGroup()`.
-This method returns a `MetricGroup` object on which you can create and
register new metrics.
+你可以通过调用从 [RichFunction]({{< ref "docs/dev/datastream/user_defined_functions"
>}}#rich-functions)
+扩展的任何用户函数的 `getRuntimeContext().getMetricGroup()` 方法访问指标系统。
+此方法返回一个 `MetricGroup` 对象，你可以在该对象上创建和注册新指标。
-### Metric types
+## 指标类型
-Flink supports `Counters`, `Gauges`, `Histograms` and `Meters`.
+Flink 支持计数器 `Counters`，量表 `Gauges`，直方图 `Histogram` 和仪表 `Meters`。
Review comment:
because I saw the
`https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/python/table/metrics/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25104) Kafka table source cannot be used as bounded
[ https://issues.apache.org/jira/browse/FLINK-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450908#comment-17450908 ] Hang Ruan commented on FLINK-25104: --- I will add stoppingOffsetsInitializer setting in the table options. If stoppingOffsetsInitializer is set, the kafka table source will be treated as bounded. > Kafka table source cannot be used as bounded > > > Key: FLINK-25104 > URL: https://issues.apache.org/jira/browse/FLINK-25104 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Hang Ruan >Priority: Minor > > In DataStream API, we could use a Kafka source as bounded by > `builder.setBounded`. > In SQL, there is not a way to set a Kafka table source as bounded. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * c0becb684fd81c109a102859b15308d26be7f69c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25105) Enables final checkpoint by default
Yun Gao created FLINK-25105: --- Summary: Enables final checkpoint by default Key: FLINK-25105 URL: https://issues.apache.org/jira/browse/FLINK-25105 Project: Flink Issue Type: Sub-task Components: Runtime / Checkpointing Affects Versions: 1.15.0 Reporter: Yun Gao Fix For: 1.15.0 We would fix all the conflict tests and enable the feature by default in this version. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25105) Enables final checkpoint by default
[ https://issues.apache.org/jira/browse/FLINK-25105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao reassigned FLINK-25105: --- Assignee: Yun Gao > Enables final checkpoint by default > --- > > Key: FLINK-25105 > URL: https://issues.apache.org/jira/browse/FLINK-25105 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Affects Versions: 1.15.0 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > Fix For: 1.15.0 > > > We would fix all the conflict tests and enable the feature by default in this > version. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * c0becb684fd81c109a102859b15308d26be7f69c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * c0becb684fd81c109a102859b15308d26be7f69c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17951: [FLINK-25095][table-planner] Fix generateComparison returnType
flinkbot edited a comment on pull request #17951: URL: https://github.com/apache/flink/pull/17951#issuecomment-982270439 ## CI report: * db68caf8d0edcbc8acb022cd2832d3ba56854eb9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27249) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25103) KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A, could you tell me how to store in the six tasks A
[
https://issues.apache.org/jira/browse/FLINK-25103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangbaohua updated FLINK-25103:
---
Issue Type: Improvement (was: Bug)
> KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A,
> could you tell me how to store in the six tasks A
>
>
> Key: FLINK-25103
> URL: https://issues.apache.org/jira/browse/FLINK-25103
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: wangbaohua
>Priority: Blocker
>
> KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A,
> excuse me how A stored in the six tasks. When I was running, I observed that
> some tasks fetched variable A was null, while others had values .The
> following code :
>
> setParallelism(9);
> ..
> public class dealStreamProcessFunction extends
> KeyedBroadcastProcessFunction,
> StandardEvent> {
> private static final Logger logger =
> LoggerFactory.getLogger(dealStreamProcessFunction.class);
> private transient ValueState> listState;
> private transient ValueState runingFlagState;
> private transient ValueState engineState;
> MapStateDescriptor> ruleStateDescriptor = new
> MapStateDescriptor<>(ContextInfo.RULE_SBROAD_CAST_STATE
> , BasicTypeInfo.STRING_TYPE_INFO
> , new ListTypeInfo<>(String.class));
> InferenceEngine engine;
> /**
> * open方法只会执行一次
> * 可以在这实现初始化的功能
> *
> * @param parameters
> * @throws Exception
> */
> @Override
> public void open(Configuration parameters) throws Exception {
> super.open(parameters);
> ValueStateDescriptor> recentOperatorsDescriptor =
> new ValueStateDescriptor>(
> "recent-operator",
> TypeInformation.of(new TypeHint>() {
> }));
> ValueStateDescriptor runingFlagDescriptor = new
> ValueStateDescriptor(
> "runingFlag",
> Boolean.class);
> ValueStateDescriptor engineDescriptor = new
> ValueStateDescriptor(
> "runingFlag1",
> InferenceEngine.class);
> engineState = getRuntimeContext().getState(engineDescriptor);
> listState = getRuntimeContext().getState(recentOperatorsDescriptor);
> runingFlagState = getRuntimeContext().getState(runingFlagDescriptor);
> logger.info("KeyedBroadcastProcessFunction open");
> }
> @Override
> public void processElement(StandardEvent standardEvent, ReadOnlyContext
> readOnlyContext, Collector collector) throws Exception {
> if(standardEvent == null){
> return;
> }
> List list = null;
> list =
> readOnlyContext.getBroadcastState(ruleStateDescriptor).get(ContextInfo.RULE_SBROAD_CAST_STATE);
> if (list == null) {
> logger.info("RulesBroadcastState is null..");
> List lst = listState.value();
> if (lst == null) {
> lst = new ArrayList<>();
> }
> lst.add(standardEvent);
> listState.update(lst);
> return;
> }
> //第一次进来
> if (runingFlagState.value() == null) {
> logger.info("runingFlagState.value() == null");
> runingFlagState.update(true);
> }
> if (((runingFlagState.value() && list.get(0).equals("1")) ||
> list.get(0).equals("0"))) {
> logger.info("action update.:" + list.size() + ":" +
> runingFlagState.value() + ":" + list.get(0));
> String flag = list.get(0);
> list.remove(0);
> InferenceEngine engine1 =
> InferenceEngine.compile(RuleReader.parseRules(list));
> engineState.update(engine1);
> if (runingFlagState.value() && flag.equals("1")) {
> runingFlagState.update(false);
> }
> }
> if (engineState.value() != null) {
> List listTmp = listState.value();
> if (listTmp != null) {
> for (StandardEvent standardEventTmp : listTmp) {
> logger.info("listState.:" + standardEventTmp);
> match(standardEventTmp, collector);
> }
> listState.clear();
> }
> match(standardEvent, collector);
> } else {
> logger.info("processElement engine is null.:");
> }
> }
> private void match(StandardEvent standardEvent, Collector
> collector) throws IOException {
> PatternMatcher matcher = engineState.value().matcher(standardEvent);
> if (matcher.find()) {
>
[jira] [Created] (FLINK-25104) Kafka table source cannot be used as bounded
Hang Ruan created FLINK-25104: - Summary: Kafka table source cannot be used as bounded Key: FLINK-25104 URL: https://issues.apache.org/jira/browse/FLINK-25104 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Reporter: Hang Ruan In DataStream API, we could use a Kafka source as bounded by `builder.setBounded`. In SQL, there is not a way to set a Kafka table source as bounded. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25085) Add a scheduled thread pool in Endpoint and close it when job terminates
[ https://issues.apache.org/jira/browse/FLINK-25085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shammon updated FLINK-25085: Description: Add a dedicated thread pool in Endpoint to schedule tasks that have a long delay such as PhysicalSlotRequestBulkCheckerImpl, heatbeat checker and some other timeout checker in JM/TM/RM. Job should shut down the thread pool and all the pending tasks will be removed when it terminates. (was: Add a dedicated thread pool in JM to schedule tasks that have a long delay such as PhysicalSlotRequestBulkCheckerImpl, heatbeat checker and some other timeout checker. Job should shut down the thread pool and all the pending tasks will be removed when it terminates. ) > Add a scheduled thread pool in Endpoint and close it when job terminates > > > Key: FLINK-25085 > URL: https://issues.apache.org/jira/browse/FLINK-25085 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > Add a dedicated thread pool in Endpoint to schedule tasks that have a long > delay such as PhysicalSlotRequestBulkCheckerImpl, heatbeat checker and some > other timeout checker in JM/TM/RM. Job should shut down the thread pool and > all the pending tasks will be removed when it terminates. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25085) Add a scheduled thread pool in Endpoint and close it when job terminates
[ https://issues.apache.org/jira/browse/FLINK-25085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shammon updated FLINK-25085: Summary: Add a scheduled thread pool in Endpoint and close it when job terminates (was: Add a scheduled thread pool in JM and close it when job terminates) > Add a scheduled thread pool in Endpoint and close it when job terminates > > > Key: FLINK-25085 > URL: https://issues.apache.org/jira/browse/FLINK-25085 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > Add a dedicated thread pool in JM to schedule tasks that have a long delay > such as PhysicalSlotRequestBulkCheckerImpl, heatbeat checker and some other > timeout checker. Job should shut down the thread pool and all the pending > tasks will be removed when it terminates. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27250) * c0becb684fd81c109a102859b15308d26be7f69c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758973297
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758973059
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink] flinkbot edited a comment on pull request #17902: [FLINK-25050][docs-zh] Translate "Metrics" page of "Operations" into Chinese
flinkbot edited a comment on pull request #17902: URL: https://github.com/apache/flink/pull/17902#issuecomment-978895303 ## CI report: * 8a398a42490ca1ab93a23b93d9e8c3c2abf14670 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27038) * 17af34f57d57f6e2731cab1d54a9f4765f383e74 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #17690: [FLINK-24490][docs] The sample code is wrong in Apache Kafka Connector page
JasonLeeCoding commented on pull request #17690: URL: https://github.com/apache/flink/pull/17690#issuecomment-982329624 @wuchong please help me take a look on this issue,thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17874: [FLINK-24046] Refactor the EmbeddedRocksDBStateBackend configuration logic
flinkbot edited a comment on pull request #17874: URL: https://github.com/apache/flink/pull/17874#issuecomment-976109395 ## CI report: * 060e52ce536943d38d0acbf1899aecd5e3ac5ed8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27248) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25025) BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-25025:
Assignee: Huang Xingbo
> BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount
> failed on AZP
> ---
>
> Key: FLINK-25025
> URL: https://issues.apache.org/jira/browse/FLINK-25025
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Assignee: Huang Xingbo
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.14.1
>
>
> The test
> {{BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount}}
> fails on AZP with:
> {code}
> Nov 23 04:20:13 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 0.167 s <<< FAILURE! - in
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.batch.BatchArrowPythonGroupWindowAggregateFunctionOperatorTest
> Nov 23 04:20:13 [ERROR] testFinishBundleTriggeredByCount Time elapsed: 0.068
> s <<< FAILURE!
> Nov 23 04:20:13 java.lang.AssertionError: expected:<6> but was:<4>
> Nov 23 04:20:13 at org.junit.Assert.fail(Assert.java:89)
> Nov 23 04:20:13 at org.junit.Assert.failNotEquals(Assert.java:835)
> Nov 23 04:20:13 at org.junit.Assert.assertEquals(Assert.java:647)
> Nov 23 04:20:13 at org.junit.Assert.assertEquals(Assert.java:633)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.util.RowDataHarnessAssertor.assertOutputEquals(RowDataHarnessAssertor.java:80)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.util.RowDataHarnessAssertor.assertOutputEquals(RowDataHarnessAssertor.java:60)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.ArrowPythonAggregateFunctionOperatorTestBase.assertOutputEquals(ArrowPythonAggregateFunctionOperatorTestBase.java:62)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.batch.BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount(BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.java:206)
> Nov 23 04:20:13 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 23 04:20:13 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 23 04:20:13 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 23 04:20:13 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 23 04:20:13 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 23 04:20:13 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 23 04:20:13 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 23 04:20:13 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Nov 23 04:20:13 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Nov 23 04:20:13 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Nov 23 04:20:13 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Nov 23 04:20:13 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Nov 23 04:20:13 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Nov 23 04:20:13 at
>
[jira] [Commented] (FLINK-25025) BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450887#comment-17450887
]
Huang Xingbo commented on FLINK-25025:
--
Ok. I will take a look.
> BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount
> failed on AZP
> ---
>
> Key: FLINK-25025
> URL: https://issues.apache.org/jira/browse/FLINK-25025
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.14.1
>
>
> The test
> {{BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount}}
> fails on AZP with:
> {code}
> Nov 23 04:20:13 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 0.167 s <<< FAILURE! - in
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.batch.BatchArrowPythonGroupWindowAggregateFunctionOperatorTest
> Nov 23 04:20:13 [ERROR] testFinishBundleTriggeredByCount Time elapsed: 0.068
> s <<< FAILURE!
> Nov 23 04:20:13 java.lang.AssertionError: expected:<6> but was:<4>
> Nov 23 04:20:13 at org.junit.Assert.fail(Assert.java:89)
> Nov 23 04:20:13 at org.junit.Assert.failNotEquals(Assert.java:835)
> Nov 23 04:20:13 at org.junit.Assert.assertEquals(Assert.java:647)
> Nov 23 04:20:13 at org.junit.Assert.assertEquals(Assert.java:633)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.util.RowDataHarnessAssertor.assertOutputEquals(RowDataHarnessAssertor.java:80)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.util.RowDataHarnessAssertor.assertOutputEquals(RowDataHarnessAssertor.java:60)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.ArrowPythonAggregateFunctionOperatorTestBase.assertOutputEquals(ArrowPythonAggregateFunctionOperatorTestBase.java:62)
> Nov 23 04:20:13 at
> org.apache.flink.table.runtime.operators.python.aggregate.arrow.batch.BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.testFinishBundleTriggeredByCount(BatchArrowPythonGroupWindowAggregateFunctionOperatorTest.java:206)
> Nov 23 04:20:13 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 23 04:20:13 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 23 04:20:13 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 23 04:20:13 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 23 04:20:13 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 23 04:20:13 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 23 04:20:13 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 23 04:20:13 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Nov 23 04:20:13 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 23 04:20:13 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Nov 23 04:20:13 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Nov 23 04:20:13 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Nov 23 04:20:13 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Nov 23 04:20:13 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Nov 23 04:20:13 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Nov 23 04:20:13 at
>
[jira] [Created] (FLINK-25103) KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A, could you tell me how to store in the six tasks A
wangbaohua created FLINK-25103:
--
Summary: KeyedBroadcastProcessFunction run set 6, parallelism
ValueState variables A, could you tell me how to store in the six tasks A
Key: FLINK-25103
URL: https://issues.apache.org/jira/browse/FLINK-25103
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.14.0
Reporter: wangbaohua
KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A,
excuse me how A stored in the six tasks. When I was running, I observed that
some tasks fetched variable A was null, while others had values .The following
code :
setParallelism(9);
..
public class dealStreamProcessFunction extends
KeyedBroadcastProcessFunction,
StandardEvent> {
private static final Logger logger =
LoggerFactory.getLogger(dealStreamProcessFunction.class);
private transient ValueState> listState;
private transient ValueState runingFlagState;
private transient ValueState engineState;
MapStateDescriptor> ruleStateDescriptor = new
MapStateDescriptor<>(ContextInfo.RULE_SBROAD_CAST_STATE
, BasicTypeInfo.STRING_TYPE_INFO
, new ListTypeInfo<>(String.class));
InferenceEngine engine;
/**
* open方法只会执行一次
* 可以在这实现初始化的功能
*
* @param parameters
* @throws Exception
*/
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
ValueStateDescriptor> recentOperatorsDescriptor =
new ValueStateDescriptor>(
"recent-operator",
TypeInformation.of(new TypeHint>() {
}));
ValueStateDescriptor runingFlagDescriptor = new
ValueStateDescriptor(
"runingFlag",
Boolean.class);
ValueStateDescriptor engineDescriptor = new
ValueStateDescriptor(
"runingFlag1",
InferenceEngine.class);
engineState = getRuntimeContext().getState(engineDescriptor);
listState = getRuntimeContext().getState(recentOperatorsDescriptor);
runingFlagState = getRuntimeContext().getState(runingFlagDescriptor);
logger.info("KeyedBroadcastProcessFunction open");
}
@Override
public void processElement(StandardEvent standardEvent, ReadOnlyContext
readOnlyContext, Collector collector) throws Exception {
if(standardEvent == null){
return;
}
List list = null;
list =
readOnlyContext.getBroadcastState(ruleStateDescriptor).get(ContextInfo.RULE_SBROAD_CAST_STATE);
if (list == null) {
logger.info("RulesBroadcastState is null..");
List lst = listState.value();
if (lst == null) {
lst = new ArrayList<>();
}
lst.add(standardEvent);
listState.update(lst);
return;
}
//第一次进来
if (runingFlagState.value() == null) {
logger.info("runingFlagState.value() == null");
runingFlagState.update(true);
}
if (((runingFlagState.value() && list.get(0).equals("1")) ||
list.get(0).equals("0"))) {
logger.info("action update.:" + list.size() + ":" +
runingFlagState.value() + ":" + list.get(0));
String flag = list.get(0);
list.remove(0);
InferenceEngine engine1 =
InferenceEngine.compile(RuleReader.parseRules(list));
engineState.update(engine1);
if (runingFlagState.value() && flag.equals("1")) {
runingFlagState.update(false);
}
}
if (engineState.value() != null) {
List listTmp = listState.value();
if (listTmp != null) {
for (StandardEvent standardEventTmp : listTmp) {
logger.info("listState.:" + standardEventTmp);
match(standardEventTmp, collector);
}
listState.clear();
}
match(standardEvent, collector);
} else {
logger.info("processElement engine is null.:");
}
}
private void match(StandardEvent standardEvent, Collector
collector) throws IOException {
PatternMatcher matcher = engineState.value().matcher(standardEvent);
if (matcher.find()) {
List actions = matcher.getActions();
for (Action action : actions) {
if (standardEvent != null) {
if(collector != null)
collector.collect(standardEvent);
else
logger.info("collector is null:" );
}
}
} else {
logger.info("no matcher:" + standardEvent);
}
}
@Override
public void processBroadcastElement(List strings, Context context,
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450886#comment-17450886
]
Yun Gao commented on FLINK-24348:
-
I copied some detailed log found be [~arvid] in another repeat issue here:
{code:java}
03:37:59,944 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Creating container for image:
confluentinc/cp-kafka:5.5.2
03:38:00,065 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Starting container with ID:
9f6a0e7664040345d6dc9a4189aea0bfcc2f191b31545e0ea0d191382454138c
03:38:00,537 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Container confluentinc/cp-kafka:5.5.2 is starting:
9f6a0e7664040345d6dc9a4189aea0bfcc2f191b31545e0ea0d191382454138c
03:38:00,640 [docker-java-stream-1817108049] INFO
org.apache.flink.connector.kafka.sink.KafkaSinkITCase[] - STDERR: sh:
1: /testcontainers_start.sh: Text file busy
03:38:00,800 [main] ERROR [confluentinc/cp-kafka:5.5.2]
[] - Could not start container
java.lang.IllegalStateException: Container did not start correctly.
at
org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:461)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:329)
~[testcontainers-1.16.0.jar:?]
at
org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
~[duct-tape-1.0.8.jar:?]
at
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
~[kafka-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
~[testcontainers-1.16.0.jar:?]
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
~[?:1.8.0_292]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_292]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_292]
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_292]
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
~[?:1.8.0_292]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_292]
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
~[?:1.8.0_292]
at
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
~[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
~[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
[jira] [Commented] (FLINK-25088) KafkaSinkITCase failed on azure due to Container did not start correctly
[
https://issues.apache.org/jira/browse/FLINK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450885#comment-17450885
]
Yun Gao commented on FLINK-25088:
-
Very thanks [~arvid] ! I'll also copy the above detailed log to there~
> KafkaSinkITCase failed on azure due to Container did not start correctly
>
>
> Key: FLINK-25088
> URL: https://issues.apache.org/jira/browse/FLINK-25088
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors / Kafka
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Assignee: Arvid Heise
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.1
>
>
> {code:java}
> Nov 29 03:38:00 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Nov 29 03:38:00 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Nov 29 03:38:00 Caused by: org.rnorth.ducttape.RetryCountExceededException:
> Retry limit hit with exception
> Nov 29 03:38:00 at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
> Nov 29 03:38:00 ... 33 more
> Nov 29 03:38:00 Caused by:
> org.testcontainers.containers.ContainerLaunchException: Could not
> create/start container
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:523)
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:329)
> Nov 29 03:38:00 at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
> Nov 29 03:38:00 ... 34 more
> Nov 29 03:38:00 Caused by: java.lang.IllegalStateException: Container did not
> start correctly.
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:461)
> Nov 29 03:38:00 ... 36 more
> Nov 29 03:38:00
> Nov 29 03:38:01 [INFO] Running
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase
> Nov 29 03:38:16 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 14.442 s - in
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [INFO] Results:
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [ERROR] Errors:
> Nov 29 03:38:16 [ERROR] KafkaSinkITCase » ContainerLaunch Container startup
> failed
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [ERROR] Tests run: 186, Failures: 0, Errors: 1, Skipped: 0
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27181=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7151
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758968020
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[jira] [Commented] (FLINK-25102) ParquetFileSystemITCase.testPartialDynamicPartition failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450881#comment-17450881
]
Jingsong Lee commented on FLINK-25102:
--
Thanks [~gaoyunhaii] for reporting, I will take a look~
> ParquetFileSystemITCase.testPartialDynamicPartition failed on azure
>
>
> Key: FLINK-25102
> URL: https://issues.apache.org/jira/browse/FLINK-25102
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2780)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3036)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2995)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1254)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1479)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.codec.SnappyCodec.createInputStream(SnappyCodec.java:75)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:109)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:103)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:99)
> Nov 29 23:00:01 at
> org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:120)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readPage(ColumnChunkPageReadStore.java:99)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:154)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:390)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:358)
> Nov 29 23:00:01 at
> org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
> Nov 29 23:00:01 at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
> Nov 29 23:00:01 at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
> Nov 29 23:00:01 ... 6 more
> Nov 29 23:00:01
> Nov 29 23:00:02 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 16.341 s - in
> org.apache.flink.formats.parquet.ParquetFileCompactionITCase
> Nov 29 23:00:03 [ERROR] Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> Nov 29 23:00:03 [INFO] Running
> org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
> Nov 29 23:00:20 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 17.033 s - in
> org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [INFO] Results:
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [ERROR] Errors:
> Nov 29 23:00:21 [ERROR] ParquetFileSystemITCase.testPartialDynamicPartition
> Nov 29 23:00:21 [ERROR] Run 1: Failed to fetch next result
> Nov 29 23:00:21 [INFO] Run 2: PASS
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [ERROR] Tests run: 42, Failures: 0, Errors: 1, Skipped: 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27235=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=11545
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17939: [FLINK-20370][table] part2: introduce 'table.exec.sink.pk-shuffle' option to auto keyby on sink's pk if parallelism are not the same
flinkbot edited a comment on pull request #17939: URL: https://github.com/apache/flink/pull/17939#issuecomment-981420515 ## CI report: * 9ccb311aec0b5f0d14197c5de37a171ea15c6097 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27244) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20370) Result is wrong when sink primary key is not the same with query
[
https://issues.apache.org/jira/browse/FLINK-20370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450880#comment-17450880
]
Wenlong Lyu commented on FLINK-20370:
-
[~lincoln.86xy] Even sink implements ParallelismProvider, parallelism can still
be undefined, because it returns an optional value.
> Result is wrong when sink primary key is not the same with query
>
>
> Key: FLINK-20370
> URL: https://issues.apache.org/jira/browse/FLINK-20370
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Jark Wu
>Assignee: lincoln lee
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Both sources are upsert-kafka which synchronizes the changes from MySQL
> tables (source_city, source_customer). The sink is another MySQL table which
> is in upsert mode with "city_name" primary key. The join key is "city_id".
> In this case, the result will be wrong when updating
> {{source_city.city_name}} column in MySQL, as the UPDATE_BEFORE is ignored
> and the old city_name is retained in the sink table.
> {code}
> Sink(table=[default_catalog.default_database.sink_kafka_count_city],
> fields=[city_name, count_customer, sum_gender], changelogMode=[NONE])
> +- Calc(select=[city_name, CAST(count_customer) AS count_customer,
> CAST(sum_gender) AS sum_gender], changelogMode=[I,UA,D])
>+- Join(joinType=[InnerJoin], where=[=(city_id, id)], select=[city_id,
> count_customer, sum_gender, id, city_name],
> leftInputSpec=[JoinKeyContainsUniqueKey],
> rightInputSpec=[JoinKeyContainsUniqueKey], changelogMode=[I,UA,D])
> :- Exchange(distribution=[hash[city_id]], changelogMode=[I,UA,D])
> : +- GlobalGroupAggregate(groupBy=[city_id], select=[city_id,
> COUNT_RETRACT(count1$0) AS count_customer, SUM_RETRACT((sum$1, count$2)) AS
> sum_gender], changelogMode=[I,UA,D])
> : +- Exchange(distribution=[hash[city_id]], changelogMode=[I])
> :+- LocalGroupAggregate(groupBy=[city_id], select=[city_id,
> COUNT_RETRACT(*) AS count1$0, SUM_RETRACT(gender) AS (sum$1, count$2)],
> changelogMode=[I])
> : +- Calc(select=[city_id, gender], changelogMode=[I,UB,UA,D])
> : +- ChangelogNormalize(key=[customer_id],
> changelogMode=[I,UB,UA,D])
> : +- Exchange(distribution=[hash[customer_id]],
> changelogMode=[UA,D])
> :+- MiniBatchAssigner(interval=[3000ms],
> mode=[ProcTime], changelogMode=[UA,D])
> : +- TableSourceScan(table=[[default_catalog,
> default_database, source_customer]], fields=[customer_id, city_id, age,
> gender, update_time], changelogMode=[UA,D])
> +- Exchange(distribution=[hash[id]], changelogMode=[I,UA,D])
> +- ChangelogNormalize(key=[id], changelogMode=[I,UA,D])
> +- Exchange(distribution=[hash[id]], changelogMode=[UA,D])
>+- MiniBatchAssigner(interval=[3000ms], mode=[ProcTime],
> changelogMode=[UA,D])
> +- TableSourceScan(table=[[default_catalog,
> default_database, source_city]], fields=[id, city_name], changelogMode=[UA,D])
> {code}
> We have suggested users to use the same key of the query as the primary key
> on sink in the documentation:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication.
> We should make this attention to be more highlight in CREATE TABLE page.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] MyLanPangzi commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
MyLanPangzi commented on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-982321397
@maosuhan This would be a good option for users if it could be converted
automatically. like this
PbCodegenSimpleDeserializer#codegen
```java
if(type instanceof IntType){
sb.append(returnInternalDataVarName).append(" =
").append(pbGetStr).append(".getNumber();");
break;
}
sb.append(returnInternalDataVarName)
.append(" =
BinaryStringData.fromString(").append(pbGetStr).append(".toString());");
break;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17684: [FLINK-23999][table-planner] Support evaluating individual window table-valued function in planner
flinkbot edited a comment on pull request #17684: URL: https://github.com/apache/flink/pull/17684#issuecomment-961007710 ## CI report: * 14a1758e9482b8f3319fb39775429d22f7a62b97 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27246) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * cd6fdccd8986b45c43d05bde8efc226a8487d139 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27245) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24935) Python module failed to compile due to "Could not create local repository"
[
https://issues.apache.org/jira/browse/FLINK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450878#comment-17450878
]
Yun Gao commented on FLINK-24935:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27236=logs=a29bcfe1-064d-50b9-354f-07802213a3c0=47ff6576-c9dc-5eab-9db8-183dcca3bede=28]
> Python module failed to compile due to "Could not create local repository"
> --
>
> Key: FLINK-24935
> URL: https://issues.apache.org/jira/browse/FLINK-24935
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> Invoking mvn with 'mvn -Dmaven.wagon.http.pool=false --settings
> /__w/1/s/tools/ci/google-mirror-settings.xml
> -Dorg.slf4j.simpleLogger.showDateTime=true
> -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
> -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
> --no-snapshot-updates -B -Dhadoop.version=2.8.3 -Dinclude_hadoop_aws
> -Dscala-2.11 clean deploy
> -DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment
> -Dmaven.repo.local=/home/vsts/work/1/.m2/repository
> -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2
> -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -U -DskipTests'
> [ERROR] Could not create local repository at /home/vsts/work/1/.m2/repository
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26625=logs=a29bcfe1-064d-50b9-354f-07802213a3c0=47ff6576-c9dc-5eab-9db8-183dcca3bede]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25102) ParquetFileSystemITCase.testPartialDynamicPartition failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450877#comment-17450877
]
Yun Gao commented on FLINK-25102:
-
Hi [~lzljs3620320] this issue seems to have the same cause with
https://issues.apache.org/jira/browse/FLINK-24763, could you have a double look
on this issue or if we need to also ignore this case~?
> ParquetFileSystemITCase.testPartialDynamicPartition failed on azure
>
>
> Key: FLINK-25102
> URL: https://issues.apache.org/jira/browse/FLINK-25102
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2780)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3036)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2995)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1254)
> Nov 29 23:00:01 at
> org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1479)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.codec.SnappyCodec.createInputStream(SnappyCodec.java:75)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:109)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:103)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:99)
> Nov 29 23:00:01 at
> org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:120)
> Nov 29 23:00:01 at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readPage(ColumnChunkPageReadStore.java:99)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:154)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:390)
> Nov 29 23:00:01 at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:358)
> Nov 29 23:00:01 at
> org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
> Nov 29 23:00:01 at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
> Nov 29 23:00:01 at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
> Nov 29 23:00:01 ... 6 more
> Nov 29 23:00:01
> Nov 29 23:00:02 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 16.341 s - in
> org.apache.flink.formats.parquet.ParquetFileCompactionITCase
> Nov 29 23:00:03 [ERROR] Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> Nov 29 23:00:03 [INFO] Running
> org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
> Nov 29 23:00:20 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 17.033 s - in
> org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [INFO] Results:
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [ERROR] Errors:
> Nov 29 23:00:21 [ERROR] ParquetFileSystemITCase.testPartialDynamicPartition
> Nov 29 23:00:21 [ERROR] Run 1: Failed to fetch next result
> Nov 29 23:00:21 [INFO] Run 2: PASS
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [INFO]
> Nov 29 23:00:21 [ERROR] Tests run: 42, Failures: 0, Errors: 1, Skipped: 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27235=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=11545
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] maosuhan commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
maosuhan commented on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-982317036 @MyLanPangzi Thanks for your feedback. I think it is doable, what about adding a new option like 'protobuf.enum-as-int'? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25102) ParquetFileSystemITCase.testPartialDynamicPartition failed on azure
Yun Gao created FLINK-25102:
---
Summary: ParquetFileSystemITCase.testPartialDynamicPartition
failed on azure
Key: FLINK-25102
URL: https://issues.apache.org/jira/browse/FLINK-25102
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2780)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3036)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2995)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1254)
Nov 29 23:00:01 at
org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1479)
Nov 29 23:00:01 at
org.apache.parquet.hadoop.codec.SnappyCodec.createInputStream(SnappyCodec.java:75)
Nov 29 23:00:01 at
org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:109)
Nov 29 23:00:01 at
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:103)
Nov 29 23:00:01 at
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:99)
Nov 29 23:00:01 at
org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:120)
Nov 29 23:00:01 at
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readPage(ColumnChunkPageReadStore.java:99)
Nov 29 23:00:01 at
org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:154)
Nov 29 23:00:01 at
org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:390)
Nov 29 23:00:01 at
org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:358)
Nov 29 23:00:01 at
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
Nov 29 23:00:01 at
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
Nov 29 23:00:01 at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
Nov 29 23:00:01 ... 6 more
Nov 29 23:00:01
Nov 29 23:00:02 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 16.341 s - in
org.apache.flink.formats.parquet.ParquetFileCompactionITCase
Nov 29 23:00:03 [ERROR] Picked up JAVA_TOOL_OPTIONS:
-XX:+HeapDumpOnOutOfMemoryError
Nov 29 23:00:03 [INFO] Running
org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
Nov 29 23:00:20 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 17.033 s - in
org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
Nov 29 23:00:21 [INFO]
Nov 29 23:00:21 [INFO] Results:
Nov 29 23:00:21 [INFO]
Nov 29 23:00:21 [ERROR] Errors:
Nov 29 23:00:21 [ERROR] ParquetFileSystemITCase.testPartialDynamicPartition
Nov 29 23:00:21 [ERROR] Run 1: Failed to fetch next result
Nov 29 23:00:21 [INFO] Run 2: PASS
Nov 29 23:00:21 [INFO]
Nov 29 23:00:21 [INFO]
Nov 29 23:00:21 [ERROR] Tests run: 42, Failures: 0, Errors: 1, Skipped: 1
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27235=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=11545
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25099) flink on yarn Accessing two HDFS Clusters
[
https://issues.apache.org/jira/browse/FLINK-25099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450876#comment-17450876
]
chenqizhu edited comment on FLINK-25099 at 11/30/21, 6:04 AM:
--
[~zuston] This is all the client logs
[^flink-chenqizhu-client-hdfsn21n163.log]
Flinkcluster in the log is the Bcluster mentioned above
was (Author: libra_816):
[~zuston] This is all the client logs
[^flink-chenqizhu-client-hdfsn21n163.log]
Flinkcluster in the log is the clusterB mentioned above
> flink on yarn Accessing two HDFS Clusters
> -
>
> Key: FLINK-25099
> URL: https://issues.apache.org/jira/browse/FLINK-25099
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / State Backends
>Affects Versions: 1.13.3
> Environment: flink : 1.13.3
> hadoop : 3.3.0
>Reporter: chenqizhu
>Priority: Major
> Attachments: flink-chenqizhu-client-hdfsn21n163.log
>
>
> Flink version 1.13 supports configuration of Hadoop properties in
> flink-conf.yaml via flink.hadoop.*. There is A requirement to write
> checkpoint to HDFS with SSDS (called cluster B) to speed checkpoint writing,
> but this HDFS cluster is not the default HDFS in the flink client (called
> cluster A by default). Yaml is configured with nameservices for cluster A and
> cluster B, which is similar to HDFS federated mode.
> The configuration is as follows:
>
> {code:java}
> flink.hadoop.dfs.nameservices: ACluster,BCluster
> flink.hadoop.fs.defaultFS: hdfs://BCluster
> flink.hadoop.dfs.ha.namenodes.ACluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn1: 10.:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn1: 10.:50070
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn2: 10.xx:50070
> flink.hadoop.dfs.client.failover.proxy.provider.ACluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.BCluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn1: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn1: 10.xx:50070
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn2: 10.x:50070
> flink.hadoop.dfs.client.failover.proxy.provider.BCluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> {code}
>
> However, an error occurred during the startup of the job, which is reported
> as follows:
> (change configuration items to A flink local client default HDFS cluster, the
> operation can be normal boot: flink.hadoop.fs.DefaultFS: hdfs: / / ACluster)
> {noformat}
> Caused by: BCluster
> java.net.UnknownHostException: BCluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:448)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:374)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:270)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:68)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:415)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:412)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:412)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:247)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:240)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:228)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
[jira] [Comment Edited] (FLINK-25099) flink on yarn Accessing two HDFS Clusters
[
https://issues.apache.org/jira/browse/FLINK-25099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450876#comment-17450876
]
chenqizhu edited comment on FLINK-25099 at 11/30/21, 6:03 AM:
--
[~zuston] This is all the client logs
[^flink-chenqizhu-client-hdfsn21n163.log]
Flinkcluster in the log is the clusterB mentioned above
was (Author: libra_816):
[~zuston] This is all the client logs
[^flink-chenqizhu-client-hdfsn21n163.log]
> flink on yarn Accessing two HDFS Clusters
> -
>
> Key: FLINK-25099
> URL: https://issues.apache.org/jira/browse/FLINK-25099
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / State Backends
>Affects Versions: 1.13.3
> Environment: flink : 1.13.3
> hadoop : 3.3.0
>Reporter: chenqizhu
>Priority: Major
> Attachments: flink-chenqizhu-client-hdfsn21n163.log
>
>
> Flink version 1.13 supports configuration of Hadoop properties in
> flink-conf.yaml via flink.hadoop.*. There is A requirement to write
> checkpoint to HDFS with SSDS (called cluster B) to speed checkpoint writing,
> but this HDFS cluster is not the default HDFS in the flink client (called
> cluster A by default). Yaml is configured with nameservices for cluster A and
> cluster B, which is similar to HDFS federated mode.
> The configuration is as follows:
>
> {code:java}
> flink.hadoop.dfs.nameservices: ACluster,BCluster
> flink.hadoop.fs.defaultFS: hdfs://BCluster
> flink.hadoop.dfs.ha.namenodes.ACluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn1: 10.:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn1: 10.:50070
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn2: 10.xx:50070
> flink.hadoop.dfs.client.failover.proxy.provider.ACluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.BCluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn1: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn1: 10.xx:50070
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn2: 10.x:50070
> flink.hadoop.dfs.client.failover.proxy.provider.BCluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> {code}
>
> However, an error occurred during the startup of the job, which is reported
> as follows:
> (change configuration items to A flink local client default HDFS cluster, the
> operation can be normal boot: flink.hadoop.fs.DefaultFS: hdfs: / / ACluster)
> {noformat}
> Caused by: BCluster
> java.net.UnknownHostException: BCluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:448)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:374)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:270)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:68)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:415)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:412)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:412)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:247)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:240)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:228)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Commented] (FLINK-25099) flink on yarn Accessing two HDFS Clusters
[
https://issues.apache.org/jira/browse/FLINK-25099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450876#comment-17450876
]
chenqizhu commented on FLINK-25099:
---
[~zuston] This is all the client logs
[^flink-chenqizhu-client-hdfsn21n163.log]
> flink on yarn Accessing two HDFS Clusters
> -
>
> Key: FLINK-25099
> URL: https://issues.apache.org/jira/browse/FLINK-25099
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / State Backends
>Affects Versions: 1.13.3
> Environment: flink : 1.13.3
> hadoop : 3.3.0
>Reporter: chenqizhu
>Priority: Major
> Attachments: flink-chenqizhu-client-hdfsn21n163.log
>
>
> Flink version 1.13 supports configuration of Hadoop properties in
> flink-conf.yaml via flink.hadoop.*. There is A requirement to write
> checkpoint to HDFS with SSDS (called cluster B) to speed checkpoint writing,
> but this HDFS cluster is not the default HDFS in the flink client (called
> cluster A by default). Yaml is configured with nameservices for cluster A and
> cluster B, which is similar to HDFS federated mode.
> The configuration is as follows:
>
> {code:java}
> flink.hadoop.dfs.nameservices: ACluster,BCluster
> flink.hadoop.fs.defaultFS: hdfs://BCluster
> flink.hadoop.dfs.ha.namenodes.ACluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn1: 10.:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn1: 10.:50070
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn2: 10.xx:50070
> flink.hadoop.dfs.client.failover.proxy.provider.ACluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.BCluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn1: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn1: 10.xx:50070
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn2: 10.x:50070
> flink.hadoop.dfs.client.failover.proxy.provider.BCluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> {code}
>
> However, an error occurred during the startup of the job, which is reported
> as follows:
> (change configuration items to A flink local client default HDFS cluster, the
> operation can be normal boot: flink.hadoop.fs.DefaultFS: hdfs: / / ACluster)
> {noformat}
> Caused by: BCluster
> java.net.UnknownHostException: BCluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:448)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:374)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:270)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:68)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:415)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:412)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:412)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:247)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:240)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:228)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Updated] (FLINK-25099) flink on yarn Accessing two HDFS Clusters
[
https://issues.apache.org/jira/browse/FLINK-25099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenqizhu updated FLINK-25099:
--
Attachment: flink-chenqizhu-client-hdfsn21n163.log
> flink on yarn Accessing two HDFS Clusters
> -
>
> Key: FLINK-25099
> URL: https://issues.apache.org/jira/browse/FLINK-25099
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / State Backends
>Affects Versions: 1.13.3
> Environment: flink : 1.13.3
> hadoop : 3.3.0
>Reporter: chenqizhu
>Priority: Major
> Attachments: flink-chenqizhu-client-hdfsn21n163.log
>
>
> Flink version 1.13 supports configuration of Hadoop properties in
> flink-conf.yaml via flink.hadoop.*. There is A requirement to write
> checkpoint to HDFS with SSDS (called cluster B) to speed checkpoint writing,
> but this HDFS cluster is not the default HDFS in the flink client (called
> cluster A by default). Yaml is configured with nameservices for cluster A and
> cluster B, which is similar to HDFS federated mode.
> The configuration is as follows:
>
> {code:java}
> flink.hadoop.dfs.nameservices: ACluster,BCluster
> flink.hadoop.fs.defaultFS: hdfs://BCluster
> flink.hadoop.dfs.ha.namenodes.ACluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn1: 10.:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn1: 10.:50070
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn2: 10.xx:50070
> flink.hadoop.dfs.client.failover.proxy.provider.ACluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.BCluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn1: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn1: 10.xx:50070
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn2: 10.x:50070
> flink.hadoop.dfs.client.failover.proxy.provider.BCluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> {code}
>
> However, an error occurred during the startup of the job, which is reported
> as follows:
> (change configuration items to A flink local client default HDFS cluster, the
> operation can be normal boot: flink.hadoop.fs.DefaultFS: hdfs: / / ACluster)
> {noformat}
> Caused by: BCluster
> java.net.UnknownHostException: BCluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:448)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:374)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:270)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:68)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:415)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:412)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:412)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:247)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:240)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:228)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
> Is there a solution to the above problems? The pain
[jira] [Commented] (FLINK-24926) Key group is not in KeyGroupRange when joining two streams with table API
[
https://issues.apache.org/jira/browse/FLINK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450874#comment-17450874
]
Yun Tang commented on FLINK-24926:
--
Thanks for the explanation [~lincoln.86xy].
You might need to pick the fix or wait for the flink-1.15 release, [~liuhb86].
> Key group is not in KeyGroupRange when joining two streams with table API
> -
>
> Key: FLINK-24926
> URL: https://issues.apache.org/jira/browse/FLINK-24926
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.14.0
>Reporter: Hongbo
>Priority: Major
>
> I have a simple test to join two streams by the event time:
>
> {code:java}
> @Test
> void testJoinStream() {
> var settings = EnvironmentSettings
> .newInstance()
> .inStreamingMode()
> .build();
> var tableEnv = TableEnvironment.create(settings);
> var configuration = tableEnv.getConfig().getConfiguration();
> configuration.setString("table.exec.resource.default-parallelism", "2");
> var testTable = tableEnv.from(TableDescriptor.forConnector("datagen")
> .schema(Schema.newBuilder()
> .column("ts", DataTypes.TIMESTAMP(3))
> .column("v", DataTypes.INT())
> .watermark("ts", "ts - INTERVAL '1' second")
> .build())
> .option(DataGenConnectorOptions.ROWS_PER_SECOND, 2L)
> .option("fields.v.kind", "sequence")
> .option("fields.v.start", "0")
> .option("fields.v.end", "100")
> .build());
> testTable.printSchema();
> tableEnv.createTemporaryView("test", testTable );
> var joined = tableEnv.sqlQuery("SELECT ts, v, v2 from test" +
> " join (SELECT ts as ts2, v as v2 from test) on ts = ts2");
> try {
> var tableResult =
> joined.executeInsert(TableDescriptor.forConnector("print").build());
> tableResult.await();
> } catch (InterruptedException | ExecutionException e) {
> throw new RuntimeException(e);
> }
> } {code}
> It failed within a few seconds:
> {code:java}
> (
> `ts` TIMESTAMP(3) *ROWTIME*,
> `v` INT,
> WATERMARK FOR `ts`: TIMESTAMP(3) AS ts - INTERVAL '1' second
> )
> 1> +I[2021-11-16T17:48:24.415, 1, 1]
> 1> +I[2021-11-16T17:48:24.415, 0, 1]
> 1> +I[2021-11-16T17:48:24.415, 1, 0]
> 1> +I[2021-11-16T17:48:24.415, 0, 0]
> java.util.concurrent.ExecutionException:
> org.apache.flink.table.api.TableException: Failed to wait job finish
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.flink.table.api.TableException: Failed to wait job finish
> at com.microstrategy.realtime.FlinkTest.testJoinStream(FlinkTest.java:123)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> at
>
[jira] [Updated] (FLINK-23047) CassandraConnectorITCase.testCassandraBatchTupleFormat fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23047:
Affects Version/s: 1.15.0
> CassandraConnectorITCase.testCassandraBatchTupleFormat fails on azure
> -
>
> Key: FLINK-23047
> URL: https://issues.apache.org/jira/browse/FLINK-23047
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Cassandra
>Affects Versions: 1.14.0, 1.12.4, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19176=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=13995
> {code}
> [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 157.28 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase
> [ERROR]
> testCassandraBatchTupleFormat(org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase)
> Time elapsed: 12.052 s <<< ERROR!
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1:9042
> (com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1]
> Timed out waiting for server response))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.createTable(CassandraConnectorITCase.java:234)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by:
[jira] [Commented] (FLINK-23047) CassandraConnectorITCase.testCassandraBatchTupleFormat fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450872#comment-17450872
]
Yun Gao commented on FLINK-23047:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27229=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=11032]
> CassandraConnectorITCase.testCassandraBatchTupleFormat fails on azure
> -
>
> Key: FLINK-23047
> URL: https://issues.apache.org/jira/browse/FLINK-23047
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Cassandra
>Affects Versions: 1.14.0, 1.12.4, 1.13.2
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19176=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=13995
> {code}
> [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 157.28 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase
> [ERROR]
> testCassandraBatchTupleFormat(org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase)
> Time elapsed: 12.052 s <<< ERROR!
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1:9042
> (com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1]
> Timed out waiting for server response))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.createTable(CassandraConnectorITCase.java:234)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
>
[GitHub] [flink] Airblader commented on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
Airblader commented on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-982311002 @shenzhu I'm addressing those in https://github.com/apache/flink/pull/17898. In the meantime for this PR you can add the violations to the file since you're only exposing additional usages of those classes and not really new kinds of violations. I don't know if ObjectPath was ever intended to be public, but de facto it simply is now — the Catalog interface entirely relies on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shenzhu commented on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
shenzhu commented on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-982309892 Hey @Airblader , I got some architecture tests failing for this PR, the failing messages are as follows ``` java.lang.AssertionError: Architecture Violation [Priority: MEDIUM] - Rule 'Return and argument types of methods annotated with @PublicEvolving must be annotated with @Public(Evolving).' was violated (3 times): org.apache.flink.table.catalog.Catalog.renameFunction(org.apache.flink.table.catalog.ObjectPath, java.lang.String, boolean): Argument leaf type org.apache.flink.table.catalog.ObjectPath does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated org.apache.flink.table.catalog.Catalog.renameFunction(org.apache.flink.table.catalog.ObjectPath, java.lang.String, boolean): Exception leaf type org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated org.apache.flink.table.catalog.Catalog.renameFunction(org.apache.flink.table.catalog.ObjectPath, java.lang.String, boolean): Exception leaf type org.apache.flink.table.catalog.exceptions.FunctionNotExistException does not satisfy: reside outside of package 'org.apache.flink..' or annotated with @Public or annotated with @PublicEvolving or annotated with @Deprecated ``` Later I [added `@PublicEvolving` annotation](https://github.com/apache/flink/pull/17788/commits/c693223bdadb16abe3c2f9b1fe53891bf1a69460) and that fixed these tests, however, I thought `ObjectPath`, `FunctionAlreadyExists` and `FunctionNotExists` are not intended for public use(not sure I'm correct here). Based on the [README](https://github.com/apache/flink/commit/b6af27e1229c8922ca9ea6e1f71dd5adeef2b7c7#diff-b723668ed95dd063b89b7edc675b3d514b1fe0107f97eda9b2cd635108d0461fR20), should I create a JIRA issue for this? Thanks for your help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25101) FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due to CorrelationIdMismatchException
[
https://issues.apache.org/jira/browse/FLINK-25101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25101:
Summary: FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce
failed due to CorrelationIdMismatchException (was:
FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due to )
> FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due
> to CorrelationIdMismatchException
> -
>
> Key: FLINK-25101
> URL: https://issues.apache.org/jira/browse/FLINK-25101
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
>
> {code:java}
> Nov 29 17:19:22 at
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> Nov 29 17:19:22 at
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> Nov 29 17:19:22
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [INFO] Results:
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [ERROR] Errors:
> Nov 29 17:19:22 [ERROR]
> FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce:592->KafkaTestBase.assertExactlyOnceForTopic:317
> » CorrelationIdMismatch
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [ERROR] Tests run: 175, Failures: 0, Errors: 1, Skipped: 0
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27226=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=35495]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25101) FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due to CorrelationIdMismatchException
[
https://issues.apache.org/jira/browse/FLINK-25101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25101:
Labels: test-stability (was: )
> FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due
> to CorrelationIdMismatchException
> -
>
> Key: FLINK-25101
> URL: https://issues.apache.org/jira/browse/FLINK-25101
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Nov 29 17:19:22 at
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> Nov 29 17:19:22 at
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> Nov 29 17:19:22 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> Nov 29 17:19:22
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [INFO] Results:
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [ERROR] Errors:
> Nov 29 17:19:22 [ERROR]
> FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce:592->KafkaTestBase.assertExactlyOnceForTopic:317
> » CorrelationIdMismatch
> Nov 29 17:19:22 [INFO]
> Nov 29 17:19:22 [ERROR] Tests run: 175, Failures: 0, Errors: 1, Skipped: 0
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27226=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=35495]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25101) FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due to
Yun Gao created FLINK-25101:
---
Summary:
FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed due to
Key: FLINK-25101
URL: https://issues.apache.org/jira/browse/FLINK-25101
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Nov 29 17:19:22 at
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
Nov 29 17:19:22 at
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
Nov 29 17:19:22 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
Nov 29 17:19:22 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
Nov 29 17:19:22 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
Nov 29 17:19:22 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
Nov 29 17:19:22 at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
Nov 29 17:19:22 at
org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
Nov 29 17:19:22 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Nov 29 17:19:22
Nov 29 17:19:22 [INFO]
Nov 29 17:19:22 [INFO] Results:
Nov 29 17:19:22 [INFO]
Nov 29 17:19:22 [ERROR] Errors:
Nov 29 17:19:22 [ERROR]
FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce:592->KafkaTestBase.assertExactlyOnceForTopic:317
» CorrelationIdMismatch
Nov 29 17:19:22 [INFO]
Nov 29 17:19:22 [ERROR] Tests run: 175, Failures: 0, Errors: 1, Skipped: 0
{code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27226=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=35495]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] shenzhu commented on pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
shenzhu commented on pull request #17919: URL: https://github.com/apache/flink/pull/17919#issuecomment-982304022 > Hi @shenzhu I left a couple more comments, but overall the approach is not 100% correct, because it applies the trimming only when casting from string types. It should also apply it when casting between binary types themselves, i.e. a byte array with 10 bytes and type `BINARY(10)`, casted to let's say a `BINARY(5)` (or `VARBINARY(5)`) should be trimmed to 5 bytes. So this logic should be applied in for all supported casts to BINARY/VARBINARY. Hey @matriv , thanks for your review! I checked the codebase and found currently we support casting from `STRING/BINARY/VARBINARY/RAW` to `BINARY/VARBINARY`. For these supported casts, `STRING` to `BINARY/VARBINARY` is supported by [StringToBinaryCastRule](https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/StringToBinaryCastRule.java), and the rest is supported via old casting rules in `ScalarOperatorGens`([BINARY|VARBINARY](https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala#L885) and [RAW](https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala#L1092)). If I understand it correctly, we have two options for this task: 1. Create `BinaryToBinaryCastRule` class and `RawToBinaryCastRule` class to follow the new casting rules 2. Update the casting logic in `ScalaOperatorGens` for `BINARY/VARBINARY` and `RAW` to add truncation logic I'm a little prefer Option 1 because seems that's the direction the community is trying to move forward(?), what do you think about this? Thanks for your help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25100) RMQSourceITCase failed on azure due to java.io.EOFException
Yun Gao created FLINK-25100:
---
Summary: RMQSourceITCase failed on azure due to
java.io.EOFException
Key: FLINK-25100
URL: https://issues.apache.org/jira/browse/FLINK-25100
Project: Flink
Issue Type: Bug
Components: Connectors/ RabbitMQ
Affects Versions: 1.14.1
Reporter: Yun Gao
{code:java}
Nov 29 12:02:05 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
Nov 29 12:02:05 at
org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
Nov 29 12:02:05 at
org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
Nov 29 12:02:05 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
Nov 29 12:02:05 at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
Nov 29 12:02:05 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
Nov 29 12:02:05 at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
Nov 29 12:02:05 at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Nov 29 12:02:05 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Nov 29 12:02:05 at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
Nov 29 12:02:05 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Nov 29 12:02:05 Caused by: com.rabbitmq.client.AlreadyClosedException:
connection is already closed due to connection error; cause:
java.io.EOFException
Nov 29 12:02:05 at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258)
Nov 29 12:02:05 at
com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:341)
Nov 29 12:02:05 at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:327)
Nov 29 12:02:05 at
com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:275)
Nov 29 12:02:05 at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)
Nov 29 12:02:05 ... 59 more
Nov 29 12:02:05
Nov 29 12:02:06 [INFO]
Nov 29 12:02:06 [INFO] Results:
Nov 29 12:02:06 [INFO]
Nov 29 12:02:06 [ERROR] Errors:
Nov 29 12:02:06 [ERROR] RMQSourceITCase.setUp:96->getRMQConnection:201 »
PossibleAuthenticationFailure
Nov 29 12:02:06 [INFO]
Nov 29 12:02:06 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0 {code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27205=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=13941]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] lindong28 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
lindong28 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758920274
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink-ml] lindong28 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
lindong28 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758920274
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.base.DoubleComparator;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag> MODEL_OUTPUT =
+new OutputTag>("MODEL_OUTPUT") {};
+
+public LogisticRegression() {
+ParamUtils.initializeMapWithDefaultValues(this.paramMap, this);
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+}
+
+public static LogisticRegression load(StreamExecutionEnvironment env,
String path)
+throws IOException {
+return
[GitHub] [flink-ml] lindong28 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
lindong28 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r758946526
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.ml.common.utils.ComparatorAdapter;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Collects distinct values in a bounded data stream. The parallelism of
the output stream is 1.
+ *
+ * @param The class type of the input data stream.
+ * @param input The bounded input data stream.
+ * @return The result data stream that contains all the distinct values.
+ */
+public static DataStream distinct(DataStream input) {
+return input.transform(
+"distinctInEachPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(input.getParallelism())
+.transform(
+"distinctInFinalPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(1);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Sorts the elements in each partition of the input bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param comparator The comparator used to sort the elements.
+ * @param The class type of input
[GitHub] [flink] flinkbot edited a comment on pull request #17950: [FLINK-25091] change Official website document "FileSink" orc compression attribute reference error
flinkbot edited a comment on pull request #17950: URL: https://github.com/apache/flink/pull/17950#issuecomment-982256290 ## CI report: * e8633bd19b6636319a85835ad9949754d93a9dbd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27247) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shenzhu commented on a change in pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
shenzhu commented on a change in pull request #17919:
URL: https://github.com/apache/flink/pull/17919#discussion_r758942835
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/StringToBinaryCastRule.java
##
@@ -45,6 +52,25 @@ public String generateExpression(
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
-return methodCall(inputTerm, "toBytes");
+int inputLength = Integer.MAX_VALUE;
Review comment:
Got it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-23390) FlinkKafkaInternalProducerITCase.testResumeTransactionAfterClosed
[
https://issues.apache.org/jira/browse/FLINK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450859#comment-17450859
]
Yun Gao commented on FLINK-23390:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27195=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=35051]
> FlinkKafkaInternalProducerITCase.testResumeTransactionAfterClosed
> -
>
> Key: FLINK-23390
> URL: https://issues.apache.org/jira/browse/FLINK-23390
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20454=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=f266c805-9429-58ed-2f9e-482e7b82f58b=6914
> {code}
> Jul 14 22:01:05 [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 49 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase
> Jul 14 22:01:05 [ERROR]
> testResumeTransactionAfterClosed(org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase)
> Time elapsed: 5.271 s <<< ERROR!
> Jul 14 22:01:05 java.lang.Exception: Unexpected exception,
> expected but was
> Jul 14 22:01:05 at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:30)
> Jul 14 22:01:05 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Jul 14 22:01:05 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Jul 14 22:01:05 at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> Jul 14 22:01:05 at java.base/java.lang.Thread.run(Thread.java:834)
> Jul 14 22:01:05 Caused by: java.lang.AssertionError: The message should have
> been successfully sent expected null, but
> was:
> Jul 14 22:01:05 at org.junit.Assert.fail(Assert.java:89)
> Jul 14 22:01:05 at org.junit.Assert.failNotNull(Assert.java:756)
> Jul 14 22:01:05 at org.junit.Assert.assertNull(Assert.java:738)
> Jul 14 22:01:05 at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.getClosedProducer(FlinkKafkaInternalProducerITCase.java:228)
> Jul 14 22:01:05 at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaInternalProducerITCase.testResumeTransactionAfterClosed(FlinkKafkaInternalProducerITCase.java:184)
> Jul 14 22:01:05 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Jul 14 22:01:05 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 14 22:01:05 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 14 22:01:05 at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> Jul 14 22:01:05 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 14 22:01:05 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 14 22:01:05 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 14 22:01:05 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 14 22:01:05 at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
> Jul 14 22:01:05 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25099) flink on yarn Accessing two HDFS Clusters
[
https://issues.apache.org/jira/browse/FLINK-25099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450853#comment-17450853
]
Junfan Zhang commented on FLINK-25099:
--
Could you make the log level debug and attach more logs ? [~libra_816]
> flink on yarn Accessing two HDFS Clusters
> -
>
> Key: FLINK-25099
> URL: https://issues.apache.org/jira/browse/FLINK-25099
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / State Backends
>Affects Versions: 1.13.3
> Environment: flink : 1.13.3
> hadoop : 3.3.0
>Reporter: chenqizhu
>Priority: Major
>
> Flink version 1.13 supports configuration of Hadoop properties in
> flink-conf.yaml via flink.hadoop.*. There is A requirement to write
> checkpoint to HDFS with SSDS (called cluster B) to speed checkpoint writing,
> but this HDFS cluster is not the default HDFS in the flink client (called
> cluster A by default). Yaml is configured with nameservices for cluster A and
> cluster B, which is similar to HDFS federated mode.
> The configuration is as follows:
>
> {code:java}
> flink.hadoop.dfs.nameservices: ACluster,BCluster
> flink.hadoop.fs.defaultFS: hdfs://BCluster
> flink.hadoop.dfs.ha.namenodes.ACluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn1: 10.:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn1: 10.:50070
> flink.hadoop.dfs.namenode.rpc-address.ACluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.ACluster.nn2: 10.xx:50070
> flink.hadoop.dfs.client.failover.proxy.provider.ACluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.BCluster: nn1,nn2
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn1: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn1: 10.xx:50070
> flink.hadoop.dfs.namenode.rpc-address.BCluster.nn2: 10.xx:9000
> flink.hadoop.dfs.namenode.http-address.BCluster.nn2: 10.x:50070
> flink.hadoop.dfs.client.failover.proxy.provider.BCluster:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> {code}
>
> However, an error occurred during the startup of the job, which is reported
> as follows:
> (change configuration items to A flink local client default HDFS cluster, the
> operation can be normal boot: flink.hadoop.fs.DefaultFS: hdfs: / / ACluster)
> {noformat}
> Caused by: BCluster
> java.net.UnknownHostException: BCluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:448)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:374)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:270)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:68)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:415)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:412)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:412)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:247)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:240)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:228)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
> Is there a solution to the above problems? The pain
