[GitHub] [flink] twalthr commented on a change in pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
twalthr commented on a change in pull request #14111: URL: https://github.com/apache/flink/pull/14111#discussion_r525878851 ## File path: docs/dev/table/sourceSinks.zh.md ## @@ -193,6 +192,149 @@ for more information. The runtime implementation of a `LookupTableSource` is a `TableFunction` or `AsyncTableFunction`. The function will be called with values for the given lookup keys during runtime. + Defining a Dynamic Table Source with Projection Push-Down Review comment: How about we just describe the interfaces in 1 or 2 sentences. Maybe in a table. And link to the corresponding class on Github. It is very difficult to keep docs and interfaces in sync. We should avoid code duplication. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18121) Support creating Docker image from local Flink distribution
[ https://issues.apache.org/jira/browse/FLINK-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234342#comment-17234342 ] Guowei Ma commented on FLINK-18121: --- +1 from the local file system. When I test https://issues.apache.org/jira/browse/FLINK-20206 I find I have to connect docker hub or build a http server myself. For me it is a little inconvenient. For example sometime the connection would be lost or timeout. > Support creating Docker image from local Flink distribution > --- > > Key: FLINK-18121 > URL: https://issues.apache.org/jira/browse/FLINK-18121 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Affects Versions: docker-1.11.0.0 >Reporter: Till Rohrmann >Priority: Major > > Currently, > https://github.com/apache/flink-docker/blob/dev-master/Dockerfile-debian.template > only supports to create a Docker image from a Flink distribution which is > hosted on a web server. I think it would be helpful if we could also create a > Docker image from a Flink distribution which is stored on one's local file > system. That way, one would not have to upload the file or start a web server > for serving it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
flinkbot edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-725222918 ## CI report: * cdc2dc09930a7064c55073381f0be79be0599d71 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9726) * 4a912e52f9520467dcdc7eeb3a19937c331e99e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #14055: [FLINK-19816] Make job state cleanup dependent on final job result
tillrohrmann closed pull request #14055: URL: https://github.com/apache/flink/pull/14055 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20208) Remove outdated in-progress files in StreamingFileSink
Alexander Trushev created FLINK-20208:
-
Summary: Remove outdated in-progress files in StreamingFileSink
Key: FLINK-20208
URL: https://issues.apache.org/jira/browse/FLINK-20208
Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem
Affects Versions: 1.11.2
Reporter: Alexander Trushev
Assume a job has StreamingFileSink with OnCheckpointRollingPolicy
In the case:
# Acknowledged checkpoint
# Event is written to new .part-X-Y.UUID1
# Job failure
# Job recovery from the checkpoint
# Event is written to new .part-X-Y.UUID2
we have the outdated part file .part-X-Y.UUID1. Where X - subtask index, Y -
part counter.
*Proposal*
Add method
{code:java}
boolean shouldRemoveOutdatedParts()
{code}
to RollingPolicy.
Add configurable parameter to OnCheckpointRollingPolicy and to
DefaultRollingPolicy that will be returned by shouldRemoveOutdatedParts() (by
default false)
We can remove such outdated part files by the next algorithm while restoring
job from a checkpoint
# After buckets state initializing check shouldRemoveOutdatedParts. If true
then (2)
# For each inactive bucket scan bucket directory
# If three conditions are true then remove part file:
part filename contains "inprogress";
subtask index from filename equals to current subtask index;
part counter from filename more than or equals to current max part counter.
I propose to remove outdated files, because the similar proposal to overwrite
outdated files has not been implemented
[https://issues.apache.org/jira/browse/FLINK-6|https://vk.com/away.php?to=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-6_key=]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14113: [FLINK-20202][python] Add the Check of Unsupported Result Type in Pandas UDAF
flinkbot commented on pull request #14113: URL: https://github.com/apache/flink/pull/14113#issuecomment-729501708 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 9bd25bd7728f8097a45b11b0a871a13277c92c9f (Wed Nov 18 07:50:07 UTC 2020) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20202).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20202) Add the Check of Unsupported Result Type in Pandas UDAF
[
https://issues.apache.org/jira/browse/FLINK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20202:
---
Labels: pull-request-available (was: )
> Add the Check of Unsupported Result Type in Pandas UDAF
> ---
>
> Key: FLINK-20202
> URL: https://issues.apache.org/jira/browse/FLINK-20202
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Reporter: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently the return type of Pandas UDAF should be a primitive data type, and
> the returned scalar can be either a python primitive type, e.g., {{int}} or
> {{float}} or a numpy data type, e.g., {{numpy.int64}} or {{numpy.float64}}.
> {{Any}} should ideally be a specific scalar type accordingly. We will add
> related DataType check and throw a more readable exception for unsupported
> DataTypes. What's more, we will add related notes in docs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20165) YARNSessionFIFOITCase.checkForProhibitedLogContents: Error occurred during initialization of boot layer java.lang.IllegalStateException: Module system already initiali
[
https://issues.apache.org/jira/browse/FLINK-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234338#comment-17234338
]
Robert Metzger commented on FLINK-20165:
I believe this is a JVM bug. Searching for "Module system already initialized"
reveals very few search results. Basically just the JVM source code, and this
ticket. We might be the first to hit this issue.
As a mitigation, I propose to update the JDK we are using in our CI docker
image. If the issue occurs in the latest JDK as well, I'll reach out to them.
> YARNSessionFIFOITCase.checkForProhibitedLogContents: Error occurred during
> initialization of boot layer java.lang.IllegalStateException: Module system
> already initialized
> --
>
> Key: FLINK-20165
> URL: https://issues.apache.org/jira/browse/FLINK-20165
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.3
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9597=logs=298e20ef-7951-5965-0e79-ea664ddc435e=8560c56f-9ec1-5c40-4ff5-9d3e882d
> {code}
> 2020-11-15T22:42:03.3053212Z 22:42:03,303 [ Time-limited test] INFO
> org.apache.flink.yarn.YARNSessionFIFOITCase [] - Finished
> testDetachedMode()
> 2020-11-15T22:42:37.9020133Z [ERROR] Tests run: 5, Failures: 2, Errors: 0,
> Skipped: 2, Time elapsed: 67.485 s <<< FAILURE! - in
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase
> 2020-11-15T22:42:37.9022015Z [ERROR]
> testDetachedMode(org.apache.flink.yarn.YARNSessionFIFOSecuredITCase) Time
> elapsed: 12.841 s <<< FAILURE!
> 2020-11-15T22:42:37.9023701Z java.lang.AssertionError:
> 2020-11-15T22:42:37.9025649Z Found a file
> /__w/2/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-logDir-nm-1_0/application_1605480087188_0002/container_1605480087188_0002_01_02/taskmanager.out
> with a prohibited string (one of [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]). Excerpts:
> 2020-11-15T22:42:37.9026730Z [
> 2020-11-15T22:42:37.9027080Z Error occurred during initialization of boot
> layer
> 2020-11-15T22:42:37.9027623Z java.lang.IllegalStateException: Module system
> already initialized
> 2020-11-15T22:42:37.9033278Z java.lang.IllegalStateException: Module system
> already initialized
> 2020-11-15T22:42:37.9033825Z ]
> 2020-11-15T22:42:37.9034291Z at org.junit.Assert.fail(Assert.java:88)
> 2020-11-15T22:42:37.9034971Z at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:479)
> 2020-11-15T22:42:37.9035814Z at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:83)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14113: [FLINK-20202][python] Add the Check of Unsupported Result Type in Pandas UDAF
HuangXingBo opened a new pull request #14113: URL: https://github.com/apache/flink/pull/14113 ## What is the purpose of the change *This pull request will add the check of unsupported result type in Pandas UDAF and related notes in docs* ## Brief change log - *Add check logic in `udf.py* - *Add notes in `vectorized_python_udfs.md` and `vectorized_python_udfs.zh.md`* ## Verifying this change This change added tests and can be verified as follows: - *Added UT `test_check_result_type` in `test_pandas_udaf.py`* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20102) Update HBase connector documentation for HBase 2.x supporting
[ https://issues.apache.org/jira/browse/FLINK-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20102. --- Resolution: Fixed Fixed in master (1.12.0): - 5f422ed5cb6eda2271c9cd13c212f84a884f6fbd - 823a14c4c99b27985406eeb4095ab882f64c07e3 > Update HBase connector documentation for HBase 2.x supporting > -- > > Key: FLINK-20102 > URL: https://issues.apache.org/jira/browse/FLINK-20102 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase, Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Assignee: hailong wang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Currently, the HBase connector page says it only supports HBase 1.4.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #14059: [FLINK-20102][docs][hbase] Update HBase connector documentation for H…
wuchong closed pull request #14059: URL: https://github.com/apache/flink/pull/14059 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19206) Add an ability to set ownerReference manually in Kubernetes
[
https://issues.apache.org/jira/browse/FLINK-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234331#comment-17234331
]
Yang Wang commented on FLINK-19206:
---
Since we set the owner reference to the JobManager deployment, not the
JobManager pod, then I think it also makes sense for the session cluster mode.
Assume that when the user explicitly set the owner reference for the JobManager
deployment, it means that he/she wants all the resources to be deleted
automatically.
> Add an ability to set ownerReference manually in Kubernetes
> ---
>
> Key: FLINK-19206
> URL: https://issues.apache.org/jira/browse/FLINK-19206
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: Mike Kaplinskiy
>Priority: Minor
>
> The current Kubernetes deployment creates a service that is the
> ownerReference of all the sub-objects (the JM & TM deployments & the rest
> service). However, something presumably has to start the cluster in the first
> place. If you are using a job cluster, that can be something like a
> kubernetes Job, a CronJob or a tool like Airflow. Unfortunately any failures
> in the Flink job can cause retries from these higher-level primitives, which
> can yield a lot of "stale clusters" that aren't GCed.
> The proposal here is to add a configuration option to set the ownerReference
> of the Flink Service. This way the service (and by proxy, all the cluster
> components) get deleted when the "parent" decides - including if the parent
> is itself a Kubernetes pod. For reference, Spark does something similar via
> {{spark.kubernetes.driver.pod.name}} (documented at
> [https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-executor-pod-garbage-collection]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14112: [FLINK-20207][python] improve the error message printed when submitting the pyflink jobs via 'flink run'.
flinkbot commented on pull request #14112: URL: https://github.com/apache/flink/pull/14112#issuecomment-729496292 ## CI report: * 61e1b28fc13ad5ddc3b7898f4f79d1905c6e145b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
flinkbot edited a comment on pull request #14111: URL: https://github.com/apache/flink/pull/14111#issuecomment-729488072 ## CI report: * 94088ed1b38fec47e5e43d7d399021003a1b7482 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
flinkbot edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-725222918 ## CI report: * cdc2dc09930a7064c55073381f0be79be0599d71 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9726) * 4a912e52f9520467dcdc7eeb3a19937c331e99e3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
AHeise commented on a change in pull request #14052:
URL: https://github.com/apache/flink/pull/14052#discussion_r525867148
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
##
@@ -98,14 +100,11 @@
private final BufferManager bufferManager;
- /**
-* Indicates the last overtaken sequence number by the most recent
{@link CheckpointBarrier}
-* before task thread started checkpoint, or {@code null} if {@link
CheckpointBarrier} hasn't
-* arrived yet.
-*/
@GuardedBy("receivedBuffers")
- @Nullable
- private Integer lastOvertakenSequenceNumber = null;
+ private int lastBarrierSequenceNumber = NONE;
Review comment:
As a reviewer of this PR, I'm more confused by separation. I think Piotr
is also not a big fan of touching the same code in multiple commits. The
question is which confusion you want to avoid? Is it about code ownership?
Then, I wouldn't mind keeping it separate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-docker] Myasuka commented on a change in pull request #43: [FLINK-19125] Adopt Jemalloc as default memory allocator to avoid memory leak
Myasuka commented on a change in pull request #43:
URL: https://github.com/apache/flink-docker/pull/43#discussion_r525867175
##
File path: docker-entrypoint.sh
##
@@ -93,21 +94,49 @@ prepare_job_manager_start() {
envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp"
"${CONF_FILE}"
}
+disable_jemalloc_env() {
+ if [ "$1" = ${COMMAND_DISABLE_JEMALLOC} ]; then
+ echo "Disable Jemalloc as the memory allocator"
+ return 0
+ else
+ echo "Enable Jemalloc as the memory allocator via appending env variable
LD_PRELOAD with /usr/lib/x86_64-linux-gnu/libjemalloc.so"
+ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
+ return 1
+ fi
+}
+
if [ "$1" = "help" ]; then
-echo "Usage: $(basename "$0")
(jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_NATIVE_KUBERNETES}|${COMMAND_HISTORY_SERVER}|help)"
+printf "Usage: $(basename "$0")
(jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_NATIVE_KUBERNETES}|${COMMAND_HISTORY_SERVER})
[${COMMAND_DISABLE_JEMALLOC}]\n"
+printf "Or $(basename "$0") help\n\n"
+printf "By default, Flink image adopts jemalloc as default memory
allocator and will disable jemalloc if option '${COMMAND_DISABLE_JEMALLOC}'
given.\n"
exit 0
elif [ "$1" = "jobmanager" ]; then
shift 1
+disable_jemalloc_env $@
+disabled_jemalloc="$?"
+if [ ${disabled_jemalloc} = 0 ]; then
+ shift 1
+fi
Review comment:
I have refactored the function and update the array of arguments within
the new `disable_jemalloc_env` function and then passed to next commands.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
AHeise commented on a change in pull request #14052:
URL: https://github.com/apache/flink/pull/14052#discussion_r525866375
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
##
@@ -1125,70 +1127,137 @@ public void testUnblockReleasedChannel() throws
Exception {
@Test
public void testPrioritySequenceNumbers() throws Exception {
- final NetworkBufferPool networkBufferPool = new
NetworkBufferPool(4, 4096);
- SingleInputGate inputGate = new SingleInputGateBuilder()
-
.setChannelFactory(InputChannelBuilder::buildRemoteChannel)
-
.setBufferPoolFactory(networkBufferPool.createBufferPool(1, 4))
- .setSegmentProvider(networkBufferPool)
- .build();
- final RemoteInputChannel channel = (RemoteInputChannel)
inputGate.getChannel(0);
- inputGate.setup();
- inputGate.requestPartitions();
+ final RemoteInputChannel channel =
buildInputGateAndGetChannel();
+ sendBuffersAndBarrier(channel, 0);
Review comment:
I'd like to see them inlined. I aim to always have a test structured in
the _given_, _when_, _then_ structure and I think with this change, you managed
to get exactly that (much better than it was before).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14112: [FLINK-20207][python] improve the error message printed when submitting the pyflink jobs via 'flink run'.
flinkbot commented on pull request #14112: URL: https://github.com/apache/flink/pull/14112#issuecomment-729492864 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 61e1b28fc13ad5ddc3b7898f4f79d1905c6e145b (Wed Nov 18 07:30:52 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20207).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
AHeise commented on a change in pull request #14052:
URL: https://github.com/apache/flink/pull/14052#discussion_r525865289
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
##
@@ -506,43 +514,75 @@ public void checkpointStarted(CheckpointBarrier barrier) {
synchronized (receivedBuffers) {
channelStatePersister.startPersisting(
barrier.getId(),
- getInflightBuffers(numBuffersOvertaken == ALL ?
receivedBuffers.getNumUnprioritizedElements() : numBuffersOvertaken));
+ getInflightBuffers());
}
}
public void checkpointStopped(long checkpointId) {
synchronized (receivedBuffers) {
channelStatePersister.stopPersisting(checkpointId);
- numBuffersOvertaken = ALL;
+ lastOvertakenSequenceNumber = null;
+ }
+ }
+
+ @VisibleForTesting
+ List getInflightBuffers() {
+ synchronized (receivedBuffers) {
+ return getInflightBuffersUnsafe();
}
}
/**
* Returns a list of buffers, checking the first n non-priority
buffers, and skipping all events.
*/
- private List getInflightBuffers(int numBuffers) {
+ private List getInflightBuffersUnsafe() {
assert Thread.holdsLock(receivedBuffers);
- if (numBuffers == 0) {
- return Collections.emptyList();
- }
-
- final List inflightBuffers = new
ArrayList<>(numBuffers);
+ final List inflightBuffers = new ArrayList<>();
Iterator iterator = receivedBuffers.iterator();
// skip all priority events (only buffers are stored anyways)
Iterators.advance(iterator,
receivedBuffers.getNumPriorityElements());
- // spill number of overtaken buffers or all of them if barrier
has not been seen yet
- for (int pos = 0; pos < numBuffers; pos++) {
- Buffer buffer = iterator.next().buffer;
- if (buffer.isBuffer()) {
- inflightBuffers.add(buffer.retainBuffer());
+ while (iterator.hasNext()) {
+ SequenceBuffer sequenceBuffer = iterator.next();
+ if (sequenceBuffer.buffer.isBuffer() &&
shouldBeSpilled(sequenceBuffer.sequenceNumber)) {
+
inflightBuffers.add(sequenceBuffer.buffer.retainBuffer());
}
}
+ lastOvertakenSequenceNumber = null;
+
return inflightBuffers;
}
+ /**
+* @return if given {@param sequenceNumber} should be spilled given
{@link #lastOvertakenSequenceNumber}.
+* We might not have yet received {@link CheckpointBarrier} and we
might need to spill everything.
+* If we have already received it, there is a bit nasty corner case of
{@link SequenceBuffer#sequenceNumber}
+* overflowing that needs to be handled as well.
+*/
+ private boolean shouldBeSpilled(int sequenceNumber) {
+ if (lastOvertakenSequenceNumber == null) {
+ return true;
+ }
+ checkState(
+ receivedBuffers.size() < Integer.MAX_VALUE / 2,
+ "Too many buffers for sequenceNumber overflow detection
code to work correctly");
+
+ boolean possibleOverflowAfterOvertaking = Integer.MAX_VALUE / 2
< lastOvertakenSequenceNumber;
+ boolean possibleOverflowBeforeOvertaking =
lastOvertakenSequenceNumber < -Integer.MAX_VALUE / 2;
+
+ if (possibleOverflowAfterOvertaking) {
+ return sequenceNumber < lastOvertakenSequenceNumber &&
sequenceNumber > 0;
+ }
+ else if (possibleOverflowBeforeOvertaking) {
+ return sequenceNumber < lastOvertakenSequenceNumber ||
sequenceNumber > 0;
+ }
+ else {
+ return sequenceNumber < lastOvertakenSequenceNumber;
+ }
Review comment:
If this is the only place, we can leave as is. We just need to be aware
of the code in other places where we compare sequence numbers (other than the
strict increment of `RemoteInputChannel`) and pull out on demand.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-20207) Improve the error message printed when submitting the pyflink jobs via 'flink run'
[
https://issues.apache.org/jira/browse/FLINK-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20207:
---
Labels: pull-request-available (was: )
> Improve the error message printed when submitting the pyflink jobs via 'flink
> run'
> --
>
> Key: FLINK-20207
> URL: https://issues.apache.org/jira/browse/FLINK-20207
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Reporter: Wei Zhong
>Priority: Major
> Labels: pull-request-available
>
> Sometimes the Java stack traces were swallowed when submitting the pyflink
> jobs via "flink run", e.g.:
> File "/home/cdh272705/poc/T24_parse.py", line 179, in
> from_kafka_to_oracle_demo
>
> main_table.execute_insert("sink")#.get_job_client().get_job_execution_result().result()
> File
> "/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/table/table.py",
> line 783, in execute_insert
> return TableResult(self._j_table.executeInsert(table_path, overwrite))
> File
> "/home/cdh272705/.local/lib/python3.6/site-packages/py4j/java_gateway.py",
> line 1286, in __call__
> answer, self.gateway_client, self.target_id, self.name)
> File
> "/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/util/exceptions.py",
> line 154, in deco
> raise exception_mapping[exception](s.split(': ', 1)[1], stack_trace)
> pyflink.util.exceptions.TableException: 'Failed to execute sql'
>
> The Java stack traces under the TableException were swallowed, which makes
> the troubleshooting difficult.
> We need to improve the error reporting logic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] WeiZhong94 opened a new pull request #14112: [FLINK-20207][python] improve the error message printed when submitting the pyflink jobs via 'flink run'.
WeiZhong94 opened a new pull request #14112: URL: https://github.com/apache/flink/pull/14112 ## What is the purpose of the change *This pull request improves the error reporting logic when submitting the pyflink jobs via 'flink run'.* ## Brief change log - *Print the Java stack trace when printing the Java exceptions.* - *Print the Python process output both on stdout and log file.* ## Verifying this change This change is already covered by existing tests, such as *PyFlink e2e test*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
AHeise commented on a change in pull request #14052:
URL: https://github.com/apache/flink/pull/14052#discussion_r525864426
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
##
@@ -506,43 +514,75 @@ public void checkpointStarted(CheckpointBarrier barrier) {
synchronized (receivedBuffers) {
channelStatePersister.startPersisting(
barrier.getId(),
- getInflightBuffers(numBuffersOvertaken == ALL ?
receivedBuffers.getNumUnprioritizedElements() : numBuffersOvertaken));
+ getInflightBuffers());
}
}
public void checkpointStopped(long checkpointId) {
synchronized (receivedBuffers) {
channelStatePersister.stopPersisting(checkpointId);
- numBuffersOvertaken = ALL;
+ lastOvertakenSequenceNumber = null;
+ }
+ }
+
+ @VisibleForTesting
+ List getInflightBuffers() {
+ synchronized (receivedBuffers) {
+ return getInflightBuffersUnsafe();
}
}
/**
* Returns a list of buffers, checking the first n non-priority
buffers, and skipping all events.
*/
- private List getInflightBuffers(int numBuffers) {
+ private List getInflightBuffersUnsafe() {
assert Thread.holdsLock(receivedBuffers);
- if (numBuffers == 0) {
- return Collections.emptyList();
- }
-
- final List inflightBuffers = new
ArrayList<>(numBuffers);
+ final List inflightBuffers = new ArrayList<>();
Iterator iterator = receivedBuffers.iterator();
// skip all priority events (only buffers are stored anyways)
Iterators.advance(iterator,
receivedBuffers.getNumPriorityElements());
- // spill number of overtaken buffers or all of them if barrier
has not been seen yet
- for (int pos = 0; pos < numBuffers; pos++) {
- Buffer buffer = iterator.next().buffer;
- if (buffer.isBuffer()) {
- inflightBuffers.add(buffer.retainBuffer());
+ while (iterator.hasNext()) {
+ SequenceBuffer sequenceBuffer = iterator.next();
+ if (sequenceBuffer.buffer.isBuffer() &&
shouldBeSpilled(sequenceBuffer.sequenceNumber)) {
Review comment:
Is there a scenario where `shouldBeSpilled` returns [`true`, `false`,
`true`] for any given buffer sequence? I don't see it happening even with
overflow.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-20107) Issues with setting InputChannel.statePersister
[ https://issues.apache.org/jira/browse/FLINK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-20107. - Resolution: Fixed > Issues with setting InputChannel.statePersister > > > Key: FLINK-20107 > URL: https://issues.apache.org/jira/browse/FLINK-20107 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing, Runtime / Network >Affects Versions: 1.12.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > ChannelStatePersister is a non-final field in InputChannel. > The need for injection is caused by different places, where network- and > state- related objects are created (task executor vs stream task). > It is set on remote and local channels upon requesting partitions, after > converting unknown or recovered channel. > Issues > # Not set on RemoteInputChannel when converting from UnknownChannel > # No visibility guarantee: written by task thread without any (explicit) > synchronization, read by network thread (checkForBarrier) > I see that "final" channels (both remote and local) are created only when the > writer is known (partitions requested). > So we can just make it final in "final" channels and pass from > recovered/unknown. For that, > * need to add to UnknownChannel > * no need to make it volatile/guarded because in "non-final" channels it's > accessed only by the task thread; and in "final" channels it will be final -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20107) Issues with setting InputChannel.statePersister
[ https://issues.apache.org/jira/browse/FLINK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234326#comment-17234326 ] Arvid Heise commented on FLINK-20107: - Merged into master as c7b2e12af37905e664685e838a6ad1a92a6b4ada. > Issues with setting InputChannel.statePersister > > > Key: FLINK-20107 > URL: https://issues.apache.org/jira/browse/FLINK-20107 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing, Runtime / Network >Affects Versions: 1.12.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > ChannelStatePersister is a non-final field in InputChannel. > The need for injection is caused by different places, where network- and > state- related objects are created (task executor vs stream task). > It is set on remote and local channels upon requesting partitions, after > converting unknown or recovered channel. > Issues > # Not set on RemoteInputChannel when converting from UnknownChannel > # No visibility guarantee: written by task thread without any (explicit) > synchronization, read by network thread (checkForBarrier) > I see that "final" channels (both remote and local) are created only when the > writer is known (partitions requested). > So we can just make it final in "final" channels and pass from > recovered/unknown. For that, > * need to add to UnknownChannel > * no need to make it volatile/guarded because in "non-final" channels it's > accessed only by the task thread; and in "final" channels it will be final -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #14051: [FLINK-20107][network] Make InputChannel.statePersister immutable
AHeise merged pull request #14051: URL: https://github.com/apache/flink/pull/14051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20207) Improve the error message printed when submitting the pyflink jobs via 'flink run'
Wei Zhong created FLINK-20207:
-
Summary: Improve the error message printed when submitting the
pyflink jobs via 'flink run'
Key: FLINK-20207
URL: https://issues.apache.org/jira/browse/FLINK-20207
Project: Flink
Issue Type: Improvement
Components: API / Python
Reporter: Wei Zhong
Sometimes the Java stack traces were swallowed when submitting the pyflink jobs
via "flink run", e.g.:
File "/home/cdh272705/poc/T24_parse.py", line 179, in from_kafka_to_oracle_demo
main_table.execute_insert("sink")#.get_job_client().get_job_execution_result().result()
File
"/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/table/table.py",
line 783, in execute_insert
return TableResult(self._j_table.executeInsert(table_path, overwrite))
File
"/home/cdh272705/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line
1286, in __call__
answer, self.gateway_client, self.target_id, self.name)
File
"/home/cdh272705/.local/lib/python3.6/site-packages/pyflink/util/exceptions.py",
line 154, in deco
raise exception_mapping[exception](s.split(': ', 1)[1], stack_trace)
pyflink.util.exceptions.TableException: 'Failed to execute sql'
The Java stack traces under the TableException were swallowed, which makes the
troubleshooting difficult.
We need to improve the error reporting logic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
flinkbot commented on pull request #14111: URL: https://github.com/apache/flink/pull/14111#issuecomment-729488072 ## CI report: * 94088ed1b38fec47e5e43d7d399021003a1b7482 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14074: [FLINK-20129][table ecosystem] Create a zh download page for all optional sql client components
flinkbot edited a comment on pull request #14074: URL: https://github.com/apache/flink/pull/14074#issuecomment-727311077 ## CI report: * 17635c6f6c3ebddef63bc22798968ea5bf7e UNKNOWN * 6a287c19e5e1838f72a332145068ae82cbf83af0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9731) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13998: [FLINK-20062][hive] ContinuousHiveSplitEnumerator should be lock-free
flinkbot edited a comment on pull request #13998: URL: https://github.com/apache/flink/pull/13998#issuecomment-724001789 ## CI report: * 6beffeeaa6311b7d58dca3eea2d876ce03bbd816 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9725) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19206) Add an ability to set ownerReference manually in Kubernetes
[
https://issues.apache.org/jira/browse/FLINK-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234322#comment-17234322
]
Mike Kaplinskiy commented on FLINK-19206:
-
Yep that’s exactly right. Setting an owner reference on the deployment would be
a perfect solution, but as you said this probably only makes sense for
application mode clusters.
> Add an ability to set ownerReference manually in Kubernetes
> ---
>
> Key: FLINK-19206
> URL: https://issues.apache.org/jira/browse/FLINK-19206
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: Mike Kaplinskiy
>Priority: Minor
>
> The current Kubernetes deployment creates a service that is the
> ownerReference of all the sub-objects (the JM & TM deployments & the rest
> service). However, something presumably has to start the cluster in the first
> place. If you are using a job cluster, that can be something like a
> kubernetes Job, a CronJob or a tool like Airflow. Unfortunately any failures
> in the Flink job can cause retries from these higher-level primitives, which
> can yield a lot of "stale clusters" that aren't GCed.
> The proposal here is to add a configuration option to set the ownerReference
> of the Flink Service. This way the service (and by proxy, all the cluster
> components) get deleted when the "parent" decides - including if the parent
> is itself a Kubernetes pod. For reference, Spark does something similar via
> {{spark.kubernetes.driver.pod.name}} (documented at
> [https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-executor-pod-garbage-collection]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19585) UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint: "Not all required tasks are currently running."
[
https://issues.apache.org/jira/browse/FLINK-19585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234320#comment-17234320
]
Arvid Heise commented on FLINK-19585:
-
Currently [~rmetzger] is investigating unexpected behavior of the mini cluster
that hinders the backport.
> UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint: "Not all
> required tasks are currently running."
>
>
> Key: FLINK-19585
> URL: https://issues.apache.org/jira/browse/FLINK-19585
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7419=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0
> {code}
> 2020-10-12T10:27:51.7667213Z [ERROR] Tests run: 4, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 13.146 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointCompatibilityITCase
> 2020-10-12T10:27:51.7675454Z [ERROR] test[type: SAVEPOINT, startAligned:
> false](org.apache.flink.test.checkpointing.UnalignedCheckpointCompatibilityITCase)
> Time elapsed: 2.168 s <<< ERROR!
> 2020-10-12T10:27:51.7676759Z java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Not all required
> tasks are currently running.
> 2020-10-12T10:27:51.7686572Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-10-12T10:27:51.7688239Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-10-12T10:27:51.7689543Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointCompatibilityITCase.runAndTakeSavepoint(UnalignedCheckpointCompatibilityITCase.java:113)
> 2020-10-12T10:27:51.7690681Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointCompatibilityITCase.test(UnalignedCheckpointCompatibilityITCase.java:97)
> 2020-10-12T10:27:51.7691513Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-12T10:27:51.7692182Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-12T10:27:51.7692964Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-12T10:27:51.7693655Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-12T10:27:51.7694489Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-12T10:27:51.7707103Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-12T10:27:51.7729199Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-12T10:27:51.7730097Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-12T10:27:51.7730833Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-10-12T10:27:51.7731500Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-12T10:27:51.7732086Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-12T10:27:51.7732781Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-12T10:27:51.7733563Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-12T10:27:51.7734735Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-12T10:27:51.7735400Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-12T10:27:51.7736075Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-12T10:27:51.7736757Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-12T10:27:51.7737432Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-12T10:27:51.7738081Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-12T10:27:51.7739008Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-12T10:27:51.7739583Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-12T10:27:51.7740173Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-12T10:27:51.7740800Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-12T10:27:51.7741470Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-12T10:27:51.7742150Z at
>
[GitHub] [flink] flinkbot edited a comment on pull request #12998: [FLINK-18731][table-planner-blink] Fix monotonicity logic of UNIX_TIMESTAMP & UUID functions
flinkbot edited a comment on pull request #12998: URL: https://github.com/apache/flink/pull/12998#issuecomment-664380377 ## CI report: * 9bca4ec897eeca4b204350866c564139b970603e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9724) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20206) Failed to start the session, but there is no clear prompt.
Guowei Ma created FLINK-20206: - Summary: Failed to start the session, but there is no clear prompt. Key: FLINK-20206 URL: https://issues.apache.org/jira/browse/FLINK-20206 Project: Flink Issue Type: Improvement Components: Deployment / Kubernetes Affects Versions: 1.12.0 Reporter: Guowei Ma Attachments: image-2020-11-18-15-12-13-530.png Use ./bin/kubernetes-session.sh to start a k8s session clustter. The log showes the session cluster successfully start but it not. Personally I prefer the yarn-session way, which could make me have a clear expectation. So I would like to propose that Flink could give more detail information about whether session cluster create success or not. !image-2020-11-18-15-12-13-530.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19206) Add an ability to set ownerReference manually in Kubernetes
[
https://issues.apache.org/jira/browse/FLINK-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234317#comment-17234317
]
Yang Wang commented on FLINK-19206:
---
[~mikekap] Thanks for the detailed explanation. I think I get your point.
Currently, all the created K8s resources(e.g. Service, ConfigMap, TaskManager
Pod) have been set the owner reference to the JobManager deployment. That means
when the deployment is deleted, all other resources will be deleted
automatically.
So your case will simply turn into "support to set owner reference for the
JobManager deployment". Right? You could set the owner reference to batch job,
cronjob or any other resource in K8s.
> Add an ability to set ownerReference manually in Kubernetes
> ---
>
> Key: FLINK-19206
> URL: https://issues.apache.org/jira/browse/FLINK-19206
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: Mike Kaplinskiy
>Priority: Minor
>
> The current Kubernetes deployment creates a service that is the
> ownerReference of all the sub-objects (the JM & TM deployments & the rest
> service). However, something presumably has to start the cluster in the first
> place. If you are using a job cluster, that can be something like a
> kubernetes Job, a CronJob or a tool like Airflow. Unfortunately any failures
> in the Flink job can cause retries from these higher-level primitives, which
> can yield a lot of "stale clusters" that aren't GCed.
> The proposal here is to add a configuration option to set the ownerReference
> of the Flink Service. This way the service (and by proxy, all the cluster
> components) get deleted when the "parent" decides - including if the parent
> is itself a Kubernetes pod. For reference, Spark does something similar via
> {{spark.kubernetes.driver.pod.name}} (documented at
> [https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-executor-pod-garbage-collection]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20205) CDC source shouldn't send UPDATE_BEFORE messages if the downstream doesn't need it
[
https://issues.apache.org/jira/browse/FLINK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-20205:
---
Assignee: Leonard Xu
> CDC source shouldn't send UPDATE_BEFORE messages if the downstream doesn't
> need it
> --
>
> Key: FLINK-20205
> URL: https://issues.apache.org/jira/browse/FLINK-20205
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Major
> Fix For: 1.12.0
>
>
> Currently, the CDC source will always generate UPDATE_BEFORE and UPDATE_AFTER
> messages for update. However, many downstream operators don't need the
> UDPATE_BEFORE, e.g. temporal join and upsert sink. Currently,
> {{debezium-cdc-source => upsert-kafka}} will generate lots of tombstone
> messages, which is quite not efficient.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
flinkbot commented on pull request #14111: URL: https://github.com/apache/flink/pull/14111#issuecomment-729477481 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 94088ed1b38fec47e5e43d7d399021003a1b7482 (Wed Nov 18 06:59:43 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20191) Add document for FLIP-95 ability interfaces
[ https://issues.apache.org/jira/browse/FLINK-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20191: --- Labels: pull-request-available (was: ) > Add document for FLIP-95 ability interfaces > --- > > Key: FLINK-20191 > URL: https://issues.apache.org/jira/browse/FLINK-20191 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka, Table SQL / API >Affects Versions: 1.12.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] fsk119 opened a new pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
fsk119 opened a new pull request #14111: URL: https://github.com/apache/flink/pull/14111 ## What is the purpose of the change Add document for the abilities interface. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20205) CDC source shouldn't send UPDATE_BEFORE messages if the downstream doesn't need it
Jark Wu created FLINK-20205:
---
Summary: CDC source shouldn't send UPDATE_BEFORE messages if the
downstream doesn't need it
Key: FLINK-20205
URL: https://issues.apache.org/jira/browse/FLINK-20205
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Jark Wu
Fix For: 1.12.0
Currently, the CDC source will always generate UPDATE_BEFORE and UPDATE_AFTER
messages for update. However, many downstream operators don't need the
UDPATE_BEFORE, e.g. temporal join and upsert sink. Currently,
{{debezium-cdc-source => upsert-kafka}} will generate lots of tombstone
messages, which is quite not efficient.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14028: [FLINK-20020][client] Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract
flinkbot edited a comment on pull request #14028: URL: https://github.com/apache/flink/pull/14028#issuecomment-725222918 ## CI report: * cdc2dc09930a7064c55073381f0be79be0599d71 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9726) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20204: Fix Version/s: 1.12.0 > Translate page 'upsert-kafka Connector' into Chinese > > > Key: FLINK-20204 > URL: https://issues.apache.org/jira/browse/FLINK-20204 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Kafka, Table SQL / > Ecosystem >Reporter: CaoZhen >Assignee: CaoZhen >Priority: Minor > Fix For: 1.12.0 > > > The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20204: Component/s: Table SQL / Ecosystem Connectors / Kafka chinese-translation > Translate page 'upsert-kafka Connector' into Chinese > > > Key: FLINK-20204 > URL: https://issues.apache.org/jira/browse/FLINK-20204 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Kafka, Table SQL / > Ecosystem >Reporter: CaoZhen >Assignee: CaoZhen >Priority: Minor > > The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20204: --- Assignee: CaoZhen > Translate page 'upsert-kafka Connector' into Chinese > > > Key: FLINK-20204 > URL: https://issues.apache.org/jira/browse/FLINK-20204 > Project: Flink > Issue Type: Sub-task >Reporter: CaoZhen >Assignee: CaoZhen >Priority: Minor > > The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…
flinkbot edited a comment on pull request #14110: URL: https://github.com/apache/flink/pull/14110#issuecomment-729457779 ## CI report: * f1865fb53a91603c17f3cebcd58105a3ab2ec234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9732) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…
flinkbot commented on pull request #14110: URL: https://github.com/apache/flink/pull/14110#issuecomment-729457779 ## CI report: * f1865fb53a91603c17f3cebcd58105a3ab2ec234 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14074: [FLINK-20129][table ecosystem] Create a zh download page for all optional sql client components
flinkbot edited a comment on pull request #14074: URL: https://github.com/apache/flink/pull/14074#issuecomment-727311077 ## CI report: * 17635c6f6c3ebddef63bc22798968ea5bf7e UNKNOWN * 503c48b2703bbcf02f32ead5aa3a2dae9f094560 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9591) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9593) * 6a287c19e5e1838f72a332145068ae82cbf83af0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9731) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20191) Add document for FLIP-95 ability interfaces
[ https://issues.apache.org/jira/browse/FLINK-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234287#comment-17234287 ] Shengkai Fang commented on FLINK-20191: --- Ok. I have rename the name of the jira to Add document for FLIP-95 ability interfaces. The modification is mainly about SupportsXXX interface. > Add document for FLIP-95 ability interfaces > --- > > Key: FLINK-20191 > URL: https://issues.apache.org/jira/browse/FLINK-20191 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka, Table SQL / API >Affects Versions: 1.12.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20113) Test K8s High Availability Service
[ https://issues.apache.org/jira/browse/FLINK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234284#comment-17234284 ] Guowei Ma commented on FLINK-20113: --- [~fly_in_gis] ok. > Test K8s High Availability Service > -- > > Key: FLINK-20113 > URL: https://issues.apache.org/jira/browse/FLINK-20113 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > > Added in https://issues.apache.org/jira/browse/FLINK-12884 > > [General Information about the Flink 1.12 release > testing|https://cwiki.apache.org/confluence/display/FLINK/1.12+Release+-+Community+Testing] > When testing a feature, consider the following aspects: > - Is the documentation easy to understand > - Are the error messages, log messages, APIs etc. easy to understand > - Is the feature working as expected under normal conditions > - Is the feature working / failing as expected with invalid input, induced > errors etc. > If you find a problem during testing, please file a ticket > (Priority=Critical; Fix Version = 1.12.0), and link it in this testing ticket. > During the testing, and once you are finished, please write a short summary > of all things you have tested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…
flinkbot commented on pull request #14110: URL: https://github.com/apache/flink/pull/14110#issuecomment-729449889 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f1865fb53a91603c17f3cebcd58105a3ab2ec234 (Wed Nov 18 06:00:53 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…
danny0405 commented on pull request #14110: URL: https://github.com/apache/flink/pull/14110#issuecomment-729449070 cc @wuchong , can you take a look for the patch ? Thanks in advance ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20200) SQL Hints are not supported in "Create View" syntax
[
https://issues.apache.org/jira/browse/FLINK-20200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20200:
---
Labels: pull-request-available (was: )
> SQL Hints are not supported in "Create View" syntax
>
>
> Key: FLINK-20200
> URL: https://issues.apache.org/jira/browse/FLINK-20200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.2
>Reporter: shizhengchao
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> I have aready set the config option `table.dynamic-table-options.enabled` to
> be true, but "SQL Hints" are not supported in View syntax. I got an error:
> {code:java}
> Exception in thread "main" java.lang.UnsupportedOperationException: class
> org.apache.calcite.sql.SqlSyntax$6: SPECIAL
> at org.apache.calcite.util.Util.needToImplement(Util.java:967)
> at org.apache.calcite.sql.SqlSyntax$6.unparse(SqlSyntax.java:116)
> at org.apache.calcite.sql.SqlOperator.unparse(SqlOperator.java:333)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:101)
> at
> org.apache.calcite.sql.SqlSelectOperator.unparse(SqlSelectOperator.java:176)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlSelect.unparse(SqlSelect.java:246)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:151)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:173)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:182)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.getQuotedSqlString(SqlToOperationConverter.java:784)
> at
> org.apache.flink.table.planner.utils.Expander$Expanded.substitute(Expander.java:169)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertViewQuery(SqlToOperationConverter.java:694)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertCreateView(SqlToOperationConverter.java:665)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:228)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:684)
> {code}
> The sql code is as follows：
> {code:java}
> drop table if exists SourceA;
> create table SourceA (
> idstring,
> name string
> ) with (
> 'connector' = 'kafka-0.11',
> 'topic' = 'MyTopic',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'Test',
> 'scan.startup.mode' = 'group-offsets',
> 'format' = 'csv'
> );
> drop table if exists print;
> create table print (
> idstring,
> name string
> ) with (
> 'connector' = 'print'
> );
> drop view if exists test_view;
> create view test_view as
> select
> *
> from SourceA /*+ OPTIONS('properties.group.id'='NewGroup') */;
> insert into print
> select * from test_view;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] danny0405 opened a new pull request #14110: [FLINK-20200][table-planner] SQL Hints are not supported in CREATE VI…
danny0405 opened a new pull request #14110: URL: https://github.com/apache/flink/pull/14110 …EW syntax ## What is the purpose of the change The view sql node `SqlTableRef` should keep the same after substitution of `SqlShuttle`. This PR fix it so that the view node with hints can be unparsed correctly. ## Brief change log - Copy and fix the `SqlTableRef` - Add a test case ## Verifying this change Added UT. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not documented This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #14074: [FLINK-20129][table ecosystem] Create a zh download page for all optional sql client components
wangxlong commented on pull request #14074: URL: https://github.com/apache/flink/pull/14074#issuecomment-729447882 Hi @wuchong , Could you help to have a look in your free time, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14077: [FLINK-20180][fs-connector][translation] Translate FileSink document into Chinese
flinkbot edited a comment on pull request #14077: URL: https://github.com/apache/flink/pull/14077#issuecomment-727773204 ## CI report: * ca8f49ba4692b44a600929f6e0c14b6da6a2ce85 UNKNOWN * 07560726c066d46c95ac580ce597ecc0d827e1ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9730) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14074: [FLINK-20129][table ecosystem] Create a zh download page for all optional sql client components
flinkbot edited a comment on pull request #14074: URL: https://github.com/apache/flink/pull/14074#issuecomment-727311077 ## CI report: * 17635c6f6c3ebddef63bc22798968ea5bf7e UNKNOWN * 503c48b2703bbcf02f32ead5aa3a2dae9f094560 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9591) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9593) * 6a287c19e5e1838f72a332145068ae82cbf83af0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20192) Externalized checkpoint references a checkpoint from a different job
[
https://issues.apache.org/jira/browse/FLINK-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234272#comment-17234272
]
Congxian Qiu commented on FLINK-20192:
--
[~Antti-Kaikkonen] you can create a savepoint and restore from it, the
savepoint does not need to reference any checkpoint files(the checkpoint files
can be deleted if you don't need to restore from it), and after 1.11, the
savepoint can also be relocated(FLINK-5763).
> Externalized checkpoint references a checkpoint from a different job
>
>
> Key: FLINK-20192
> URL: https://issues.apache.org/jira/browse/FLINK-20192
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Checkpointing
>Affects Versions: 1.11.2
>Reporter: Antti Kaikkonen
>Priority: Major
> Attachments: _metadata
>
>
> When I try to restore from an externalized checkpoint located at:
> +/home/anttkaik/flink/checkpoints/0fc94de8d94e123585b5baed6972dbe8/chk-12+ I
> get the following error:
>
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:204)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at
> java.lang.Thread.run(Thread.java:748) Caused by:
> org.apache.flink.util.FlinkException: Could not restore keyed state backend
> for FunctionGroupOperator_6b87a4870d0e21cecbbe271bd893cfcc_(2/4) from any of
> the 1 provided restore options. at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:144)
> ... 9 more Caused by:
> org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected
> exception. at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:329)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:535)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:301)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> ... 11 more Caused by: java.io.FileNotFoundException:
> /home/anttkaik/flink/checkpoints/01dbaf21d7c5e8f8eabd3602e086bb89/shared/0a3c0c1d-c924-4e6d-b6ad-463a75c9fce8
> (No such file or directory) at java.io.FileInputStream.open0(Native
> Method) at java.io.FileInputStream.open(FileInputStream.java:195) at
> java.io.FileInputStream.(FileInputStream.java:138) at
> org.apache.flink.core.fs.local.LocalDataInputStream.(LocalDataInputStream.java:50)
> at
> org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:143)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.open(SafetyNetWrapperFileSystem.java:85)
> at
> org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:126)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
> at
> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #14109: [FLINK-20197] Fix the current processing time issue in FileSink
flinkbot edited a comment on pull request #14109: URL: https://github.com/apache/flink/pull/14109#issuecomment-729395050 ## CI report: * 581a109520a1378e755f9e197c55d91838a8d736 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9729) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20184) update hive streaming read and temporal table documents
[ https://issues.apache.org/jira/browse/FLINK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-20184: Assignee: Leonard Xu > update hive streaming read and temporal table documents > --- > > Key: FLINK-20184 > URL: https://issues.apache.org/jira/browse/FLINK-20184 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Fix For: 1.12.0 > > > The hive streaming read and temporal table document has been out of style, we > need to update it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20203) Could not find any document about how to build a Flink image from local build.
[ https://issues.apache.org/jira/browse/FLINK-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234265#comment-17234265 ] Yang Wang commented on FLINK-20203: --- Maybe FLINK-18121 is related. > Could not find any document about how to build a Flink image from local build. > -- > > Key: FLINK-20203 > URL: https://issues.apache.org/jira/browse/FLINK-20203 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Priority: Critical > > If user wants to use or try some feature that does not include in the > "official" Flink image the user might need to build a docker image based on > his local build. But there is such > document([https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/docker.html)] > So I would like to propose that we might need to introduce some documentation > about how to build the image from local build. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14077: [FLINK-20180][fs-connector][translation] Translate FileSink document into Chinese
flinkbot edited a comment on pull request #14077: URL: https://github.com/apache/flink/pull/14077#issuecomment-727773204 ## CI report: * ca8f49ba4692b44a600929f6e0c14b6da6a2ce85 UNKNOWN * 58a2d8188e8ff838e0126cd2241cb89f27a4c9d2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9681) * 07560726c066d46c95ac580ce597ecc0d827e1ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9730) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234245#comment-17234245 ] CaoZhen commented on FLINK-20204: - Hi [~fsk119] ,[~jark] Can I translate this doc? > Translate page 'upsert-kafka Connector' into Chinese > > > Key: FLINK-20204 > URL: https://issues.apache.org/jira/browse/FLINK-20204 > Project: Flink > Issue Type: Sub-task >Reporter: CaoZhen >Priority: Minor > > The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
CaoZhen created FLINK-20204: --- Summary: Translate page 'upsert-kafka Connector' into Chinese Key: FLINK-20204 URL: https://issues.apache.org/jira/browse/FLINK-20204 Project: Flink Issue Type: Sub-task Reporter: CaoZhen The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14109: [FLINK-20197] Fix the current processing time issue in FileSink
flinkbot edited a comment on pull request #14109: URL: https://github.com/apache/flink/pull/14109#issuecomment-729395050 ## CI report: * 581a109520a1378e755f9e197c55d91838a8d736 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9729) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20191) Add document for FLIP-95 ability interfaces
[ https://issues.apache.org/jira/browse/FLINK-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang updated FLINK-20191: -- Summary: Add document for FLIP-95 ability interfaces (was: Add document for FLIP-95 ablities interface) > Add document for FLIP-95 ability interfaces > --- > > Key: FLINK-20191 > URL: https://issues.apache.org/jira/browse/FLINK-20191 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka, Table SQL / API >Affects Versions: 1.12.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20191) Add document for FLIP-95 ablities interface
[ https://issues.apache.org/jira/browse/FLINK-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang updated FLINK-20191: -- Summary: Add document for FLIP-95 ablities interface (was: Add document for FLIP-95 interface) > Add document for FLIP-95 ablities interface > --- > > Key: FLINK-20191 > URL: https://issues.apache.org/jira/browse/FLINK-20191 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka, Table SQL / API >Affects Versions: 1.12.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20203) Could not find any document about how to build a Flink image from local build.
Guowei Ma created FLINK-20203: - Summary: Could not find any document about how to build a Flink image from local build. Key: FLINK-20203 URL: https://issues.apache.org/jira/browse/FLINK-20203 Project: Flink Issue Type: Improvement Affects Versions: 1.12.0 Reporter: Guowei Ma If user wants to use or try some feature that does not include in the "official" Flink image the user might need to build a docker image based on his local build. But there is such document([https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/docker.html)] So I would like to propose that we might need to introduce some documentation about how to build the image from local build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14109: [FLINK-20197] Fix the current processing time issue in FileSink
flinkbot commented on pull request #14109: URL: https://github.com/apache/flink/pull/14109#issuecomment-729395050 ## CI report: * 581a109520a1378e755f9e197c55d91838a8d736 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14077: [FLINK-20180][fs-connector][translation] Translate FileSink document into Chinese
flinkbot edited a comment on pull request #14077: URL: https://github.com/apache/flink/pull/14077#issuecomment-727773204 ## CI report: * ca8f49ba4692b44a600929f6e0c14b6da6a2ce85 UNKNOWN * 58a2d8188e8ff838e0126cd2241cb89f27a4c9d2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9681) * 07560726c066d46c95ac580ce597ecc0d827e1ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14109: [FLINK-20197] Fix the current processing time issue in FileSink
flinkbot commented on pull request #14109: URL: https://github.com/apache/flink/pull/14109#issuecomment-729391957 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 581a109520a1378e755f9e197c55d91838a8d736 (Wed Nov 18 04:08:06 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20197).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20197) FileSink fails with UnsupportedOperationException when using default values
[
https://issues.apache.org/jira/browse/FLINK-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20197:
---
Labels: pull-request-available (was: )
> FileSink fails with UnsupportedOperationException when using default values
> ---
>
> Key: FLINK-20197
> URL: https://issues.apache.org/jira/browse/FLINK-20197
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When using the default values for the new {{FileSink}}, it fails with
> {code}
> Caused by: java.lang.UnsupportedOperationException: not supported
> at
> org.apache.flink.connector.file.sink.writer.FileWriter$BucketerContext.currentProcessingTime(FileWriter.java:296)
> ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:111)
> ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner.getBucketId(DateTimeBucketAssigner.java:55)
> ~[flink-file-sink-common-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:189)
> ~[flink-connector-files-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.operators.sink.AbstractSinkWriterOperator.processElement(AbstractSinkWriterOperator.java:80)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamGroupedReduceOperator.processElement(StreamGroupedReduceOperator.java:68)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538)
> ~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> {code}
> The problem seems that the
> {{FileWriter.BucketerContext.currentProcessingTime}} throws an
> {{UnsupportedOperationException}}. This feels not correct. A job for
> reproducing the problem can be found
>
[GitHub] [flink] gaoyunhaii opened a new pull request #14109: [FLINK-20197] Fix the current processing time issue in FileSink
gaoyunhaii opened a new pull request #14109: URL: https://github.com/apache/flink/pull/14109 ## What is the purpose of the change This PR fixes the bug that could not acquire processing time in FileSink. ## Brief change log - 581a109520a1378e755f9e197c55d91838a8d736 returns the current processing time when requested. ## Verifying this change - Adds new tests to verify the requested processing time is passed correctly. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14108: [FLINK-20183][python] Fix the default PYTHONPATH is overwritten in client side
flinkbot edited a comment on pull request #14108: URL: https://github.com/apache/flink/pull/14108#issuecomment-729368144 ## CI report: * 90d6ba76b1efb64f7dd9949f5bbbe027185636bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9728) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14108: [FLINK-20183][python] Fix the default PYTHONPATH is overwritten in client side
flinkbot commented on pull request #14108: URL: https://github.com/apache/flink/pull/14108#issuecomment-729368144 ## CI report: * 90d6ba76b1efb64f7dd9949f5bbbe027185636bf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20113) Test K8s High Availability Service
[ https://issues.apache.org/jira/browse/FLINK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234209#comment-17234209 ] Yang Wang commented on FLINK-20113: --- [~maguowei] Thanks for volunteering to do the K8s HA service test. Ping me if you need any help to building the image, run the session/application cluster with HA configured. > Test K8s High Availability Service > -- > > Key: FLINK-20113 > URL: https://issues.apache.org/jira/browse/FLINK-20113 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > > Added in https://issues.apache.org/jira/browse/FLINK-12884 > > [General Information about the Flink 1.12 release > testing|https://cwiki.apache.org/confluence/display/FLINK/1.12+Release+-+Community+Testing] > When testing a feature, consider the following aspects: > - Is the documentation easy to understand > - Are the error messages, log messages, APIs etc. easy to understand > - Is the feature working as expected under normal conditions > - Is the feature working / failing as expected with invalid input, induced > errors etc. > If you find a problem during testing, please file a ticket > (Priority=Critical; Fix Version = 1.12.0), and link it in this testing ticket. > During the testing, and once you are finished, please write a short summary > of all things you have tested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18640) Fix PostgresDialect doesn't quote the identifiers
[ https://issues.apache.org/jira/browse/FLINK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18640: Component/s: Table SQL / Ecosystem > Fix PostgresDialect doesn't quote the identifiers > - > > Key: FLINK-18640 > URL: https://issues.apache.org/jira/browse/FLINK-18640 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Affects Versions: 1.9.1, 1.10.1 >Reporter: 毛宗良 >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.10.3, 1.11.4 > > > Flink jdbc throw exceptions when read a postgresql table with scheam, like > "ods.t_test". BY debugging the source code, I found a bug about dealing the > table name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18654) Correct missleading documentation in "Partitioned Scan" section of JDBC connector
[ https://issues.apache.org/jira/browse/FLINK-18654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18654: Component/s: Table SQL / Ecosystem > Correct missleading documentation in "Partitioned Scan" section of JDBC > connector > - > > Key: FLINK-18654 > URL: https://issues.apache.org/jira/browse/FLINK-18654 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Documentation, Table SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: Jark Wu >Priority: Major > Fix For: 1.12.0, 1.11.4 > > > In > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/jdbc.html#partitioned-scan > > Notice that scan.partition.lower-bound and scan.partition.upper-bound are > > just used to decide the partition stride, not for filtering the rows in > > table. So all rows in the table will be partitioned and returned. > The "not for filtering the rows in table" is not correct, actually, if > partition bounds is defined, it only scans rows in the bound range. > Besides, maybe it would be better to add some practice suggestion, for > example, > "If it is a batch job, I think it also doable to get the max and min value > first before submitting the flink job." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20200) SQL Hints are not supported in "Create View" syntax
[
https://issues.apache.org/jira/browse/FLINK-20200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-20200:
---
Assignee: Danny Chen
> SQL Hints are not supported in "Create View" syntax
>
>
> Key: FLINK-20200
> URL: https://issues.apache.org/jira/browse/FLINK-20200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.2
>Reporter: shizhengchao
>Assignee: Danny Chen
>Priority: Major
>
> I have aready set the config option `table.dynamic-table-options.enabled` to
> be true, but "SQL Hints" are not supported in View syntax. I got an error:
> {code:java}
> Exception in thread "main" java.lang.UnsupportedOperationException: class
> org.apache.calcite.sql.SqlSyntax$6: SPECIAL
> at org.apache.calcite.util.Util.needToImplement(Util.java:967)
> at org.apache.calcite.sql.SqlSyntax$6.unparse(SqlSyntax.java:116)
> at org.apache.calcite.sql.SqlOperator.unparse(SqlOperator.java:333)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:101)
> at
> org.apache.calcite.sql.SqlSelectOperator.unparse(SqlSelectOperator.java:176)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlSelect.unparse(SqlSelect.java:246)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:151)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:173)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:182)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.getQuotedSqlString(SqlToOperationConverter.java:784)
> at
> org.apache.flink.table.planner.utils.Expander$Expanded.substitute(Expander.java:169)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertViewQuery(SqlToOperationConverter.java:694)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertCreateView(SqlToOperationConverter.java:665)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:228)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:684)
> {code}
> The sql code is as follows：
> {code:java}
> drop table if exists SourceA;
> create table SourceA (
> idstring,
> name string
> ) with (
> 'connector' = 'kafka-0.11',
> 'topic' = 'MyTopic',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'Test',
> 'scan.startup.mode' = 'group-offsets',
> 'format' = 'csv'
> );
> drop table if exists print;
> create table print (
> idstring,
> name string
> ) with (
> 'connector' = 'print'
> );
> drop view if exists test_view;
> create view test_view as
> select
> *
> from SourceA /*+ OPTIONS('properties.group.id'='NewGroup') */;
> insert into print
> select * from test_view;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20200) SQL Hints are not supported in "Create View" syntax
[
https://issues.apache.org/jira/browse/FLINK-20200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20200:
Fix Version/s: 1.12.0
> SQL Hints are not supported in "Create View" syntax
>
>
> Key: FLINK-20200
> URL: https://issues.apache.org/jira/browse/FLINK-20200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.2
>Reporter: shizhengchao
>Assignee: Danny Chen
>Priority: Major
> Fix For: 1.12.0
>
>
> I have aready set the config option `table.dynamic-table-options.enabled` to
> be true, but "SQL Hints" are not supported in View syntax. I got an error:
> {code:java}
> Exception in thread "main" java.lang.UnsupportedOperationException: class
> org.apache.calcite.sql.SqlSyntax$6: SPECIAL
> at org.apache.calcite.util.Util.needToImplement(Util.java:967)
> at org.apache.calcite.sql.SqlSyntax$6.unparse(SqlSyntax.java:116)
> at org.apache.calcite.sql.SqlOperator.unparse(SqlOperator.java:333)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:101)
> at
> org.apache.calcite.sql.SqlSelectOperator.unparse(SqlSelectOperator.java:176)
> at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:470)
> at org.apache.calcite.sql.SqlSelect.unparse(SqlSelect.java:246)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:151)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:173)
> at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:182)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.getQuotedSqlString(SqlToOperationConverter.java:784)
> at
> org.apache.flink.table.planner.utils.Expander$Expanded.substitute(Expander.java:169)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertViewQuery(SqlToOperationConverter.java:694)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertCreateView(SqlToOperationConverter.java:665)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:228)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:684)
> {code}
> The sql code is as follows：
> {code:java}
> drop table if exists SourceA;
> create table SourceA (
> idstring,
> name string
> ) with (
> 'connector' = 'kafka-0.11',
> 'topic' = 'MyTopic',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'Test',
> 'scan.startup.mode' = 'group-offsets',
> 'format' = 'csv'
> );
> drop table if exists print;
> create table print (
> idstring,
> name string
> ) with (
> 'connector' = 'print'
> );
> drop view if exists test_view;
> create view test_view as
> select
> *
> from SourceA /*+ OPTIONS('properties.group.id'='NewGroup') */;
> insert into print
> select * from test_view;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525731003 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -137,186 +137,182 @@ the main path ``` -## 4. Design and Structure +## 4. 设计和结构 -While it is hard to exactly specify what constitutes a good design, there are some properties that can serve as a _litmus test_ for a good design. If these properties are given, the chances are good that the design is going into a good direction. If these properties cannot be achieved, there is a high probability that the design is flawed. +虽然很难确切地指定一个好的设计是由什么构成的，但是有一些属性可以作为好的设计的试金石。如果设计上拥有这些属性，那么就有可能得到好的发展。否则，设计就很有可能存在缺陷。 -### Immutability and Eager Initialization +### 不变性（Immutability）和急切初始化（Eager Initialization） -1. Try to use immutable types where possible, especially for APIs, messages, identifiers, properties, configuration, etc. -2. A good general approach is to try and make as many fields of a class `final` as possible. -3. Classes that are used as keys in maps should be strictly immutable and only have `final` fields (except maybe auxiliary fields, like lazy cached hash codes). -4. Eagerly initialize classes. There should be no `init()` or `setup()` methods. Once the constructor completes, the object should be usable. +1. 尽可能尝试使用不可变类型，尤其是 API、消息、标识符、属性、配置等等。 +2. 一个好的通用方法是尽可能地将类中的字段设置为 `final`。 +3. 在 map 中作为 key 的类应该是严格不可变的，并且只有 final 字段（可能除了辅助的字段，如延迟缓存的 hash code）。 +4. 急切初始化类。不应该有 `init()` 或 `setup()` 方法。构造函数完成后，对象应该就可用。 Review comment: ```suggestion 4. 预初始化类。不应该有 `init()` 或 `setup()` 方法。构造函数完成后，对象应该就可用。 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18734) Add documentation for DynamoStreams Consumer CDC
[ https://issues.apache.org/jira/browse/FLINK-18734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-18734: - Fix Version/s: (was: 1.11.3) 1.11.4 > Add documentation for DynamoStreams Consumer CDC > > > Key: FLINK-18734 > URL: https://issues.apache.org/jira/browse/FLINK-18734 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis, Documentation >Affects Versions: 1.11.1 >Reporter: Vinay >Priority: Minor > Labels: CDC, documentation > Fix For: 1.12.0, 1.11.4 > > > Flink already supports CDC for DynamoDb - > https://issues.apache.org/jira/browse/FLINK-4582 by reading the data from > DynamoStreams but there is no documentation for the same. Given that Flink > now supports CDC for Debezium as well , we should add the documentation for > Dynamo CDC so that more users can use this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16268) Failed to run rank over window with Hive built-in functions
[
https://issues.apache.org/jira/browse/FLINK-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234208#comment-17234208
]
Rui Li commented on FLINK-16268:
[~xintongsong] Yes it can be closed
> Failed to run rank over window with Hive built-in functions
> ---
>
> Key: FLINK-16268
> URL: https://issues.apache.org/jira/browse/FLINK-16268
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test:
> {code}
> @Test
> public void test() throws Exception {
> hiveShell.execute("create table emp (dep string,name
> string,salary int)");
> hiveShell.insertInto("default", "emp").addRow("1", "A",
> 1).addRow("1", "B", 2).addRow("2", "C", 3).commit();
> TableEnvironment tableEnv = // create table env...
> tableEnv.unloadModule("core");
> tableEnv.loadModule("hive", new
> HiveModule(hiveCatalog.getHiveVersion()));
> tableEnv.loadModule("core", CoreModule.INSTANCE);
> List results =
> TableUtils.collectToList(tableEnv.sqlQuery("select dep,name,rank() over
> (partition by dep order by salary) as rnk from emp"));
> }
> {code}
> fails with:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.flink.table.functions.hive.conversion.HiveInspectors.toInspectors(HiveInspectors.java:126)
> at
> org.apache.flink.table.functions.hive.HiveGenericUDF.getHiveResultType(HiveGenericUDF.java:97)
> at
> org.apache.flink.table.functions.hive.HiveScalarFunction.getResultType(HiveScalarFunction.java:75)
> at
> org.apache.flink.table.planner.functions.utils.UserDefinedFunctionUtils$.getResultTypeOfScalarFunction(UserDefinedFunctionUtils.scala:620)
> at
> org.apache.flink.table.planner.expressions.PlannerScalarFunctionCall.resultType(call.scala:165)
> at
> org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:75)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:213)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:134)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:134)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:89)
> at
> org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:39)
> at
> org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:135)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:83)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:84)
> ..
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19887) Table program cannot be compiled when using Scala package object
[
https://issues.apache.org/jira/browse/FLINK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-19887:
-
Fix Version/s: (was: 1.11.3)
1.11.4
> Table program cannot be compiled when using Scala package object
>
>
> Key: FLINK-19887
> URL: https://issues.apache.org/jira/browse/FLINK-19887
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.2
> Environment: 1.11.2
> 2.12
> jdk：1.8
>
>Reporter: 谢波
>Priority: Major
> Fix For: 1.11.4
>
>
> {code:scala}
> package object analysis {
> case class UserBehavior(userId: Long, productId: Long, categoryId: Long,
> behavior: String, ts: Long)
> case class ItemViewCount(var windowEnd: Long,var itemId: Long,var count:
> Long)
> }
>
> def main(args: Array[String]): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val tableEnv = StreamTableEnvironment.create(env)
> env.setParallelism(1)
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> val input = env.readTextFile("mock-data/UserBehavior.csv")
> .map(e => {
> val split = e.split(",")
> UserBehavior(split(0).toLong, split(1).toLong, split(2).toLong, split(3),
> split(4).toLong)
> })
> .assignAscendingTimestamps(_.ts * 1000L)
> // dataStreamApi(input)
> //包对象下的样例类会导致编译错误，这是一个BUG
> val table = tableEnv.fromDataStream(input, $"productId", $"behavior",
> $"ts".rowtime)
> table.printSchema()
> table
> .window(Slide over 1.hour every 5.minutes on $"ts" as $"w")
> .groupBy($"w", $"productId")
> .select($"w".end, $"productId", $"productId".count)
> .toAppendStream[Row]
> .print("table ")
> // table.toAppendStream[Row].print("table")
> // tableEnv.execute("table")
> env.execute("hot item analysis")
> }
> {code}
>
>
> {code}
> rootroot |-- productId: BIGINT |-- behavior: STRING |-- ts: TIMESTAMP(3)
> *ROWTIME*
> /* 1 *//* 2 */ public class SourceConversion$4 extends
> org.apache.flink.table.runtime.operators.AbstractProcessStreamOperator/* 3 */
> implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator \{/* 4 *//* 5
> */ private final Object[] references;/* 6 */ private transient
> org.apache.flink.table.data.util.DataFormatConverters.CaseClassConverter
> converter$0;/* 7 */ org.apache.flink.table.data.GenericRowData out =
> new org.apache.flink.table.data.GenericRowData(3);/* 8 */ private
> final org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement
> = new org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);/* 9
> *//* 10 */ public SourceConversion$4(/* 11 */ Object[]
> references,/* 12 */
> org.apache.flink.streaming.runtime.tasks.StreamTask task,/* 13 */
> org.apache.flink.streaming.api.graph.StreamConfig config,/* 14 */
> org.apache.flink.streaming.api.operators.Output output,/* 15 */
> org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
> processingTimeService) throws Exception {/* 16 */ this.references =
> references;/* 17 */ converter$0 =
> (((org.apache.flink.table.data.util.DataFormatConverters.CaseClassConverter)
> references[0]));/* 18 */ this.setup(task, config, output);/* 19 */
> if (this instanceof
> org.apache.flink.streaming.api.operators.AbstractStreamOperator) {/* 20 */
> ((org.apache.flink.streaming.api.operators.AbstractStreamOperator)
> this)/* 21 */
> .setProcessingTimeService(processingTimeService);/* 22 */ }/* 23 */
> }/* 24 *//* 25 */ @Override/* 26 */ public void open()
> throws Exception \{/* 27 */ super.open();/* 28 */ /* 29 */
> }/* 30 *//* 31 */ @Override/* 32 */ public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception \{/* 33 */
> org.apache.flink.table.data.RowData in1 =
> (org.apache.flink.table.data.RowData) (org.apache.flink.table.data.RowData)
> converter$0.toInternal((com.hiscat.flink.user.behavior.analysis.package.UserBehavior)
> element.getValue());/* 34 */ /* 35 */ long field$1;/* 36
> */ boolean isNull$1;/* 37 */
> org.apache.flink.table.data.binary.BinaryStringData field$2;/* 38 */
> boolean isNull$2;/* 39 */ org.apache.flink.table.data.TimestampData
> result$3;/* 40 */ boolean isNull$3;/* 41 */ isNull$1 =
> in1.isNullAt(1);/* 42 */ field$1 = -1L;/* 43 */ if
> (!isNull$1) {/* 44 */ field$1 = in1.getLong(1);/* 45 */
> }/* 46 */ isNull$2 = in1.isNullAt(3);/* 47 */
[jira] [Resolved] (FLINK-16268) Failed to run rank over window with Hive built-in functions
[
https://issues.apache.org/jira/browse/FLINK-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-16268.
Resolution: Fixed
> Failed to run rank over window with Hive built-in functions
> ---
>
> Key: FLINK-16268
> URL: https://issues.apache.org/jira/browse/FLINK-16268
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test:
> {code}
> @Test
> public void test() throws Exception {
> hiveShell.execute("create table emp (dep string,name
> string,salary int)");
> hiveShell.insertInto("default", "emp").addRow("1", "A",
> 1).addRow("1", "B", 2).addRow("2", "C", 3).commit();
> TableEnvironment tableEnv = // create table env...
> tableEnv.unloadModule("core");
> tableEnv.loadModule("hive", new
> HiveModule(hiveCatalog.getHiveVersion()));
> tableEnv.loadModule("core", CoreModule.INSTANCE);
> List results =
> TableUtils.collectToList(tableEnv.sqlQuery("select dep,name,rank() over
> (partition by dep order by salary) as rnk from emp"));
> }
> {code}
> fails with:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.flink.table.functions.hive.conversion.HiveInspectors.toInspectors(HiveInspectors.java:126)
> at
> org.apache.flink.table.functions.hive.HiveGenericUDF.getHiveResultType(HiveGenericUDF.java:97)
> at
> org.apache.flink.table.functions.hive.HiveScalarFunction.getResultType(HiveScalarFunction.java:75)
> at
> org.apache.flink.table.planner.functions.utils.UserDefinedFunctionUtils$.getResultTypeOfScalarFunction(UserDefinedFunctionUtils.scala:620)
> at
> org.apache.flink.table.planner.expressions.PlannerScalarFunctionCall.resultType(call.scala:165)
> at
> org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:75)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:213)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:134)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:134)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:89)
> at
> org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:39)
> at
> org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:135)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:83)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:84)
> ..
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20086) Add documentation for the open method of UserDefinedFunction
[ https://issues.apache.org/jira/browse/FLINK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-20086: - Fix Version/s: (was: 1.11.3) 1.11.4 > Add documentation for the open method of UserDefinedFunction > > > Key: FLINK-20086 > URL: https://issues.apache.org/jira/browse/FLINK-20086 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Dian Fu >Priority: Major > Fix For: 1.12.0, 1.11.4 > > > According to the questions asked by PyFlink users so far, many users are not > aware that there is a open method in UserDefinedFunction where they could > perform initialization work. This method is especially useful for ML users > where they could perform ML mode initialization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18460) If the Python UDFs are called before calling the Python Dependency Management API, the Python Dependency Management would not will not take effect.
[ https://issues.apache.org/jira/browse/FLINK-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-18460: - Fix Version/s: (was: 1.11.3) 1.11.4 > If the Python UDFs are called before calling the Python Dependency Management > API, the Python Dependency Management would not will not take effect. > --- > > Key: FLINK-18460 > URL: https://issues.apache.org/jira/browse/FLINK-18460 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.10.1, 1.11.0, 1.12.0 >Reporter: Wei Zhong >Priority: Minor > Fix For: 1.12.0, 1.10.3, 1.11.4 > > > When developing PyFlink Job, If the python UDFs was called before specifying > the python dependencies, the python dependencies would not take effect on the > previously called python UDFs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20108) SingleThreadFetcherManager may add splits to a shutting down SplitFetcher
[
https://issues.apache.org/jira/browse/FLINK-20108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-20108:
-
Fix Version/s: (was: 1.11.3)
1.11.4
> SingleThreadFetcherManager may add splits to a shutting down SplitFetcher
> -
>
> Key: FLINK-20108
> URL: https://issues.apache.org/jira/browse/FLINK-20108
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
> Fix For: 1.11.4
>
>
> Currently the split fetchers are only removed from the
> {{SplitFetcherManager.fetchers}} when the thread exit. The may cause problem
> because when {{SplitFetcherManager.addSplits()}} is called, it may see a
> shutting down split fetcher and adds splits to it. These splits will then
> just be lost.
> This issue is actually already fixed in FLINK-18128. The fix needs to
> cherry-picked to 1.11.3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525718280 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -30,95 +30,95 @@ Each file must include the Apache license information as a header. */ ``` -## 2. Tools +## 2. 工具 Review comment: ```suggestion ## 2. 工具 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19535) SourceCoordinator should avoid fail job multiple times.
[
https://issues.apache.org/jira/browse/FLINK-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-19535:
-
Fix Version/s: (was: 1.11.3)
1.11.4
> SourceCoordinator should avoid fail job multiple times.
> ---
>
> Key: FLINK-19535
> URL: https://issues.apache.org/jira/browse/FLINK-19535
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
> Fix For: 1.12.0, 1.11.4
>
>
> Currently the {{SourceCoordinator}} may invoke
> {{SourceCoordinatorContext#failJob()}} multiple times from the same instance.
> This may cause the job to failover multiple times unnecessarily. The
> {{SourceCoordinator}} should instead just fail the job once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525728552 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -30,95 +30,95 @@ Each file must include the Apache license information as a header. */ ``` -## 2. Tools +## 2. 工具 Review comment: ```suggestion ## 2. 工具 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20202) Add the Check of Unsupported Result Type in Pandas UDAF
Huang Xingbo created FLINK-20202:
Summary: Add the Check of Unsupported Result Type in Pandas UDAF
Key: FLINK-20202
URL: https://issues.apache.org/jira/browse/FLINK-20202
Project: Flink
Issue Type: Improvement
Components: API / Python
Reporter: Huang Xingbo
Fix For: 1.12.0
Currently the return type of Pandas UDAF should be a primitive data type, and
the returned scalar can be either a python primitive type, e.g., {{int}} or
{{float}} or a numpy data type, e.g., {{numpy.int64}} or {{numpy.float64}}.
{{Any}} should ideally be a specific scalar type accordingly. We will add
related DataType check and throw a more readable exception for unsupported
DataTypes. What's more, we will add related notes in docs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19415) Move Hive document to "Table & SQL Connectors" from "Table API & SQL"
[ https://issues.apache.org/jira/browse/FLINK-19415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-19415: - Fix Version/s: (was: 1.11.3) 1.11.4 > Move Hive document to "Table & SQL Connectors" from "Table API & SQL" > - > > Key: FLINK-19415 > URL: https://issues.apache.org/jira/browse/FLINK-19415 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Affects Versions: 1.11.2 >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.11.4 > > > After the 1.11 release, heard some voices recently: How can't Hive's > documents be found in the "Table & SQL Connectors". > Actually, Hive's documents are in the "Table API & SQL". Since the "Table & > SQL Connectors" document was extracted separately, Hive is a little out of > place. > And Hive's code is also in "flink-connector-hive", which should be a > connector. > Hive also includes the concept of HiveCatalog. And Catalog should be a part > of the connector. > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-Hive-document-to-quot-Table-amp-SQL-Connectors-quot-from-quot-Table-API-amp-SQL-quot-td45186.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18880) Allow to configure the heap memory used by the gateway server
[ https://issues.apache.org/jira/browse/FLINK-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-18880: - Fix Version/s: (was: 1.11.3) 1.11.4 > Allow to configure the heap memory used by the gateway server > - > > Key: FLINK-18880 > URL: https://issues.apache.org/jira/browse/FLINK-18880 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.11.0 >Reporter: Dian Fu >Priority: Major > Fix For: 1.12.0, 1.11.4 > > > Currently, it doesn't allow to configure the heap memory used by the gateway > server. It may cause OOM issues in scenarios such as Table.to_pandas when the > content of the Table is big. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19228) Avoid accessing FileSystem in client for file system connector
[
https://issues.apache.org/jira/browse/FLINK-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-19228:
-
Fix Version/s: (was: 1.11.3)
1.11.4
> Avoid accessing FileSystem in client for file system connector
> ---
>
> Key: FLINK-19228
> URL: https://issues.apache.org/jira/browse/FLINK-19228
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Assignee: Nicholas Jiang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.4
>
>
> On the client, there may not be a corresponding file system plugin, so we can
> not access the specific file system. We can not access the file system on the
> client, but put the work on the job manager or task manager.
> Currently, it seems that only creating temporary directory through Filesystem
> in {{toStagingPath}}, but this is completely avoidable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525726673 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -137,186 +137,182 @@ the main path ``` -## 4. Design and Structure +## 4. 设计和结构 -While it is hard to exactly specify what constitutes a good design, there are some properties that can serve as a _litmus test_ for a good design. If these properties are given, the chances are good that the design is going into a good direction. If these properties cannot be achieved, there is a high probability that the design is flawed. +虽然很难确切地指定一个好的设计是由什么构成的，但是有一些属性可以作为好的设计的试金石。如果设计上拥有这些属性，那么就有可能得到好的发展。否则，设计就很有可能存在缺陷。 -### Immutability and Eager Initialization +### 不变性（Immutability）和急切初始化（Eager Initialization） -1. Try to use immutable types where possible, especially for APIs, messages, identifiers, properties, configuration, etc. -2. A good general approach is to try and make as many fields of a class `final` as possible. -3. Classes that are used as keys in maps should be strictly immutable and only have `final` fields (except maybe auxiliary fields, like lazy cached hash codes). -4. Eagerly initialize classes. There should be no `init()` or `setup()` methods. Once the constructor completes, the object should be usable. +1. 尽可能尝试使用不可变类型，尤其是 API、消息、标识符、属性、配置等等。 +2. 一个好的通用方法是尽可能地将类中的字段设置为 `final`。 +3. 在 map 中作为 key 的类应该是严格不可变的，并且只有 final 字段（可能除了辅助的字段，如延迟缓存的 hash code）。 Review comment: ```suggestion 3. 在 map 中作为 key 的类应该是严格不可变的，并且只有 final 字段（可能除了辅助的字段，如延迟缓存的 key）。 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path
[ https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-18785: - Fix Version/s: (was: 1.11.3) 1.11.4 > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path > -- > > Key: FLINK-18785 > URL: https://issues.apache.org/jira/browse/FLINK-18785 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing, Runtime / > Coordination >Affects Versions: 1.10.0, 1.10.1 > Environment: flink on yarn > flink-1.10.x > jdk8 > flink-conf.yaml yarn.application-attempts: 2 (or just delete this config) > yarn-2.7.2 >Reporter: Kai Chen >Priority: Major > Fix For: 1.12.0, 1.10.3, 1.11.4 > > Attachments: flink_savepoint_path_do_not_exits.jpg, > image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, > jobmanager.log.attemp2-13 > > > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path. > I just run this cmd: > bin/flink run -m yarn-cluster -s "hdfs:///do/not/exist/path" > examples/streaming/ > WindowJoin.jar > when i visit UI，i meet this: > !image-2020-07-31-19-04-19-241.png! > in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader > election > > Here is the stack trace in `jobmanager.err`: > ERROR ConnectionState Authentication failed > ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Could not start > recovered job 94b0911af12b61d3ee905baf1. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218) > at > org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown > Source) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at >
[jira] [Commented] (FLINK-18775) Rework PyFlink Documentation
[ https://issues.apache.org/jira/browse/FLINK-18775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234207#comment-17234207 ] Xintong Song commented on FLINK-18775: -- [~zhongwei] [~bigjoy] Is this ticket still targeting for 1.11.3? I'm aware that many of the subtasks are resolved on the master branch only. > Rework PyFlink Documentation > > > Key: FLINK-18775 > URL: https://issues.apache.org/jira/browse/FLINK-18775 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Affects Versions: 1.11.0, 1.11.1 >Reporter: sunjincheng >Assignee: Wei Zhong >Priority: Major > Labels: beginner > Fix For: 1.12.0, 1.11.3 > > > Since the release of Flink 1.11, users of PyFlink have continued to grow. > According to the feedback we received, current Flink documentation is not > very friendly to PyFlink users. There are two shortcomings: > # Python related content is mixed in the Java/Scala documentation, which > makes it difficult for users who only focus on PyFlink to read. > # There is already a "Python Table API" section in the Table API document to > store PyFlink documents, but the number of articles is small and the content > is fragmented. It is difficult for beginners to learn from it. > In addition, > [FLIP-130|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298] > introduced the Python DataStream API. Many documents will be added for those > new APIs. In order to increase the readability and maintainability of the > PyFlink document, we would like to rework it via this umbrella JIRA. > > The detail can be found in > [FLIP-133|https://cwiki.apache.org/confluence/display/FLINK/FLIP-133%3A+Rework+PyFlink+Documentation] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525725073 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -137,186 +137,182 @@ the main path ``` -## 4. Design and Structure +## 4. 设计和结构 -While it is hard to exactly specify what constitutes a good design, there are some properties that can serve as a _litmus test_ for a good design. If these properties are given, the chances are good that the design is going into a good direction. If these properties cannot be achieved, there is a high probability that the design is flawed. +虽然很难确切地指定一个好的设计是由什么构成的，但是有一些属性可以作为好的设计的试金石。如果设计上拥有这些属性，那么就有可能得到好的发展。否则，设计就很有可能存在缺陷。 -### Immutability and Eager Initialization +### 不变性（Immutability）和急切初始化（Eager Initialization） -1. Try to use immutable types where possible, especially for APIs, messages, identifiers, properties, configuration, etc. -2. A good general approach is to try and make as many fields of a class `final` as possible. -3. Classes that are used as keys in maps should be strictly immutable and only have `final` fields (except maybe auxiliary fields, like lazy cached hash codes). -4. Eagerly initialize classes. There should be no `init()` or `setup()` methods. Once the constructor completes, the object should be usable. +1. 尽可能尝试使用不可变类型，尤其是 API、消息、标识符、属性、配置等等。 +2. 一个好的通用方法是尽可能地将类中的字段设置为 `final`。 +3. 在 map 中作为 key 的类应该是严格不可变的，并且只有 final 字段（可能除了辅助的字段，如延迟缓存的 hash code）。 +4. 急切初始化类。不应该有 `init()` 或 `setup()` 方法。构造函数完成后，对象应该就可用。 -### Nullability of the Mutable Parts +### 可变部件（Mutable Parts）的可空性（Nullability） -For nullability, the Flink codebase aims to follow these conventions: +Flink 代码库对于可空性旨在遵循以下约定: -* Fields, parameters, and return types are always non-null, unless indicated otherwise -* All fields, parameters and method types that can be null should be annotated with `@javax.annotation.Nullable`. -That way, you get warnings from IntelliJ about all sections where you have to reason about potential null values. -* For all mutable (non-final) fields that are not annotated, the assumption is that while the field value changes, there always is a value. -* This should be double check whether these can in fact not be null throughout the lifetime of the object. +* 除非有特殊说明，否则字段、参数和返回类型始终都是非 null。 +* 所有可以为 null 的字段、参数和方法类型都要使用 `@javax.annotation.Nullable` 注解（Annotated）。 +这样 IntelliJ 就能够对可能出现的 null 值进行警告. +* 对于那些没有添加注解的可变（not-final）字段，就无法确定字段值是否为 null。 +* 此时应该仔细的检查这些值在对象的整个生命周期中是否可以不为 null。 -_Note: This means that `@Nonnull` annotations are usually not necessary, but can be used in certain cases to override a previous annotation, or to point non-nullability out in a context where one would expect a nullable value._ +_注意: 大部分情况下是不需要 `@Nonnull` 注解的，但有些时候可以用来覆盖之前的注解，或者在不可以为 null 的上下文（Context）中，还想要得到 null 值。_ -`Optional` is a good solution as a return type for method that may or may not have a result, so nullable return types are good candidates to be replaced with `Optional`. -See also [usage of Java Optional](code-style-and-quality-java.md#java-optional). +对于不确定是否有结果返回的方法，`Optional` 作为方法的返回类型是个很好的解决方案，可以用 `Optional` 来代替所有可以为 null 的返回类型。 +参考 [Java Optional 的用法](code-style-and-quality-java.zh.md#java-optional). -### Avoid Code Duplication +### 避免重复的代码 -1. Whenever you are about to copy/paste some code, or reproduce a similar type of functionality in a different place, think about the ways how to refactor/reuse/abstract the changes to avoid the duplication. -2. Common behavior between different specializations should be shared in a common component (or a shared superclass). -3. Always use “private static final” constants instead of duplicating strings or other special values at different locations. Constants should be declared in the top member area of a class. +1. 当你准备复制/粘贴一些代码，或者在不同的地方实现类似的功能时，就要考虑怎么去重构、复用、抽象来避免重复的代码。 +2. 不同模块之间的相同特性应该抽象到公共组件（或父类）中。 +3. 常量应该声明在类顶部的成员区域中，并且是使用“private static final”修饰，而不是在不同的地方复制。 -### Design for Testability +### 可测性设计（Design for Testability） -Code that is easily testable typically has good separation of concerns and is structured to be reusable outside the original context (by being easily reusable in tests). +容易进行测试的代码通常能够很好的使关注点分离，并且可以在其他地方重复使用（测试的时候很容易重复使用）。 -A good summary or problems / symptoms and recommended refactoring is in the PDF linked below. -Please note that while the examples in the PDF often use a dependency injection framework (Guice), it works in the same way without such a framework.[^1] +下面的 PDF 链接中有对问题的总结和重构的建议。需要注意的是，虽然 PDF 中的示例使用 Guice 作为依赖注入框架，但是如果没有使用这个框架，它也能达到相同的效果。[^1] [http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf](http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf) -Here is a compact summary of the most important aspects. +下面是重点方面的简要总结 -**Inject dependencies** +**依赖关系的注入** -Reusability becomes easier if constructors don’t
[GitHub] [flink-web] ClownfishYang commented on a change in pull request #394: [FLINK-13680] Translate "Common Rules" page into Chinese
ClownfishYang commented on a change in pull request #394: URL: https://github.com/apache/flink-web/pull/394#discussion_r525722961 ## File path: contributing/code-style-and-quality-common.zh.md ## @@ -137,186 +137,182 @@ the main path ``` -## 4. Design and Structure +## 4. 设计和结构 -While it is hard to exactly specify what constitutes a good design, there are some properties that can serve as a _litmus test_ for a good design. If these properties are given, the chances are good that the design is going into a good direction. If these properties cannot be achieved, there is a high probability that the design is flawed. +虽然很难确切地指定一个好的设计是由什么构成的，但是有一些属性可以作为好的设计的试金石。如果设计上拥有这些属性，那么就有可能得到好的发展。否则，设计就很有可能存在缺陷。 -### Immutability and Eager Initialization +### 不变性（Immutability）和急切初始化（Eager Initialization） -1. Try to use immutable types where possible, especially for APIs, messages, identifiers, properties, configuration, etc. -2. A good general approach is to try and make as many fields of a class `final` as possible. -3. Classes that are used as keys in maps should be strictly immutable and only have `final` fields (except maybe auxiliary fields, like lazy cached hash codes). -4. Eagerly initialize classes. There should be no `init()` or `setup()` methods. Once the constructor completes, the object should be usable. +1. 尽可能尝试使用不可变类型，尤其是 API、消息、标识符、属性、配置等等。 +2. 一个好的通用方法是尽可能地将类中的字段设置为 `final`。 +3. 在 map 中作为 key 的类应该是严格不可变的，并且只有 final 字段（可能除了辅助的字段，如延迟缓存的 hash code）。 +4. 急切初始化类。不应该有 `init()` 或 `setup()` 方法。构造函数完成后，对象应该就可用。 -### Nullability of the Mutable Parts +### 可变部件（Mutable Parts）的可空性（Nullability） -For nullability, the Flink codebase aims to follow these conventions: +Flink 代码库对于可空性旨在遵循以下约定: -* Fields, parameters, and return types are always non-null, unless indicated otherwise -* All fields, parameters and method types that can be null should be annotated with `@javax.annotation.Nullable`. -That way, you get warnings from IntelliJ about all sections where you have to reason about potential null values. -* For all mutable (non-final) fields that are not annotated, the assumption is that while the field value changes, there always is a value. -* This should be double check whether these can in fact not be null throughout the lifetime of the object. +* 除非有特殊说明，否则字段、参数和返回类型始终都是非 null。 +* 所有可以为 null 的字段、参数和方法类型都要使用 `@javax.annotation.Nullable` 注解（Annotated）。 +这样 IntelliJ 就能够对可能出现的 null 值进行警告. +* 对于那些没有添加注解的可变（not-final）字段，就无法确定字段值是否为 null。 +* 此时应该仔细的检查这些值在对象的整个生命周期中是否可以不为 null。 -_Note: This means that `@Nonnull` annotations are usually not necessary, but can be used in certain cases to override a previous annotation, or to point non-nullability out in a context where one would expect a nullable value._ +_注意: 大部分情况下是不需要 `@Nonnull` 注解的，但有些时候可以用来覆盖之前的注解，或者在不可以为 null 的上下文（Context）中，还想要得到 null 值。_ -`Optional` is a good solution as a return type for method that may or may not have a result, so nullable return types are good candidates to be replaced with `Optional`. -See also [usage of Java Optional](code-style-and-quality-java.md#java-optional). +对于不确定是否有结果返回的方法，`Optional` 作为方法的返回类型是个很好的解决方案，可以用 `Optional` 来代替所有可以为 null 的返回类型。 +参考 [Java Optional 的用法](code-style-and-quality-java.zh.md#java-optional). -### Avoid Code Duplication +### 避免重复的代码 -1. Whenever you are about to copy/paste some code, or reproduce a similar type of functionality in a different place, think about the ways how to refactor/reuse/abstract the changes to avoid the duplication. -2. Common behavior between different specializations should be shared in a common component (or a shared superclass). -3. Always use “private static final” constants instead of duplicating strings or other special values at different locations. Constants should be declared in the top member area of a class. +1. 当你准备复制/粘贴一些代码，或者在不同的地方实现类似的功能时，就要考虑怎么去重构、复用、抽象来避免重复的代码。 +2. 不同模块之间的相同特性应该抽象到公共组件（或父类）中。 +3. 常量应该声明在类顶部的成员区域中，并且是使用“private static final”修饰，而不是在不同的地方复制。 -### Design for Testability +### 可测性设计（Design for Testability） -Code that is easily testable typically has good separation of concerns and is structured to be reusable outside the original context (by being easily reusable in tests). +容易进行测试的代码通常能够很好的使关注点分离，并且可以在其他地方重复使用（测试的时候很容易重复使用）。 -A good summary or problems / symptoms and recommended refactoring is in the PDF linked below. -Please note that while the examples in the PDF often use a dependency injection framework (Guice), it works in the same way without such a framework.[^1] +下面的 PDF 链接中有对问题的总结和重构的建议。需要注意的是，虽然 PDF 中的示例使用 Guice 作为依赖注入框架，但是如果没有使用这个框架，它也能达到相同的效果。[^1] [http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf](http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf) -Here is a compact summary of the most important aspects. +下面是重点方面的简要总结 -**Inject dependencies** +**依赖关系的注入** -Reusability becomes easier if constructors don’t
[jira] [Updated] (FLINK-18706) Stop with savepoint cannot guarantee exactly-once for kafka source
[ https://issues.apache.org/jira/browse/FLINK-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-18706: - Fix Version/s: (was: 1.11.3) 1.11.4 > Stop with savepoint cannot guarantee exactly-once for kafka source > -- > > Key: FLINK-18706 > URL: https://issues.apache.org/jira/browse/FLINK-18706 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Runtime / Checkpointing >Affects Versions: 1.10.1, 1.11.1 >Reporter: Yumeng Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.10.3, 1.11.4 > > > When I run stop-with-savepoint command with my old job and submit a new job > with the previous sync-savepoint, I find sometimes my new job will consume a > few duplicate data. Here is my case. I have a data generation job with > parallelism 1, which will generate long number incrementally and send the > data to Kafka topicA which only has one partition. Then I have another > consumer job with parallelism 1, which reads data from topicA and does > nothing processing, just print these numbers to system out. For example, > after doing stop-with-savepoint, my consumer job has printed sequence > 0,1,2,3...40,41,42,43. Then I start the consumer job again from that > sync-savepoint. It prints 41,42,43,44..., which means it has consumed some > duplicate data. > I think the reason is that we fail to guarantee the mutual exclusion between > canceling source task and sending data to downstream by checkpoint lock. It > may send some data to downstream first before sync-savepoint completed and > then cancel the task. Therefore, We need to keep the source operator running > in the synchronous savepoint mailbox loop for triggerCheckpoint method before > synchronous savepoint completed and keep checking running state before > sending data to downstream for Kafka connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
