[GitHub] [flink] flinkbot commented on issue #8106: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called
flinkbot commented on issue #8106: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8106#issuecomment-479336247 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] an0 opened a new pull request #8106: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called
an0 opened a new pull request #8106: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8106 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #8086: [FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table runtime
KurtYoung commented on a change in pull request #8086:
[FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table
runtime
URL: https://github.com/apache/flink/pull/8086#discussion_r271575050
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/bundle/AbstractMapBundleOperator.java
##
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.bundle;
+
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTrigger;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTriggerCallback;
+import org.apache.flink.table.runtime.context.ExecutionContextImpl;
+import org.apache.flink.table.runtime.util.StreamRecordCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link AbstractMapBundleOperator} simply used a java Map to store the
input elements
+ * in key-value form. The map key is typically the same with the state key, so
we can do some
+ * optimizations before accessing states, like pre aggregate values for each
key. And we will
+ * only need to access state every key we have, but not every element we
processed.
+ *
+ * NOTES: if all elements we processed have different keys, such operator
will only increase
+ * memory footprint, and will not have any performance improvement.
+ *
+ * @paramThe type of the key in the bundle map
+ * @paramThe type of the value in the bundle map
+ * @param Input type for the operator.
+ * @param Output type for the operator.
+ */
+public abstract class AbstractMapBundleOperator
+ extends AbstractStreamOperator
+ implements OneInputStreamOperator,
BundleTriggerCallback {
+
+ private static final long serialVersionUID = 5081841938324118594L;
+
+ /** The map in heap to store elements. */
+ private final transient Map bundle;
+
+ /** The trigger that determines how many elements should be put into a
bundle. */
+ private final BundleTrigger bundleTrigger;
+
+ /** The function used to process when receiving element. */
+ private final MapBundleFunction function;
+
+ /** Output for stream records. */
+ private transient Collector collector;
+
+ private transient int numOfElements = 0;
+
+ private transient volatile boolean isInFinishingBundle = false;
+ private transient Object checkpointingLock;
+
+ protected AbstractMapBundleOperator(
+ MapBundleFunction function,
+ BundleTrigger bundleTrigger) {
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ this.bundle = new HashMap<>();
+ this.function = checkNotNull(function, "function is null");
+ this.bundleTrigger = checkNotNull(bundleTrigger, "bundleTrigger
is null");
+ }
+
+ @Override
+ public void setup(
+ StreamTask containingTask,
+ StreamConfig config,
+ Output> output) {
+ super.setup(containingTask, config, output);
+ this.checkpointingLock =
getContainingTask().getCheckpointLock();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ function.open(new ExecutionContextImpl(this,
getRuntimeContext()));
+
+ this.numOfElements = 0;
+ this.collector =
[GitHub] [flink] an0 closed pull request #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called
an0 closed pull request #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8104 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8105: [FLINK-11309][core] Make blocking result partitions consumable
flinkbot commented on issue #8105: [FLINK-11309][core] Make blocking result partitions consumable URL: https://github.com/apache/flink/pull/8105#issuecomment-479332135 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] eaglewatcherwb opened a new pull request #8105: [FLINK-11309][core] Make blocking result partitions consumable
eaglewatcherwb opened a new pull request #8105: [FLINK-11309][core] Make blocking result partitions consumable URL: https://github.com/apache/flink/pull/8105 multiple times to speed up batch recoveries Change-Id: Id304257d201a9ea2c0f650c027204aabe2b2e9b9 ## What is the purpose of the change make SpillableSubpartition repeatably readable to enable region failover ## Brief change log - BufferConsumers are hold in buffers until the corresponding ExecutionVertex terminates. - SpillableSubpartition is not released until the corresponding ExecutionVertex terminates. - SpillableSubpartition creates multi SpillableSubpartitionViews, each of which is corresponding to a Execution Attempt. - Accommodate relative unit tests for repeatably readable SpillableSubpartition. - Add new unit test testPartitionRepeatedlyReadable to verify this modification. ## Verifying this change This change added tests and can be verified as follows: - Manually verified the change by running a 8 node cluser with 1 JobManagers and 8 TaskManagers, a batch WordCount example using `BATCH_FORCED` and and `region` failover-strategy, fail execution attempt of Reduce Vertex after read data. Job terminates after failover. - New unit test is added to verify this modification. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? https://docs.google.com/document/d/1uXuJFiKODf241CKci3b0JnaF3zQ-Wt0V9wmC7kYwX-M/edit?usp=sharing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called
flinkbot commented on issue #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8104#issuecomment-479328213 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] an0 opened a new pull request #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called
an0 opened a new pull request #8104: [FLINK-12092] [docs]Clarity when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8104 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12092) Doc of when `onTimer(...)` is called is inaccurate
[
https://issues.apache.org/jira/browse/FLINK-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-12092:
---
Labels: pull-request-available (was: )
> Doc of when `onTimer(...)` is called is inaccurate
> --
>
> Key: FLINK-12092
> URL: https://issues.apache.org/jira/browse/FLINK-12092
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Ling Wang
>Priority: Minor
> Labels: pull-request-available
>
> The doc says:
> {quote}When a timer's particular time is reached, the `onTimer(...)` method
> is, the `onTimer(...)` method is called.
> {quote}
> That's not accurate, in fact misleading. It contradicts what is said later in
> the same doc:
> {quote}Since event-time timers only fire with watermarks coming in
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] an0 closed pull request #8103: [FLINK-12092] [Documentation]
an0 closed pull request #8103: [FLINK-12092] [Documentation] URL: https://github.com/apache/flink/pull/8103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12092) Doc of when `onTimer(...)` is called is inaccurate
[
https://issues.apache.org/jira/browse/FLINK-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ling Wang updated FLINK-12092:
--
Description:
The doc says:
{quote}When a timer's particular time is reached, the `onTimer(...)` method is,
the `onTimer(...)` method is called.
{quote}
That's not accurate, in fact misleading. It contradicts what is said later in
the same doc:
{quote}Since event-time timers only fire with watermarks coming in
{quote}
> Doc of when `onTimer(...)` is called is inaccurate
> --
>
> Key: FLINK-12092
> URL: https://issues.apache.org/jira/browse/FLINK-12092
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Ling Wang
>Priority: Minor
>
> The doc says:
> {quote}When a timer's particular time is reached, the `onTimer(...)` method
> is, the `onTimer(...)` method is called.
> {quote}
> That's not accurate, in fact misleading. It contradicts what is said later in
> the same doc:
> {quote}Since event-time timers only fire with watermarks coming in
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12092) Doc of when `onTimer(...)` is called is inacc
Ling Wang created FLINK-12092: - Summary: Doc of when `onTimer(...)` is called is inacc Key: FLINK-12092 URL: https://issues.apache.org/jira/browse/FLINK-12092 Project: Flink Issue Type: Improvement Reporter: Ling Wang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12092) Doc of when `onTimer(...)` is called is inaccurate
[ https://issues.apache.org/jira/browse/FLINK-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ling Wang updated FLINK-12092: -- Priority: Minor (was: Major) > Doc of when `onTimer(...)` is called is inaccurate > -- > > Key: FLINK-12092 > URL: https://issues.apache.org/jira/browse/FLINK-12092 > Project: Flink > Issue Type: Improvement >Reporter: Ling Wang >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12092) Doc of when `onTimer(...)` is called is inaccurate
[ https://issues.apache.org/jira/browse/FLINK-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ling Wang updated FLINK-12092: -- Summary: Doc of when `onTimer(...)` is called is inaccurate (was: Doc of when `onTimer(...)` is called is inacc) > Doc of when `onTimer(...)` is called is inaccurate > -- > > Key: FLINK-12092 > URL: https://issues.apache.org/jira/browse/FLINK-12092 > Project: Flink > Issue Type: Improvement >Reporter: Ling Wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12092) Doc of when `onTimer(...)` is called is inaccurate
[ https://issues.apache.org/jira/browse/FLINK-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ling Wang updated FLINK-12092: -- Component/s: Documentation > Doc of when `onTimer(...)` is called is inaccurate > -- > > Key: FLINK-12092 > URL: https://issues.apache.org/jira/browse/FLINK-12092 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ling Wang >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8103: Improve the doc of when `onTimer(...)` is called
flinkbot commented on issue #8103: Improve the doc of when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8103#issuecomment-479323083 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] an0 opened a new pull request #8103: Improve the doc of when `onTimer(...)` is called
an0 opened a new pull request #8103: Improve the doc of when `onTimer(...)` is called URL: https://github.com/apache/flink/pull/8103 The current doc is not accurate. ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] eaglewatcherwb closed pull request #7537: [FLINK-11309][core] make SpillableSubpartition repeatably read to enable
eaglewatcherwb closed pull request #7537: [FLINK-11309][core] make SpillableSubpartition repeatably read to enable URL: https://github.com/apache/flink/pull/7537 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #8086: [FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table runtime
JingsongLi commented on a change in pull request #8086:
[FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table
runtime
URL: https://github.com/apache/flink/pull/8086#discussion_r271567786
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/bundle/MapBundleFunction.java
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.bundle;
+
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.table.runtime.context.ExecutionContext;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.util.Map;
+
+/**
+ * Basic interface for map bundle processing.
+ *
+ * @paramThe type of the key in the bundle map
+ * @paramThe type of the value in the bundle map
+ * @param Type of the input elements.
+ * @param Type of the returned elements.
+ */
+public abstract class MapBundleFunction implements Function {
+
+ private static final long serialVersionUID = -6672219582127325882L;
+
+ protected transient ExecutionContext ctx;
+
+ public void open(ExecutionContext ctx) throws Exception {
+ this.ctx = Preconditions.checkNotNull(ctx);
+ }
+
+ /**
+* Adds the given input to the given value, returning the new bundle
value.
+*
+* @param value the existing bundle value, maybe null
+* @param input the given input, not null
+*/
+ public abstract V addInput(@Nullable V value, IN input);
+
+ /**
+* Called when a bundle is finished. Transform a bundle to zero, one,
or more output elements.
+*/
+ public abstract void finishBundle(Map buffer, Collector out)
throws Exception;
+
Review comment:
Need add a method `endInput` to handle group aggregate without keys in
bounded streaming?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #8086: [FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table runtime
JingsongLi commented on a change in pull request #8086:
[FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table
runtime
URL: https://github.com/apache/flink/pull/8086#discussion_r271568421
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/bundle/AbstractMapBundleOperator.java
##
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.bundle;
+
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTrigger;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTriggerCallback;
+import org.apache.flink.table.runtime.context.ExecutionContextImpl;
+import org.apache.flink.table.runtime.util.StreamRecordCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link AbstractMapBundleOperator} simply used a java Map to store the
input elements
+ * in key-value form. The map key is typically the same with the state key, so
we can do some
+ * optimizations before accessing states, like pre aggregate values for each
key. And we will
+ * only need to access state every key we have, but not every element we
processed.
+ *
+ * NOTES: if all elements we processed have different keys, such operator
will only increase
+ * memory footprint, and will not have any performance improvement.
+ *
+ * @paramThe type of the key in the bundle map
+ * @paramThe type of the value in the bundle map
+ * @param Input type for the operator.
+ * @param Output type for the operator.
+ */
+public abstract class AbstractMapBundleOperator
+ extends AbstractStreamOperator
+ implements OneInputStreamOperator,
BundleTriggerCallback {
+
+ private static final long serialVersionUID = 5081841938324118594L;
+
+ /** The map in heap to store elements. */
+ private final transient Map bundle;
+
+ /** The trigger that determines how many elements should be put into a
bundle. */
+ private final BundleTrigger bundleTrigger;
+
+ /** The function used to process when receiving element. */
+ private final MapBundleFunction function;
+
+ /** Output for stream records. */
+ private transient Collector collector;
+
+ private transient int numOfElements = 0;
+
+ private transient volatile boolean isInFinishingBundle = false;
+ private transient Object checkpointingLock;
+
+ protected AbstractMapBundleOperator(
+ MapBundleFunction function,
+ BundleTrigger bundleTrigger) {
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ this.bundle = new HashMap<>();
+ this.function = checkNotNull(function, "function is null");
+ this.bundleTrigger = checkNotNull(bundleTrigger, "bundleTrigger
is null");
+ }
+
+ @Override
+ public void setup(
+ StreamTask containingTask,
+ StreamConfig config,
+ Output> output) {
+ super.setup(containingTask, config, output);
+ this.checkpointingLock =
getContainingTask().getCheckpointLock();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ function.open(new ExecutionContextImpl(this,
getRuntimeContext()));
+
+ this.numOfElements = 0;
+ this.collector =
[GitHub] [flink] JingsongLi commented on a change in pull request #8086: [FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table runtime
JingsongLi commented on a change in pull request #8086:
[FLINK-12062][table-runtime-blink] Introduce bundle operator to streaming table
runtime
URL: https://github.com/apache/flink/pull/8086#discussion_r271566995
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/bundle/AbstractMapBundleOperator.java
##
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.bundle;
+
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTrigger;
+import org.apache.flink.table.runtime.bundle.trigger.BundleTriggerCallback;
+import org.apache.flink.table.runtime.context.ExecutionContextImpl;
+import org.apache.flink.table.runtime.util.StreamRecordCollector;
+import org.apache.flink.util.Collector;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link AbstractMapBundleOperator} simply used a java Map to store the
input elements
+ * in key-value form. The map key is typically the same with the state key, so
we can do some
+ * optimizations before accessing states, like pre aggregate values for each
key. And we will
+ * only need to access state every key we have, but not every element we
processed.
+ *
+ * NOTES: if all elements we processed have different keys, such operator
will only increase
+ * memory footprint, and will not have any performance improvement.
+ *
+ * @paramThe type of the key in the bundle map
+ * @paramThe type of the value in the bundle map
+ * @param Input type for the operator.
+ * @param Output type for the operator.
+ */
+public abstract class AbstractMapBundleOperator
+ extends AbstractStreamOperator
+ implements OneInputStreamOperator,
BundleTriggerCallback {
+
+ private static final long serialVersionUID = 5081841938324118594L;
+
+ /** The map in heap to store elements. */
+ private final transient Map bundle;
+
+ /** The trigger that determines how many elements should be put into a
bundle. */
+ private final BundleTrigger bundleTrigger;
+
+ /** The function used to process when receiving element. */
+ private final MapBundleFunction function;
+
+ /** Output for stream records. */
+ private transient Collector collector;
+
+ private transient int numOfElements = 0;
+
+ private transient volatile boolean isInFinishingBundle = false;
+ private transient Object checkpointingLock;
+
+ protected AbstractMapBundleOperator(
+ MapBundleFunction function,
+ BundleTrigger bundleTrigger) {
+ chainingStrategy = ChainingStrategy.ALWAYS;
+ this.bundle = new HashMap<>();
+ this.function = checkNotNull(function, "function is null");
+ this.bundleTrigger = checkNotNull(bundleTrigger, "bundleTrigger
is null");
+ }
+
+ @Override
+ public void setup(
+ StreamTask containingTask,
+ StreamConfig config,
+ Output> output) {
+ super.setup(containingTask, config, output);
+ this.checkpointingLock =
getContainingTask().getCheckpointLock();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ function.open(new ExecutionContextImpl(this,
getRuntimeContext()));
+
+ this.numOfElements = 0;
+ this.collector =
[jira] [Created] (FLINK-12091) Can Flink support the historyserver function like spark?
彭洋 created FLINK-12091: -- Summary: Can Flink support the historyserver function like spark? Key: FLINK-12091 URL: https://issues.apache.org/jira/browse/FLINK-12091 Project: Flink Issue Type: Improvement Reporter: 彭洋 Most of the tasks are run as flink on yarn, but when the job hangs, flink's webui can't be accessed, although you can view some logs according to the env.log.dir configuration path but there is no way to view the full log. Information, can you develop a function like spark historyserver, you can still view the full log of this job after the job is hung up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12090) Can the log on the Apache Flink Dashboard be partitioned by day?
彭洋 created FLINK-12090: -- Summary: Can the log on the Apache Flink Dashboard be partitioned by day? Key: FLINK-12090 URL: https://issues.apache.org/jira/browse/FLINK-12090 Project: Flink Issue Type: Improvement Affects Versions: 1.7.2 Reporter: 彭洋 The flink job is always executed by the real-time task, so its taskmanager log and jobmanager log will be larger and larger. It will be very inconvenient to view on the flink webui. The loading is very slow, and the log can be partitioned according to the day. Or you can set the maximum retention time of the log, only keep the logs of the last few days, so it is more convenient to view the log, and there is another point, can you output gc information to the log? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-6227) Introduce the DataConsumptionException for downstream task failure
[
https://issues.apache.org/jira/browse/FLINK-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhijiang reassigned FLINK-6227:
---
Assignee: zhijiang
> Introduce the DataConsumptionException for downstream task failure
> --
>
> Key: FLINK-6227
> URL: https://issues.apache.org/jira/browse/FLINK-6227
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> It is part of FLIP-1.
> We define a new special exception to indicate the downstream task failure in
> consuming upstream data.
> The {{JobManager}} will receive and consider this special exception to
> calculate the minimum connected sub-graph which is called {{FailoverRegion}}.
> So the {{DataConsumptionException}} should contain {{ResultPartitionID}}
> information for {{JobMaster}} tracking upstream executions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12089) Broadcasting Dynamic Configuration
langjingxiang created FLINK-12089:
-
Summary: Broadcasting Dynamic Configuration
Key: FLINK-12089
URL: https://issues.apache.org/jira/browse/FLINK-12089
Project: Flink
Issue Type: Improvement
Components: API / Scala, Runtime / Network
Affects Versions: 1.7.2
Reporter: langjingxiang
flink Broadcasting Dynamic Configuration now the way is:
datastream :broadcasting stream to join
dataset: broadcasting dataset
but Intrusion to user business is relatively large ，
I think we can design an API:
evn.broadcast(confName,confFunction,SchedulingTime)
dataStream.map(new RichMapFunction(){
public void open(Configuration parameters) throws Exception {
Objec broadcastSet = getRuntimeContext().getBroadcastVariable(confName);
}})
Job Manager Schedule confFunction broadcast to Task Context
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12088) Introduce unbounded streaming inner join operator
Kurt Young created FLINK-12088: -- Summary: Introduce unbounded streaming inner join operator Key: FLINK-12088 URL: https://issues.apache.org/jira/browse/FLINK-12088 Project: Flink Issue Type: Improvement Components: Table SQL / Runtime Reporter: Kurt Young Assignee: Kurt Young This operator is responsible for unbounded streaming inner join, and will be optimized in following cases: # If the join keys (with equality condition) are also primary key, we will have a more efficient state layout # If the inputs have primary keys, but join keys are not primary key, we can also come up with an efficient state layout # Inputs don't have primary keys, this will go to default implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12070) Make blocking result partitions consumable multiple times
[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808312#comment-16808312
]
BoWang commented on FLINK-12070:
Totally agree with this proposal.
I would like to take this Jira, and the [design
doc|https://docs.google.com/document/d/1uXuJFiKODf241CKci3b0JnaF3zQ-Wt0V9wmC7kYwX-M/edit?usp=sharing]
is ready. Would you mind to have a look at the design? [~till.rohrmann]
> Make blocking result partitions consumable multiple times
> -
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Till Rohrmann
>Priority: Major
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-12087) Introduce over window operators to blink batch
[ https://issues.apache.org/jira/browse/FLINK-12087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12087: --- Labels: pull-request-available (was: ) > Introduce over window operators to blink batch > -- > > Key: FLINK-12087 > URL: https://issues.apache.org/jira/browse/FLINK-12087 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > > Introduce NonBufferOverWindowOperator: Some over windows do not need to > buffer data, such as rank, rows between unbounded preceding and 0, etc. We > introduce NonBufferOverWindowOperator to reduce the overhead of data copy in > buffer. > Introduce BufferDataOverWindowOperator and OverWindowFrame: 1. Minimize > duplicate computation in various OverWindowFrame implementations. 2. An > OverWindowOperator can compute multiple window frames. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8102: [FLINK-12087][table-runtime-blink] Introduce over window operators to blink batch
flinkbot commented on issue #8102: [FLINK-12087][table-runtime-blink] Introduce over window operators to blink batch URL: https://github.com/apache/flink/pull/8102#issuecomment-479311696 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi opened a new pull request #8102: [FLINK-12087][table-runtime-blink] Introduce over window operators to blink batch
JingsongLi opened a new pull request #8102: [FLINK-12087][table-runtime-blink] Introduce over window operators to blink batch URL: https://github.com/apache/flink/pull/8102 ## What is the purpose of the change Introduce NonBufferOverWindowOperator: Some over windows do not need to buffer data, such as rank, rows between unbounded preceding and 0, etc. We introduce NonBufferOverWindowOperator to reduce the overhead of data copy in buffer. Introduce BufferDataOverWindowOperator and OverWindowFrame: 1. Minimize duplicate computation in various OverWindowFrame implementations. 2. An OverWindowOperator can compute multiple window frames. ## Verifying this change ut ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-12087) Introduce over window operators to blink batch
Jingsong Lee created FLINK-12087: Summary: Introduce over window operators to blink batch Key: FLINK-12087 URL: https://issues.apache.org/jira/browse/FLINK-12087 Project: Flink Issue Type: New Feature Components: Table SQL / Runtime Reporter: Jingsong Lee Assignee: Jingsong Lee Introduce NonBufferOverWindowOperator: Some over windows do not need to buffer data, such as rank, rows between unbounded preceding and 0, etc. We introduce NonBufferOverWindowOperator to reduce the overhead of data copy in buffer. Introduce BufferDataOverWindowOperator and OverWindowFrame: 1. Minimize duplicate computation in various OverWindowFrame implementations. 2. An OverWindowOperator can compute multiple window frames. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271556407
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.MINUS_PREFIX;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.RANGE_TO;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
+
+ private final List
inputFieldReferences;
+
+ public
ColumnsOperationExpander(List
inputFieldReferences) {
+ this.inputFieldReferences = inputFieldReferences;
+ }
+
+ @Override
+ public List visitCall(CallExpression call) {
+
+ List result;
+
+ String definitionName = call.getFunctionDefinition().getName();
+ if (definitionName.equals(COLUMNS.getName())) {
+ result = resolveArgsOfColumns(call.getChildren(),
false);
+ } else if (definitionName.equals(MINUS_PREFIX.getName())
+ && call.getChildren().get(0) instanceof CallExpression
+ && ((CallExpression)
call.getChildren().get(0)).getFunctionDefinition().getName()
+ .equals(COLUMNS.getName())) {
+
+ result =
resolveArgsOfColumns(call.getChildren().get(0).getChildren(), true);
+ } else {
+ List args = call.getChildren()
+ .stream()
+ .flatMap(c -> c.accept(this).stream())
+ .collect(Collectors.toList());
+ result = Collections.singletonList(new
CallExpression(call.getFunctionDefinition(), args));
+ }
+
+ return result;
+ }
+
+ @Override
+ protected List defaultMethod(Expression expression) {
+ return Collections.singletonList(expression);
+ }
+
+ /**
+* Expand the columns expression in the input Expression List.
+*/
+ private List resolveArgsOfColumns(List args,
boolean isReverseProjection) {
+
+ List finalResult = new LinkedList<>();
+ List result = args.stream()
+ .flatMap(e -> resolveSingleArg(e).stream())
+ .collect(Collectors.toList());
+
+ if (isReverseProjection) {
+ for (UnresolvedFieldReferenceExpression field:
inputFieldReferences) {
+ if (indexOfName(result, field.getName()) == -1)
{
+ finalResult.add(field);
+ }
+ }
+ } else {
+ finalResult.addAll(result);
+ }
+
+ return finalResult;
+ }
+
+ /**
+* Expand the columns expression in the input Expression.
+*/
+ private List
resolveSingleArg(Expression arg) {
+
+ List result;
+
+ if (arg instanceof ValueLiteralExpression &&
+
IntegerTypeInfo.class.isAssignableFrom(((ValueLiteralExpression)
arg).getType().getClass())) {
+ result =
Collections.singletonList(inputFieldReferences.get((int)
((ValueLiteralExpression) arg).getValue() - 1));
+ } else if (arg instanceof
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271551611
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/tableImpl.scala
##
@@ -630,18 +672,25 @@ class OverWindowedTableImpl(
overWindows)
}
+ private def expandColumnsInOverWindow(overWindows: Seq[OverWindow]):
Seq[OverWindow] = {
+overWindows.map { e =>
+ val expanedPartitioning = tableImpl.resolveCalls(e.getPartitioning)
Review comment:
`expaned` -> `expanded`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271550350
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/tableImpl.scala
##
@@ -568,16 +589,37 @@ class WindowGroupedTableImpl(
override def select(fields: Expression*): Table = {
selectInternal(
groupKeys,
- window,
+ expandColumnsInGroupWindow(window),
fields)
}
+ private def expandColumnsInGroupWindow(window: GroupWindow): GroupWindow = {
+
+val resolvedTimeField = tableImpl.resolveCalls(Seq(window.getTimeField))
+assert(resolvedTimeField.size() == 1, "TimeField of window only support
one column.")
Review comment:
`TimeField of window only support one column.` -> `Group Window only
supports a single time field column.`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271549409
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/tableImpl.scala
##
@@ -477,6 +484,18 @@ class TableImpl(
private def wrap(operation: TableOperation): Table = {
new TableImpl(tableEnv, operation)
}
+
+ private[flink] def getOutputFieldReferences:
Seq[UnresolvedFieldReferenceExpression] = {
+operationTree.asInstanceOf[LogicalNode]
+ .output.map(a => new UnresolvedFieldReferenceExpression(a.name))
+ }
+
+ private[flink] def resolveCalls(fields: Seq[Expression]): Seq[Expression] = {
+val outputFieldReferences = operationTree.asInstanceOf[LogicalNode]
+ .output.map(a => new UnresolvedFieldReferenceExpression(a.name))
+val expander = new ColumnsOperationExpander(outputFieldReferences)
Review comment:
`outputFieldReferences ` -> `getOutputFieldReferences`
Then remove :
`val outputFieldReferences = operationTree.asInstanceOf[LogicalNode]
.output.map(a => new UnresolvedFieldReferenceExpression(a.name))`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271553097
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/tableImpl.scala
##
@@ -630,18 +672,25 @@ class OverWindowedTableImpl(
overWindows)
}
+ private def expandColumnsInOverWindow(overWindows: Seq[OverWindow]):
Seq[OverWindow] = {
+overWindows.map { e =>
+ val expanedPartitioning = tableImpl.resolveCalls(e.getPartitioning)
+ new OverWindow(e.getAlias, expanedPartitioning, e.getOrder,
e.getPreceding, e.getFollowing)
Review comment:
`e.getOrder` ->` tableImpl.resolveCalls(e.getOrder)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-12043) Null value check in array serializers classes
[
https://issues.apache.org/jira/browse/FLINK-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808292#comment-16808292
]
Quan Shi commented on FLINK-12043:
--
Hi, [~yunta]
I make a test on version 1.6.0. Here is my way:
# Define a map function, return type is "String[]"
# Do nothing in map fucntion, just return null
{color:#00}Then, Flink throw an exception, stack trace:{color}
{color:#00}
java.lang.NullPointerException at
org.apache.flink.api.common.typeutils.base.array.StringArraySerializer.copy(StringArraySerializer.java:54)
at
org.apache.flink.api.common.typeutils.base.array.StringArraySerializer.copy(StringArraySerializer.java:33)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:577)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
at
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at
java.lang.Thread.run(Thread.java:748)
{color}
{color:#00} {color}
I could test on 1.7.x if needed.
> Null value check in array serializers classes
> -
>
> Key: FLINK-12043
> URL: https://issues.apache.org/jira/browse/FLINK-12043
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.7.2
>Reporter: Quan Shi
>Assignee: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Null pointer exception when get length of "_from"_ if _"from"_ is null in
> copy() method:
>
> Involved classes:
> {code:java}
> // code placeholder
> public String[] copy(String[] from) {
>String[] target = new String[from.length];
>System.arraycopy(from, 0, target, 0, from.length);
>return target;
> }
> {code}
> Involved serializer classes in package
> "org.apache.flink.api.common.typeutils.base.array"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11912) Expose per partition Kafka lag metric in Flink Kafka connector
[ https://issues.apache.org/jira/browse/FLINK-11912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808287#comment-16808287 ] Jiangjie Qin commented on FLINK-11912: -- [~suez1224] Thanks for the explanation. Got it. I think this approach works. The potential metric leak is an orthogonal existing issue that we can resolve separately. > Expose per partition Kafka lag metric in Flink Kafka connector > -- > > Key: FLINK-11912 > URL: https://issues.apache.org/jira/browse/FLINK-11912 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Affects Versions: 1.6.4, 1.7.2 >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Major > > In production, it's important that we expose the Kafka lag by partition > metric in order for users to diagnose which Kafka partition is lagging. > However, although the Kafka lag by partition metrics are available in > KafkaConsumer after 0.10.2, Flink was not able to properly register it > because the metrics are only available after the consumer start polling data > from partitions. I would suggest the following fix: > 1) In KafkaConsumerThread.run(), allocate a manualRegisteredMetricSet. > 2) in the fetch loop, as KafkaConsumer discovers new partitions, manually add > MetricName for those partitions that we want to register into > manualRegisteredMetricSet. > 3) in the fetch loop, check if manualRegisteredMetricSet is empty. If not, > try to search for the metrics available in KafkaConsumer, and if found, > register it and remove the entry from manualRegisteredMetricSet. > The overhead of the above approach is bounded and only incur when discovering > new partitions, and registration is done once the KafkaConsumer have the > metrics exposed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-12086) AsyncFunction - Add access to a user defined Object for cleanup on timeout
[
https://issues.apache.org/jira/browse/FLINK-12086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808282#comment-16808282
]
Guowei Ma edited comment on FLINK-12086 at 4/3/19 1:30 AM:
---
I think there might be another way to access user object other than change the
API.
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction {
private static final long serialVersionUID = 1L;
private HashMap, Future> libFutures = new
HashMap();
@Override
public void asyncInvoke(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = null; // submit something in a library
thread-pool
libFutures.put(resultFuture, future);
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
libFutures.remove(resultFuture);
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = libFutures.remove(resultFuture);
if (future != null) {
future.cancel(true);
}
resultFuture.complete(Collections.emptySet());
}
}
{code}
was (Author: maguowei):
I think there might be another way to access user object other than change the
API.
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction {
private static final long serialVersionUID = 1L;
private HashMap, Future> libFutures = new
HashMap();
@Override
public void asyncInvoke(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = null; // submit something in a library
thread-pool
libFutures.put(resultFuture, future);
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
libFutures.remove(resultFuture);
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = libFutures.remove(resultFuture);
if (future != null) {
future.cancel(true);
}
resultFuture.complete(Collections.emptySet());
}
}
{code}
> AsyncFunction - Add access to a user defined Object for cleanup on timeout
> --
>
> Key: FLINK-12086
> URL: https://issues.apache.org/jira/browse/FLINK-12086
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Felix Wollschläger
>Priority: Major
>
> When executing async-requests it would be nice to have access to a user
> defined object to perform cleanup when the process times out.
> For example, when executing Cassandra-Queries I'm using the drivers
> threadpool to submit Statements, which returns a
> com.datastax.driver.core.ResultSetFutre (
> [https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ResultSetFuture.html]
> ). When I run into a timeout I could cancel the Future because waiting for
> it to complete is unnecessary in that case.
>
> The API could be extendend to something like this:
>
> Adding an Type-Parameter to the AsnyFunction Interface:
> {code:java}
> AsyncFunction{code}
> Updating the asnyInvoke-Method to return the user-defined object:
> {code:java}
> T asyncInvoke(IN input, ResultFuture future) throws Exception;{code}
> Updating the timeout-Method to accept the user-defined object:
> {code:java}
> void timeout(IN input, T obj, ResultFuture resultFuture) throws
> Exception{code}
>
> An example Implementation could look like this:
> {code:java}
> package dev.codeflush;
> import org.apache.flink.streaming.api.functions.async.AsyncFunction;
>
[jira] [Commented] (FLINK-12086) AsyncFunction - Add access to a user defined Object for cleanup on timeout
[
https://issues.apache.org/jira/browse/FLINK-12086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808282#comment-16808282
]
Guowei Ma commented on FLINK-12086:
---
I think there might be another way to access user object other than change the
API.
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction {
private static final long serialVersionUID = 1L;
private HashMap, Future> libFutures = new
HashMap();
@Override
public void asyncInvoke(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = null; // submit something in a library
thread-pool
libFutures.put(resultFuture, future);
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
libFutures.remove(resultFuture);
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, ResultFuture resultFuture)
throws Exception {
Future future = libFutures.remove(resultFuture);
if (future != null) {
future.cancel(true);
}
resultFuture.complete(Collections.emptySet());
}
}
{code}
> AsyncFunction - Add access to a user defined Object for cleanup on timeout
> --
>
> Key: FLINK-12086
> URL: https://issues.apache.org/jira/browse/FLINK-12086
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Felix Wollschläger
>Priority: Major
>
> When executing async-requests it would be nice to have access to a user
> defined object to perform cleanup when the process times out.
> For example, when executing Cassandra-Queries I'm using the drivers
> threadpool to submit Statements, which returns a
> com.datastax.driver.core.ResultSetFutre (
> [https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ResultSetFuture.html]
> ). When I run into a timeout I could cancel the Future because waiting for
> it to complete is unnecessary in that case.
>
> The API could be extendend to something like this:
>
> Adding an Type-Parameter to the AsnyFunction Interface:
> {code:java}
> AsyncFunction{code}
> Updating the asnyInvoke-Method to return the user-defined object:
> {code:java}
> T asyncInvoke(IN input, ResultFuture future) throws Exception;{code}
> Updating the timeout-Method to accept the user-defined object:
> {code:java}
> void timeout(IN input, T obj, ResultFuture resultFuture) throws
> Exception{code}
>
> An example Implementation could look like this:
> {code:java}
> package dev.codeflush;
> import org.apache.flink.streaming.api.functions.async.AsyncFunction;
> import org.apache.flink.streaming.api.functions.async.ResultFuture;
> import java.util.Collections;
> import java.util.concurrent.CompletableFuture;
> import java.util.concurrent.ExecutionException;
> import java.util.concurrent.Future;
> public class SomeAsyncFunction implements AsyncFunction Future> {
> private static final long serialVersionUID = 1L;
>
> @Override
> public Future asyncInvoke(Integer input, ResultFuture
> resultFuture) throws Exception {
> Future future = null; // submit something in a library
> thread-pool
> CompletableFuture.runAsync(() -> {
> try {
> resultFuture.complete(Collections.singleton(future.get()));
> } catch (ExecutionException e) {
> // handle this
> } catch (InterruptedException e) {
> // handle that
> }
> });
>
> return future;
> }
> @Override
> public void timeout(Integer input, Future future,
> ResultFuture resultFuture) throws Exception {
> future.cancel(true);
> resultFuture.complete(Collections.emptySet());
> }
> }
> {code}
> As it currently is, submitted tasks in the asnyInvoke-Method will use
> resources (Threads, IO) even if the application is no longer in a state where
> it could do something meaningful with the result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-12041) Introduce ResettableExternalBuffer to blink batch
[ https://issues.apache.org/jira/browse/FLINK-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-12041. -- Resolution: Fixed Fix Version/s: 1.9.0 fixed in 3b817a03f49f252480a13c20098371fb3a98566c > Introduce ResettableExternalBuffer to blink batch > - > > Key: FLINK-12041 > URL: https://issues.apache.org/jira/browse/FLINK-12041 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Introduce ResettableExternalBuffer: > A resettable external buffer for binary row. It stores records in memory and > spill to disk when memory is not enough. When the spill is completed, the > records are written to memory again. The returned iterator reads the data in > write order (read spilled records first). > # It supports infinite length. > # It can open multiple Iterators. > # It support new iterator with beginRow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] KurtYoung merged pull request #8066: [FLINK-12041][table-runtime-blink] Introduce ResettableExternalBuffer to blink batch
KurtYoung merged pull request #8066: [FLINK-12041][table-runtime-blink] Introduce ResettableExternalBuffer to blink batch URL: https://github.com/apache/flink/pull/8066 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12086) AsyncFunction - Add access to a user defined Object for cleanup on timeout
[
https://issues.apache.org/jira/browse/FLINK-12086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Wollschläger updated FLINK-12086:
---
Description:
When executing async-requests it would be nice to have access to a user defined
object to perform cleanup when the process times out.
For example, when executing Cassandra-Queries I'm using the drivers threadpool
to submit Statements, which returns a com.datastax.driver.core.ResultSetFutre (
[https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ResultSetFuture.html]
). When I run into a timeout I could cancel the Future because waiting for it
to complete is unnecessary in that case.
The API could be extendend to something like this:
Adding an Type-Parameter to the AsnyFunction Interface:
{code:java}
AsyncFunction{code}
Updating the asnyInvoke-Method to return the user-defined object:
{code:java}
T asyncInvoke(IN input, ResultFuture future) throws Exception;{code}
Updating the timeout-Method to accept the user-defined object:
{code:java}
void timeout(IN input, T obj, ResultFuture resultFuture) throws
Exception{code}
An example Implementation could look like this:
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction> {
private static final long serialVersionUID = 1L;
@Override
public Future asyncInvoke(Integer input, ResultFuture
resultFuture) throws Exception {
Future future = null; // submit something in a library
thread-pool
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, Future future,
ResultFuture resultFuture) throws Exception {
future.cancel(true);
resultFuture.complete(Collections.emptySet());
}
}
{code}
As it currently is, submitted tasks in the asnyInvoke-Method will use resources
(Threads, IO) even if the application is no longer in a state where it could do
something meaningful with the result.
was:
When executing async-requests it would be nice to have access to a user defined
object to perform cleanup when the process times out.
For example, when executing Cassandra-Queries I'm using the drivers threadpool
to submit Statements, which returns a com.datastax.driver.core.ResultSetFutre (
[https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ResultSetFuture.html]
). When I run into a timeout I could cancel the Future because waiting for it
to complete is unnecessary in that case.
The API could be extendend to something like this:
Adding an Type-Parameter to the AsnyFunction Interface:
{code:java}
AsyncFunction{code}
Updating the asnyInvoke-Method to return the user-defined object:
{code:java}
T asyncInvoke(IN input, ResultFuture future) throws Exception;{code}
Updating the timeout-Method to accept the user-defined object:
{code:java}
void timeout(IN input, T obj, ResultFuture resultFuture) throws
Exception{code}
An example Implementation could look like this:
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction> {
private static final long serialVersionUID = 1L;
@Override
public Future asyncInvoke(Integer input, ResultFuture
resultFuture) throws Exception {
Future future = null; // submit something in a library
thread-pool
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, Future future,
ResultFuture resultFuture) throws Exception {
future.cancel(true);
resultFuture.complete(Collections.emptySet());
}
}
{code}
> AsyncFunction - Add access to a user defined Object for cleanup on timeout
>
[jira] [Created] (FLINK-12086) AsyncFunction - Add access to a user defined Object for cleanup on timeout
Felix Wollschläger created FLINK-12086:
--
Summary: AsyncFunction - Add access to a user defined Object for
cleanup on timeout
Key: FLINK-12086
URL: https://issues.apache.org/jira/browse/FLINK-12086
Project: Flink
Issue Type: Improvement
Components: API / DataStream
Reporter: Felix Wollschläger
When executing async-requests it would be nice to have access to a user defined
object to perform cleanup when the process times out.
For example, when executing Cassandra-Queries I'm using the drivers threadpool
to submit Statements, which returns a com.datastax.driver.core.ResultSetFutre (
[https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ResultSetFuture.html]
). When I run into a timeout I could cancel the Future because waiting for it
to complete is unnecessary in that case.
The API could be extendend to something like this:
Adding an Type-Parameter to the AsnyFunction Interface:
{code:java}
AsyncFunction{code}
Updating the asnyInvoke-Method to return the user-defined object:
{code:java}
T asyncInvoke(IN input, ResultFuture future) throws Exception;{code}
Updating the timeout-Method to accept the user-defined object:
{code:java}
void timeout(IN input, T obj, ResultFuture resultFuture) throws
Exception{code}
An example Implementation could look like this:
{code:java}
package dev.codeflush;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
public class SomeAsyncFunction implements AsyncFunction> {
private static final long serialVersionUID = 1L;
@Override
public Future asyncInvoke(Integer input, ResultFuture
resultFuture) throws Exception {
Future future = null; // submit something in a library
thread-pool
CompletableFuture.runAsync(() -> {
try {
resultFuture.complete(Collections.singleton(future.get()));
} catch (ExecutionException e) {
// handle this
} catch (InterruptedException e) {
// handle that
}
});
return future;
}
@Override
public void timeout(Integer input, Future future,
ResultFuture resultFuture) throws Exception {
future.cancel(true);
resultFuture.complete(Collections.emptySet());
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-11240) Implement external catalog factory and descriptor
[ https://issues.apache.org/jira/browse/FLINK-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-11240. Resolution: Fixed Fix Version/s: 1.9.0 Fixed in 1.9.0: bd2dae698a068543ae270b8cb3cc5a858c8ffe88 > Implement external catalog factory and descriptor > - > > Key: FLINK-11240 > URL: https://issues.apache.org/jira/browse/FLINK-11240 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to the efforts done in FLINK-8240 and FLINK-8866, implement > descriptor-based loading of external catalogs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] asfgit closed pull request #7390: [FLINK-11240] [table] External Catalog Factory and Descriptor
asfgit closed pull request #7390: [FLINK-11240] [table] External Catalog Factory and Descriptor URL: https://github.com/apache/flink/pull/7390 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11240) Implement external catalog factory and descriptor
[ https://issues.apache.org/jira/browse/FLINK-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther updated FLINK-11240: - Issue Type: New Feature (was: Sub-task) Parent: (was: FLINK-10744) > Implement external catalog factory and descriptor > - > > Key: FLINK-11240 > URL: https://issues.apache.org/jira/browse/FLINK-11240 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Eron Wright >Assignee: Eron Wright >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Similar to the efforts done in FLINK-8240 and FLINK-8866, implement > descriptor-based loading of external catalogs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] flinkbot edited a comment on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment
flinkbot edited a comment on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment URL: https://github.com/apache/flink/pull/8095#issuecomment-478609770 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @azagrebin * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @azagrebin * ❓ 3. Needs [attention] from. * ✅ 4. The change fits into the overall [architecture]. - Approved by @azagrebin * ✅ 5. Overall code [quality] is good. - Approved by @azagrebin Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] azagrebin commented on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment
azagrebin commented on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment URL: https://github.com/apache/flink/pull/8095#issuecomment-479142968 @flinkbot approve all This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] azagrebin commented on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment
azagrebin commented on issue #8095: [FLINK-12078][network] Abstract TaskEventPublisher interface for simplifying NetworkEnvironment URL: https://github.com/apache/flink/pull/8095#issuecomment-479142820 @zhijiangW could you restart the timed out build in your CI and try to get a green build there? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #8098: [BP-1.7][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann closed pull request #8098: [BP-1.7][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #8097: [BP-1.8][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann commented on issue #8097: [BP-1.8][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8097#issuecomment-479105356 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #8098: [BP-1.7][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann commented on issue #8098: [BP-1.7][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8098#issuecomment-479105442 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #8097: [BP-1.8][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann closed pull request #8097: [BP-1.8][FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8097 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-12075) Not able to submit jobs on YARN when there's a firewall
[ https://issues.apache.org/jira/browse/FLINK-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-12075. --- Resolution: Fixed Fix Version/s: 1.7.3 Fixed via master: 03499035cc3e0c3e5a5307cadd9c8159f668047e 065306d85e693759af36aaccc91df97c90965da8 5210948a2666084e0834cd47c695e696722f426f 1.8.0: a1c568ba62b70f43ca07eeed754d34266fd94bd8 1b502f77fb8eb79c26d61dbd980013f30b65b0aa e12b64c64eaca89cdede48138a3d5fee8eac336b 1.7.3: b75463a2456f40df51b0cc5eb2ff712642b2629e 440f8b7b5f6ea334f466bbfbf9569e9a67391d28 > Not able to submit jobs on YARN when there's a firewall > --- > > Key: FLINK-12075 > URL: https://issues.apache.org/jira/browse/FLINK-12075 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.7.2, 1.8.0 >Reporter: Richard Deurwaarder >Assignee: Till Rohrmann >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.3, 1.8.0 > > Time Spent: 20m > Remaining Estimate: 0h > > If there is a firewall around the YARN cluster and the machine, submitting > flink job it is unpractical because new flink clusters start up with random > ports for REST communication. > > FLINK-5758 should've fixed this. But it seems FLINK-11081 either undid the > changes or did not implement this. The relevant code is changed in > FLINK-11081 > ([https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102]) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] asfgit closed pull request #8096: [FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
asfgit closed pull request #8096: [FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8096 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #8096: [FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann commented on issue #8096: [FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified URL: https://github.com/apache/flink/pull/8096#issuecomment-479097604 Thanks for the review @walterddr and @uce. Merging after addressing Ufuk's comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-7730) TableFunction LEFT OUTER joins with ON predicates are broken
[
https://issues.apache.org/jira/browse/FLINK-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807900#comment-16807900
]
Haisheng Yuan commented on FLINK-7730:
--
The Calcite issue
[CALCITE-2004|https://issues.apache.org/jira/browse/CALCITE-2004] has been
fixed. The restrictions shall be removed when we upgrade Calcite to v1.20.0,
which will be released in next 2 or 3 months.
> TableFunction LEFT OUTER joins with ON predicates are broken
>
>
> Key: FLINK-7730
> URL: https://issues.apache.org/jira/browse/FLINK-7730
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.3.2, 1.4.0
>Reporter: Fabian Hueske
>Assignee: Xingcan Cui
>Priority: Critical
> Fix For: 1.3.0, 1.4.0
>
>
> TableFunction left outer joins with predicates in the ON clause are broken.
> Apparently, the are no tests for this and it has never worked. I observed
> issues on several layers:
> - Table Function does not correctly validate equality predicate:
> {{leftOuterJoin(func1('c) as 'd, 'a.cast(Types.STRING) === 'd)}} is rejected
> because the predicate is not considered as an equality predicate (the cast
> needs to be pushed down).
> - Plans cannot be correctly translated: {{leftOuterJoin(func1('c) as 'd, 'c
> === 'd)}} gives an optimizer exception.
> - SQL queries get translated but produce incorrect results. For example
> {{SELECT a, b, c, d FROM MyTable LEFT OUTER JOIN LATERAL TABLE(tfunc(c)) AS
> T(d) ON d = c}} returns an empty result if the condition {{d = c}} never
> returns true. However, the outer side should be preserved and padded with
> nulls.
> So there seem to be many issues with table function outer joins. Especially,
> the wrong result produced by SQL queries need to be quickly fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #8096: [FLINK-12075][yarn] Set RestOptions.BIND_PORT only to 0 if not specified
tillrohrmann commented on a change in pull request #8096: [FLINK-12075][yarn]
Set RestOptions.BIND_PORT only to 0 if not specified
URL: https://github.com/apache/flink/pull/8096#discussion_r271383717
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/entrypoint/YarnEntrypointUtilsTest.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn.entrypoint;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.clusterframework.BootstrapTools;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.hadoop.yarn.api.ApplicationConstants;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for the {@link YarnEntrypointUtils}.
+ */
+public class YarnEntrypointUtilsTest extends TestLogger {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(YarnEntrypointUtilsTest.class);
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new
TemporaryFolder();
+
+ /**
+* Tests that the REST ports are correctly set when loading a {@link
Configuration}
+* with unspecified REST options.
+*/
+ @Test
+ public void testRestPortOptionsUnspecified() throws IOException {
+ final File workingDirectory = TEMPORARY_FOLDER.newFolder();
+ final Configuration initialConfiguration = new Configuration();
+
+ final Configuration configuration =
loadConfiguration(workingDirectory, initialConfiguration);
+
+ // having not specified the ports should set the rest bind port
to 0
+ assertThat(configuration.getString(RestOptions.BIND_PORT),
is(equalTo("0")));
+ }
+
+ /**
+* Tests that the binding REST port is set to the REST port if set.
+*/
+ @Test
+ public void testRestPortSpecified() throws IOException {
+ final File workingDirectory = TEMPORARY_FOLDER.newFolder();
+ final Configuration initialConfiguration = new Configuration();
+ final int port = 1337;
+ initialConfiguration.setInteger(RestOptions.PORT, port);
+
+ final Configuration configuration =
loadConfiguration(workingDirectory, initialConfiguration);
+
+ // if the bind port is not specified it should fall back to the
rest port
+ assertThat(configuration.getString(RestOptions.BIND_PORT),
is(equalTo(String.valueOf(port;
+ }
+
+ /**
+* Tests that the binding REST port has precedence over the REST port
if both are set.
+*/
+ @Test
+ public void testRestPortAndBindingPortSpecified() throws IOException {
+ final File workingDirectory = TEMPORARY_FOLDER.newFolder();
+ final Configuration initialConfiguration = new Configuration();
+ final int port = 1337;
+ final String bindingPortRange = "1337-7331";
+ initialConfiguration.setInteger(RestOptions.PORT, port);
+ initialConfiguration.setString(RestOptions.BIND_PORT,
bindingPortRange);
+
+ final Configuration configuration =
loadConfiguration(workingDirectory, initialConfiguration);
+
+ // bind port should have precedence over the rest port
+ assertThat(configuration.getString(RestOptions.BIND_PORT),
is(equalTo(bindingPortRange)));
+ }
+
+ @Nonnull
+ private Configuration loadConfiguration(File workingDirectory,
Configuration initialConfiguration) throws IOException {
Review comment:
Good points. Will change it.
This is an automated message from the Apache Git Service.
To
[jira] [Commented] (FLINK-7865) Remove predicate restrictions on TableFunction left outer join
[
https://issues.apache.org/jira/browse/FLINK-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807880#comment-16807880
]
Xingcan Cui commented on FLINK-7865:
Thanks for working on that, [~hyuan] ! I'll take care of this issue on Flink
side.
> Remove predicate restrictions on TableFunction left outer join
> --
>
> Key: FLINK-7865
> URL: https://issues.apache.org/jira/browse/FLINK-7865
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Major
>
> To cover up the improper translation of lateral table left outer join
> (CALCITE-2004), we have temporarily forbidden the predicates (except {{true}}
> literal) in Table API (FLINK-7853) and SQL (FLINK-7854). Once the issue has
> been fixed in Calcite, we should remove the restrictions. The tasks may
> include removing Table API/SQL condition check, removing validation tests,
> enabling integration tests, updating the documents, etc.
> See [this thread on Calcite dev
> list|https://lists.apache.org/thread.html/16caeb8b1649c4da85f9915ea723c6c5b3ced0b96914cadc24ee4e15@%3Cdev.calcite.apache.org%3E]
> for more information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7865) Remove predicate restrictions on TableFunction left outer join
[
https://issues.apache.org/jira/browse/FLINK-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807877#comment-16807877
]
Haisheng Yuan commented on FLINK-7865:
--
The blocker is resolved in Calcite 1.20.
> Remove predicate restrictions on TableFunction left outer join
> --
>
> Key: FLINK-7865
> URL: https://issues.apache.org/jira/browse/FLINK-7865
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Major
>
> To cover up the improper translation of lateral table left outer join
> (CALCITE-2004), we have temporarily forbidden the predicates (except {{true}}
> literal) in Table API (FLINK-7853) and SQL (FLINK-7854). Once the issue has
> been fixed in Calcite, we should remove the restrictions. The tasks may
> include removing Table API/SQL condition check, removing validation tests,
> enabling integration tests, updating the documents, etc.
> See [this thread on Calcite dev
> list|https://lists.apache.org/thread.html/16caeb8b1649c4da85f9915ea723c6c5b3ced0b96914cadc24ee4e15@%3Cdev.calcite.apache.org%3E]
> for more information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-7865) Remove predicate restrictions on TableFunction left outer join
[
https://issues.apache.org/jira/browse/FLINK-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui reassigned FLINK-7865:
--
Assignee: Xingcan Cui
> Remove predicate restrictions on TableFunction left outer join
> --
>
> Key: FLINK-7865
> URL: https://issues.apache.org/jira/browse/FLINK-7865
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Major
>
> To cover up the improper translation of lateral table left outer join
> (CALCITE-2004), we have temporarily forbidden the predicates (except {{true}}
> literal) in Table API (FLINK-7853) and SQL (FLINK-7854). Once the issue has
> been fixed in Calcite, we should remove the restrictions. The tasks may
> include removing Table API/SQL condition check, removing validation tests,
> enabling integration tests, updating the documents, etc.
> See [this thread on Calcite dev
> list|https://lists.apache.org/thread.html/16caeb8b1649c4da85f9915ea723c6c5b3ced0b96914cadc24ee4e15@%3Cdev.calcite.apache.org%3E]
> for more information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12085) Improve Mesos job cluster entry point 'MesosJobClusterEntrypoint'
Jacky Yin created FLINK-12085: - Summary: Improve Mesos job cluster entry point 'MesosJobClusterEntrypoint' Key: FLINK-12085 URL: https://issues.apache.org/jira/browse/FLINK-12085 Project: Flink Issue Type: Improvement Components: Deployment / Mesos Affects Versions: 1.7.2 Reporter: Jacky Yin Assignee: Jacky Yin Currently, the Mesos job cluster is not that easy to use. You have to manually serialize the job graph and specify the job graph file path. And the only way to include the user code jars is to put them in the `lib` folder. `StandaloneJobClusterEntrypoint` is a good example to learn. It can include the user code jars and allow to specify the entry class. That way we would not need to generate the JobGraph yourself and then serialize it. I would like to enhance `MesosJobClusterEntrypoint` similarly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271361144
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
I would be strongly in favor merging this after #8062. This PR introduces a
new complex feature. If we merge it before #8062 it immediately introduces
technical debt. It also mixes once again responsibilities of the `TableImpl`
with column resolution, which we should limit as much as possible,
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271361144
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
I would be strongly in favor merging this after #8062. This PR introduces a
new complex feature. If we merge it before #8062 it immediately introduces
technical debt.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dawidwys merged pull request #8080: [hotfix][runtime] Improve exception message in SingleInputGate
dawidwys merged pull request #8080: [hotfix][runtime] Improve exception message in SingleInputGate URL: https://github.com/apache/flink/pull/8080 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12014) Flink CEP Doc missing "SkipToNextStrategy"
[ https://issues.apache.org/jira/browse/FLINK-12014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-12014. Resolution: Fixed Fix Version/s: 1.9.0 Fixed in 8a56b8943865d42fa6479e5f3cc5bbd1f210e502 > Flink CEP Doc missing "SkipToNextStrategy" > -- > > Key: FLINK-12014 > URL: https://issues.apache.org/jira/browse/FLINK-12014 > Project: Flink > Issue Type: Bug > Components: Documentation, Library / CEP >Reporter: Tom Goong >Assignee: Tom Goong >Priority: Minor > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > in cep.md file，The following description misses *SkipToNextStrategy：* > > *After Match Skip Strategy* > For a given pattern, the same event may be assigned to multiple successful > matches. To control to how many matches an event will be assigned, you need > to specify the skip strategy called `AfterMatchSkipStrategy`. There are four > types of skip strategies, listed as follows: > * *NO_SKIP*: Every possible match will be emitted. > * *SKIP_PAST_LAST_EVENT*: Discards every partial match that > started after the match started but before it ended. > * *SKIP_TO_FIRST*: Discards every partial match that > started after the match started but before the first event of *PatternName* > occurred. > * *SKIP_TO_LAST*: Discards every partial match that started > after the match started but before the last event of *PatternName* occurred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12014) Flink CEP Doc missing "SkipToNextStrategy"
[ https://issues.apache.org/jira/browse/FLINK-12014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-12014: - Priority: Minor (was: Major) > Flink CEP Doc missing "SkipToNextStrategy" > -- > > Key: FLINK-12014 > URL: https://issues.apache.org/jira/browse/FLINK-12014 > Project: Flink > Issue Type: Bug > Components: Documentation, Library / CEP >Reporter: Tom Goong >Assignee: Tom Goong >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > in cep.md file，The following description misses *SkipToNextStrategy：* > > *After Match Skip Strategy* > For a given pattern, the same event may be assigned to multiple successful > matches. To control to how many matches an event will be assigned, you need > to specify the skip strategy called `AfterMatchSkipStrategy`. There are four > types of skip strategies, listed as follows: > * *NO_SKIP*: Every possible match will be emitted. > * *SKIP_PAST_LAST_EVENT*: Discards every partial match that > started after the match started but before it ended. > * *SKIP_TO_FIRST*: Discards every partial match that > started after the match started but before the first event of *PatternName* > occurred. > * *SKIP_TO_LAST*: Discards every partial match that started > after the match started but before the last event of *PatternName* occurred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] dawidwys merged pull request #8049: [FLINK-12014][Doc]Flink CEP Doc missing "SkipToNextStrategy"
dawidwys merged pull request #8049: [FLINK-12014][Doc]Flink CEP Doc missing "SkipToNextStrategy" URL: https://github.com/apache/flink/pull/8049 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dawidwys commented on issue #8049: [FLINK-12014][Doc]Flink CEP Doc missing "SkipToNextStrategy"
dawidwys commented on issue #8049: [FLINK-12014][Doc]Flink CEP Doc missing "SkipToNextStrategy" URL: https://github.com/apache/flink/pull/8049#issuecomment-479041274 Sorry @Tom-Goong, my mistake, didn't spot it. +1, merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12020) Add documentation for mesos-appmaster-job.sh
[
https://issues.apache.org/jira/browse/FLINK-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-12020.
-
Resolution: Fixed
Fix Version/s: 1.8.0
1.7.3
Fixed via
1.9.0: b36bca8aa1ceb018ecda47387ab4447941485dae
1.8.0: 057f7fad49d4480ff9630975724dfa3937790c32
1.7.3: 4b784b256b7884fa6f9070fab4e7ca1e36587d1c
> Add documentation for mesos-appmaster-job.sh
>
>
> Key: FLINK-12020
> URL: https://issues.apache.org/jira/browse/FLINK-12020
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Mesos, Documentation
>Affects Versions: 1.7.2, 1.8.0
>Reporter: Till Rohrmann
>Assignee: Jacky Yin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.7.3, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Flink documentation is currently lacking information about the
> {{mesos-appmaster-job.sh}} and how to use it. It would be helpful for our
> users to add documentation and examples how to use it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] asfgit closed pull request #8084: [FLINK-12020] [Deployment/Mesos/Documentation] Add documentation for mesos-appmaster-job.sh
asfgit closed pull request #8084: [FLINK-12020] [Deployment/Mesos/Documentation] Add documentation for mesos-appmaster-job.sh URL: https://github.com/apache/flink/pull/8084 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12075) Not able to submit jobs on YARN when there's a firewall
[ https://issues.apache.org/jira/browse/FLINK-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807818#comment-16807818 ] Aljoscha Krettek commented on FLINK-12075: -- Thanks for testing this on short notice, [~Xeli]! > Not able to submit jobs on YARN when there's a firewall > --- > > Key: FLINK-12075 > URL: https://issues.apache.org/jira/browse/FLINK-12075 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.7.2, 1.8.0 >Reporter: Richard Deurwaarder >Assignee: Till Rohrmann >Priority: Blocker > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > If there is a firewall around the YARN cluster and the machine, submitting > flink job it is unpractical because new flink clusters start up with random > ports for REST communication. > > FLINK-5758 should've fixed this. But it seems FLINK-11081 either undid the > changes or did not implement this. The relevant code is changed in > FLINK-11081 > ([https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102]) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12083) Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which has already added into classpath.
[ https://issues.apache.org/jira/browse/FLINK-12083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kelun wang closed FLINK-12083. -- Resolution: Invalid > Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which > has already added into classpath. > > > Key: FLINK-12083 > URL: https://issues.apache.org/jira/browse/FLINK-12083 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: shaded-6.0 >Reporter: kelun wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12083) Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which has already added into classpath.
[ https://issues.apache.org/jira/browse/FLINK-12083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kelun wang updated FLINK-12083: --- Attachment: (was: image-2019-04-02-22-24-40-700.png) > Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which > has already added into classpath. > > > Key: FLINK-12083 > URL: https://issues.apache.org/jira/browse/FLINK-12083 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: shaded-6.0 >Reporter: kelun wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12083) Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which has already added into classpath.
[ https://issues.apache.org/jira/browse/FLINK-12083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kelun wang updated FLINK-12083: --- Description: (was: Hello, I'm running a flink job that write data into hbase ,but one of the task（slot） failed to read hbase-site.xml in the taskmanager, then use the default zk config， erro message like below !image-2019-04-02-22-24-40-700.png! which makes the task failed to connect zk，does any one who ever met this？ ) > Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which > has already added into classpath. > > > Key: FLINK-12083 > URL: https://issues.apache.org/jira/browse/FLINK-12083 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: shaded-6.0 >Reporter: kelun wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12084) Flink API extensions do not seems to include split method
Jacopo Gobbi created FLINK-12084: Summary: Flink API extensions do not seems to include split method Key: FLINK-12084 URL: https://issues.apache.org/jira/browse/FLINK-12084 Project: Flink Issue Type: Wish Components: API / Scala Affects Versions: 1.7.2 Reporter: Jacopo Gobbi In the list of Scala API extensions Flink does not seems to provide a method for the split function to be deconstructed using partial function. https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/scala_api_extensions.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12083) Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which has already added into classpath.
kelun wang created FLINK-12083: -- Summary: Sink into Hbase, one of the task(slot) failed to read hbase-site.xml which has already added into classpath. Key: FLINK-12083 URL: https://issues.apache.org/jira/browse/FLINK-12083 Project: Flink Issue Type: Bug Components: API / DataStream Affects Versions: shaded-6.0 Reporter: kelun wang Attachments: image-2019-04-02-22-24-40-700.png Hello, I'm running a flink job that write data into hbase ,but one of the task（slot） failed to read hbase-site.xml in the taskmanager, then use the default zk config， erro message like below !image-2019-04-02-22-24-40-700.png! which makes the task failed to connect zk，does any one who ever met this？ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] hequn8128 commented on issue #8087: [FLINK-12029][table] Add column operations for TableApi
hequn8128 commented on issue #8087: [FLINK-12029][table] Add column operations for TableApi URL: https://github.com/apache/flink/pull/8087#issuecomment-479021520 @sunjincheng121 Thanks for your valuable suggestions. It makes the PR much better. I have addressed all your comments and updated the PR. Would be great if you can take another look. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-12037) High parallelism E2E test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-12037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806596#comment-16806596
]
Guowei Ma edited comment on FLINK-12037 at 4/2/19 1:29 PM:
---
>From the log:
The TaskManager(b15f5702d9e1cc57e353824b3ec4b792) doesn't get response from jm
after it offer its' slot. Since that the taskmanager release the slot and close
the connection with jm after 10s.
For TM
11:38:11,436 this taskmanger b15f5702d9e1cc57e353824b3ec4b792 is started
11:40:36,761 Receive slot request
11:40:43,972 Establish JobManager connection for job
f149b50ac8c6291371ce54b30fb0a072
11:40:43,989 Offer reserved slots to the leader of job
11:40:46,766 Free slot TaskSlot
During 11:40:43~11:40:46 JM is mainly deploying the task.
There is no obvious information about what the JM is doing. There is also no
obvious information that "Deploying" is time consuming.
!image-2019-04-02-18-30-20-634.png!
Some deployings have 100ms interval
was (Author: maguowei):
>From the log:
The TaskManager(b15f5702d9e1cc57e353824b3ec4b792) doesn't get response from jm
after it offer its' slot. Since that the taskmanager release the slot and close
the connection with jm after 10s.
For TM
11:38:11,436 this taskmanger b15f5702d9e1cc57e353824b3ec4b792 is started
11:40:36,761 Receive slot request
11:40:43,972 Establish JobManager connection for job
f149b50ac8c6291371ce54b30fb0a072
11:40:43,989 Offer reserved slots to the leader of job
11:40:46,766 Free slot TaskSlot
During 11:40:43~11:40:46 JM is mainly deploying the task.
!image-2019-04-02-18-30-20-634.png!
Some deployings have 100ms interval
> High parallelism E2E test failed on Travis
> --
>
> Key: FLINK-12037
> URL: https://issues.apache.org/jira/browse/FLINK-12037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Tests
>Affects Versions: 1.8.1
>Reporter: Chesnay Schepler
>Priority: Major
> Labels: test-stability
> Attachments: image-2019-04-02-18-30-20-634.png
>
>
> https://travis-ci.org/apache/flink/jobs/511335563
> {code}
> org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: f149b50ac8c6291371ce54b30fb0a072)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:483)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:471)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
> at
> org.apache.flink.batch.HighParallelismIterationsTestProgram.main(HighParallelismIterationsTestProgram.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
> ... 22 more
> Caused by: org.apache.flink.util.FlinkException: TaskExecutor
> akka.tcp://flink@127.0.1.1:37009/user/taskmanager_0 has no more allocated
>
[GitHub] [flink] pengkangjing commented on a change in pull request #8060: [FLINK-12021] Deploy execution in topological sorted order
pengkangjing commented on a change in pull request #8060: [FLINK-12021] Deploy
execution in topological sorted order
URL: https://github.com/apache/flink/pull/8060#discussion_r271297840
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/concurrent/FutureUtils.java
##
@@ -598,8 +593,11 @@ private void handleCompletedFuture(T value, Throwable
throwable) {
complete(Collections.emptyList());
}
else {
+ int counter = 0;
for (CompletableFuture future :
resultFutures) {
-
future.whenComplete(this::handleCompletedFuture);
+ final int index = counter;
+ counter++;
+ future.whenComplete((value, throwable)
-> handleCompletedFuture(index, value, throwable));
Review comment:
test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pengkangjing commented on a change in pull request #8060: [FLINK-12021] Deploy execution in topological sorted order
pengkangjing commented on a change in pull request #8060: [FLINK-12021] Deploy
execution in topological sorted order
URL: https://github.com/apache/flink/pull/8060#discussion_r271297840
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/concurrent/FutureUtils.java
##
@@ -598,8 +593,11 @@ private void handleCompletedFuture(T value, Throwable
throwable) {
complete(Collections.emptyList());
}
else {
+ int counter = 0;
for (CompletableFuture future :
resultFutures) {
-
future.whenComplete(this::handleCompletedFuture);
+ final int index = counter;
+ counter++;
+ future.whenComplete((value, throwable)
-> handleCompletedFuture(index, value, throwable));
Review comment:
test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271286544
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
Make sense for my side. That depends on which PR be merged first. another
should rebase code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271282393
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
@sunjincheng121 @dawidwys Thanks a lot for your advice. Maybe we can keep
`ColumnsOperationExpander` here for now and replace it to a ResolverRule once
#8062 is merged?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271279312
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/BuiltInFunctionDefinitions.java
##
@@ -319,6 +319,14 @@
public static final FunctionDefinition CURRENT_ROW =
new FunctionDefinition("currentRow", OTHER_FUNCTION);
+ // columns
+ public static final FunctionDefinition COLUMNS =
+ new FunctionDefinition("columns", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_RANGE =
Review comment:
+1 for `RANG_TO`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8101: [hotfix][docs] fix error in Process Function Doc
flinkbot commented on issue #8101: [hotfix][docs] fix error in Process Function Doc URL: https://github.com/apache/flink/pull/8101#issuecomment-478966308 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zsq12138 opened a new pull request #8101: [hotfix][docs] fix error in Process Function Doc
zsq12138 opened a new pull request #8101: [hotfix][docs] fix error in Process Function Doc URL: https://github.com/apache/flink/pull/8101 ## What is the purpose of the change In the example given,it used the `ProcessFunction` on a keyed stream. But in the new version, it has been deprecated and it has been replaced with `KeyedProcessFunction`. The example code also has some errors in the Scala snippet. Using the original scala example can cause unnecessary problems for users. ## Brief change log - *Replace the deprecated `ProcessFunction` with `KeyedProcessFunction` And fix some errors in the Scala snippet* This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3
[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807691#comment-16807691
]
Tao Yang commented on FLINK-8801:
-
Hi, [~NicoK].
I have doubt about this PR which directly use modification time of local file
as the timestamp of local resource in YARN then set it as the modification time
for remote file via FileSystem#setTimes interface, but most file systems seem
not implement the FileSystem#setTimes including s3/s3a/s3n/azure/aliyun-oss
etc, so that inconsistency maybe exist between timestamp of local resource in
YARN and modification time of remote file which can cause problems.
Could you please help to solve this doubt? Thanks!
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.3, 1.5.0
>
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
dawidwys commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271265913
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
This should be another `ResolverRule` once #8062 is merged.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271262488
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/BuiltInFunctionDefinitions.java
##
@@ -319,6 +319,14 @@
public static final FunctionDefinition CURRENT_ROW =
new FunctionDefinition("currentRow", OTHER_FUNCTION);
+ // columns
+ public static final FunctionDefinition COLUMNS =
+ new FunctionDefinition("columns", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_RANGE =
Review comment:
Good idea! Maybe we can name it `RANG_TO`. We may add `RANGE_UNTIL` later.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271262488
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/BuiltInFunctionDefinitions.java
##
@@ -319,6 +319,14 @@
public static final FunctionDefinition CURRENT_ROW =
new FunctionDefinition("currentRow", OTHER_FUNCTION);
+ // columns
+ public static final FunctionDefinition COLUMNS =
+ new FunctionDefinition("columns", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_RANGE =
Review comment:
Maybe we can name it `RANG_TO`. We may add `RANGE_UNTIL` later. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
hequn8128 commented on a change in pull request #8087: [FLINK-12029][table] Add
column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271260519
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/BuiltInFunctionDefinitions.java
##
@@ -319,6 +319,14 @@
public static final FunctionDefinition CURRENT_ROW =
new FunctionDefinition("currentRow", OTHER_FUNCTION);
+ // columns
+ public static final FunctionDefinition COLUMNS =
+ new FunctionDefinition("columns", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_RANGE =
+ new FunctionDefinition("columnsRange", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_INVERSE_SELECTION =
+ new FunctionDefinition("columnsInverseSelection",
OTHER_FUNCTION);
+
Review comment:
Using `MINUS_PREFIX` directly is better. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271252502
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/ColumnsOperationExpander.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.expressions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.IntegerTypeInfo;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_INVERSE_SELECTION;
+import static
org.apache.flink.table.expressions.BuiltInFunctionDefinitions.COLUMNS_RANGE;
+
+/**
+ * Expands column expressions to it's real parent's input references.
+ */
+@Internal
+public class ColumnsOperationExpander extends
ApiExpressionDefaultVisitor> {
Review comment:
How about move `ColumnsOperationExpander` into `ApiExpressionUtils`, similar
as `FieldReferenceExtractor`. I think all of the Expression processes should
be placed there. But I feel that this division is not very obvious. Do you have
some suggestions for this? @dawidwys
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zsq12138 closed pull request #8091: [hotfix][docs] fix error in Process Function Doc
zsq12138 closed pull request #8091: [hotfix][docs] fix error in Process Function Doc URL: https://github.com/apache/flink/pull/8091 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8100: [FLINK-12082] Bump up the jython-standalone version
flinkbot commented on issue #8100: [FLINK-12082] Bump up the jython-standalone version URL: https://github.com/apache/flink/pull/8100#issuecomment-478954902 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271235635
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala
##
@@ -1126,6 +1137,12 @@ trait ImplicitExpressionConversions {
implicit def symbol2FieldExpression(sym: Symbol): Expression =
unresolvedFieldRef(sym.name)
+ implicit def scalaRange2Expression(range: Range.Inclusive): Expression = {
Review comment:
`scalaRange2Expression` -> `scalaRange2RangeExpression`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271233589
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/BuiltInFunctionDefinitions.java
##
@@ -319,6 +319,14 @@
public static final FunctionDefinition CURRENT_ROW =
new FunctionDefinition("currentRow", OTHER_FUNCTION);
+ // columns
+ public static final FunctionDefinition COLUMNS =
+ new FunctionDefinition("columns", OTHER_FUNCTION);
+ public static final FunctionDefinition COLUMNS_RANGE =
Review comment:
Does the `COLUMNS_RANGE` can be used in other cases? such as lambdas? If so
we may named `COMMON_RANGE`, but I am fine with `COLUMNS_RANGE` for the current
PR. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087:
[FLINK-12029][table] Add column operations for TableApi
URL: https://github.com/apache/flink/pull/8087#discussion_r271222041
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala
##
@@ -119,9 +119,15 @@ trait ImplicitExpressionOperations {
def unary_! : Expression = call(NOT, expr)
/**
-* Returns negative numeric.
+* Returns negative numeric or inverse selection of columns.
*/
- def unary_- : Expression = call(MINUS_PREFIX, expr)
+ def unary_- : Expression = {
Review comment:
I think we using `MINUS_PREFIX` directly. It's valuable to do this effort.
Is that makes sense to you?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-12082) Bump up the jython-standalone version
[ https://issues.apache.org/jira/browse/FLINK-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12082: --- Labels: pull-request-available (was: ) > Bump up the jython-standalone version > - > > Key: FLINK-12082 > URL: https://issues.apache.org/jira/browse/FLINK-12082 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: aloyszhang >Assignee: aloyszhang >Priority: Major > Labels: pull-request-available > > jyhont's CVE : > h1. [CVE-2016-4000|https://www.cvedetails.com/cve/CVE-2016-4000/] > > we can update to 2.7.1b3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [flink] sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi
sunjincheng121 commented on a change in pull request #8087: [FLINK-12029][table] Add column operations for TableApi URL: https://github.com/apache/flink/pull/8087#discussion_r271204798 ## File path: flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/api/tableImpl.scala ## @@ -30,8 +30,8 @@ import org.apache.flink.table.plan.OperationTreeBuilder import org.apache.flink.table.plan.logical._ import org.apache.flink.table.util.JavaScalaConversionUtil.toJava -import _root_.scala.collection.JavaConversions._ import _root_.scala.collection.JavaConverters._ +import _root_.scala.collection.JavaConversions._ Review comment: useless change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
