[jira] [Commented] (FLINK-7386) Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490254#comment-16490254
 ] 

ASF GitHub Bot commented on FLINK-7386:
---

Github user cjolif commented on the issue:

https://github.com/apache/flink/pull/6043
  
> When planning to switch to REST, are we speaking of an implementation 
that works directly against Elasticsearch's REST API? Or are we thinking of 
using Elasticsearch's RestHighLevelClient?

To me at least, yes, when I say REST I mean RestHighLevelClient.


> Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ 
> client
> 
>
> Key: FLINK-7386
> URL: https://issues.apache.org/jira/browse/FLINK-7386
> Project: Flink
>  Issue Type: Improvement
>  Components: ElasticSearch Connector
>Reporter: Dawid Wysakowicz
>Assignee: Fang Yong
>Priority: Critical
> Fix For: 1.5.0
>
>
> In Elasticsearch 5.2.0 client the class {{BulkProcessor}} was refactored and 
> has no longer the method {{add(ActionRequest)}}.
> For more info see: https://github.com/elastic/elasticsearch/pull/20109



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6043: [FLINK-7386] evolve RequestIndexer API to make it working...

[cjolif]

Github user cjolif commented on the issue:

https://github.com/apache/flink/pull/6043
  
> When planning to switch to REST, are we speaking of an implementation 
that works directly against Elasticsearch's REST API? Or are we thinking of 
using Elasticsearch's RestHighLevelClient?

To me at least, yes, when I say REST I mean RestHighLevelClient.


---

[jira] [Commented] (FLINK-9407) Support orc rolling sink writer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490241#comment-16490241
 ] 

ASF GitHub Bot commented on FLINK-9407:
---

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/6075

[FLINK-9407] [hdfs connector] Support orc rolling sink writer

## What is the purpose of the change
In production environment, we often write file in orc format. So, this PR 
support a writer that for bucketingsink. And currently only basic data types 
are supported in this PR.

## Brief change log
Add ```OrcFileWriter``` and the test file.

## Verifying this change
Tests it with ```OrcFileWriterTest```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (don't know)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: ( no )
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (not documented now)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9407-orc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6075


commit d653d39f6cb74075874eb890cab36012d85dbecf
Author: zhangminglei 
Date:   2018-05-25T04:58:01Z

[FLINK-9407] [hdfs connector] Support orc rolling sink writer




> Support orc rolling sink writer
> ---
>
> Key: FLINK-9407
> URL: https://issues.apache.org/jira/browse/FLINK-9407
> Project: Flink
>  Issue Type: New Feature
>  Components: filesystem-connector
>Reporter: mingleizhang
>Assignee: mingleizhang
>Priority: Major
>
> Currently, we only support {{StringWriter}}, {{SequenceFileWriter}} and 
> {{AvroKeyValueSinkWriter}}. I would suggest add an orc writer for rolling 
> sink.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6075: [FLINK-9407] [hdfs connector] Support orc rolling ...

[zhangminglei]

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/6075

[FLINK-9407] [hdfs connector] Support orc rolling sink writer

## What is the purpose of the change
In production environment, we often write file in orc format. So, this PR 
support a writer that for bucketingsink. And currently only basic data types 
are supported in this PR.

## Brief change log
Add ```OrcFileWriter``` and the test file.

## Verifying this change
Tests it with ```OrcFileWriterTest```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (don't know)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: ( no )
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (not documented now)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9407-orc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6075


commit d653d39f6cb74075874eb890cab36012d85dbecf
Author: zhangminglei 
Date:   2018-05-25T04:58:01Z

[FLINK-9407] [hdfs connector] Support orc rolling sink writer




---

[jira] [Commented] (FLINK-8985) End-to-end test: CLI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490212#comment-16490212
 ] 

ASF GitHub Bot commented on FLINK-8985:
---

Github user walterddr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5863#discussion_r190788353
  
--- Diff: flink-end-to-end-tests/test-scripts/test_cli_api.sh ---
@@ -0,0 +1,196 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+
+start_cluster
+
+# Test for CLI commands.
+# verify only the return code the content correctness of the API results.

+PERIODIC_JOB_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-api-test/target/PeriodicStreamingJob.jar
+JOB_ID_REGEX_EXTRACTOR=".*JobID ([0-9,a-f]*)"
+SAVE_POINT_REGEX_EXTRACTOR=".*Savepoint stored in (.*)\\."
+JOB_INFO_PACT_DATA_SOURCE_REGEX_EXTRACTOR="\"pact\": \"(Data Source)\""
+JOB_INFO_PACT_DATA_SINK_REGEX_EXTRACTOR="\"pact\": \"(Data Sink)\""
+JOB_LIST_REGEX_EXTRACTOR_BY_STATUS="([0-9,a-f]*) :"
+
+EXIT_CODE=0
+
+function extract_job_id_from_job_submission_return() {
+if [[ $1 =~ $JOB_ID_REGEX_EXTRACTOR ]];
+then
+JOB_ID="${BASH_REMATCH[1]}";
+else
+JOB_ID=""
+fi
+echo "$JOB_ID"
+}
+
+function extract_savepoint_path_from_savepoint_return() {
+if [[ $1 =~ $SAVE_POINT_REGEX_EXTRACTOR ]];
+then
+SAVEPOINT_PATH="${BASH_REMATCH[1]}";
+else
+SAVEPOINT_PATH=""
+fi
+echo "$SAVEPOINT_PATH"
+}
+
+function extract_valid_pact_from_job_info_return() {
+PACT_MATCH=0
+if [[ $1 =~ $JOB_INFO_PACT_DATA_SOURCE_REGEX_EXTRACTOR ]];
+then
+PACT_MATCH=$PACT_MATCH
+else
+PACT_MATCH=-1
+fi
+if [[ $1 =~ $JOB_INFO_PACT_DATA_SINK_REGEX_EXTRACTOR ]];
+then
+PACT_MATCH=$PACT_MATCH
+else
+PACT_MATCH=-1
+fi
+echo ${PACT_MATCH}
+}
+
+function extract_valid_job_list_by_type_from_job_list_return() {
+JOB_LIST_MATCH=0
+JOB_LIST_REGEX_EXTRACTOR="$JOB_LIST_REGEX_EXTRACTOR_BY_STATUS $2 $3"
+if [[ $1 =~ $JOB_LIST_REGEX_EXTRACTOR ]];
+then
+JOB_LIST_MATCH=$JOB_LIST_MATCH
+else
+JOB_LIST_MATCH=-1
+fi
+echo ${JOB_LIST_MATCH}
+}
+
+function cleanup_cli_test() {
+  stop_cluster
+  $FLINK_DIR/bin/taskmanager.sh stop-all
--- End diff --

checking the clean up code seems like it doesn't explicitly call stopping 
all tm (similar to `start-cluster.sh`, I need to explicitly call `taskmanager 
start`). I should remove only the `stop-cluster` actually.


> End-to-end test: CLI
> 
>
> Key: FLINK-8985
> URL: https://issues.apache.org/jira/browse/FLINK-8985
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Rong Rong
>Priority: Major
>
> We should an end-to-end test which verifies that all client commands are 
> working correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5863: [FLINK-8985][e2etest] initial support for End-to-e...

[walterddr]

Github user walterddr commented on a diff in the pull request:

https://github.com/apache/flink/pull/5863#discussion_r190788353
  
--- Diff: flink-end-to-end-tests/test-scripts/test_cli_api.sh ---
@@ -0,0 +1,196 @@
+#!/usr/bin/env bash

+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+source "$(dirname "$0")"/common.sh
+
+start_cluster
+
+# Test for CLI commands.
+# verify only the return code the content correctness of the API results.

+PERIODIC_JOB_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-api-test/target/PeriodicStreamingJob.jar
+JOB_ID_REGEX_EXTRACTOR=".*JobID ([0-9,a-f]*)"
+SAVE_POINT_REGEX_EXTRACTOR=".*Savepoint stored in (.*)\\."
+JOB_INFO_PACT_DATA_SOURCE_REGEX_EXTRACTOR="\"pact\": \"(Data Source)\""
+JOB_INFO_PACT_DATA_SINK_REGEX_EXTRACTOR="\"pact\": \"(Data Sink)\""
+JOB_LIST_REGEX_EXTRACTOR_BY_STATUS="([0-9,a-f]*) :"
+
+EXIT_CODE=0
+
+function extract_job_id_from_job_submission_return() {
+if [[ $1 =~ $JOB_ID_REGEX_EXTRACTOR ]];
+then
+JOB_ID="${BASH_REMATCH[1]}";
+else
+JOB_ID=""
+fi
+echo "$JOB_ID"
+}
+
+function extract_savepoint_path_from_savepoint_return() {
+if [[ $1 =~ $SAVE_POINT_REGEX_EXTRACTOR ]];
+then
+SAVEPOINT_PATH="${BASH_REMATCH[1]}";
+else
+SAVEPOINT_PATH=""
+fi
+echo "$SAVEPOINT_PATH"
+}
+
+function extract_valid_pact_from_job_info_return() {
+PACT_MATCH=0
+if [[ $1 =~ $JOB_INFO_PACT_DATA_SOURCE_REGEX_EXTRACTOR ]];
+then
+PACT_MATCH=$PACT_MATCH
+else
+PACT_MATCH=-1
+fi
+if [[ $1 =~ $JOB_INFO_PACT_DATA_SINK_REGEX_EXTRACTOR ]];
+then
+PACT_MATCH=$PACT_MATCH
+else
+PACT_MATCH=-1
+fi
+echo ${PACT_MATCH}
+}
+
+function extract_valid_job_list_by_type_from_job_list_return() {
+JOB_LIST_MATCH=0
+JOB_LIST_REGEX_EXTRACTOR="$JOB_LIST_REGEX_EXTRACTOR_BY_STATUS $2 $3"
+if [[ $1 =~ $JOB_LIST_REGEX_EXTRACTOR ]];
+then
+JOB_LIST_MATCH=$JOB_LIST_MATCH
+else
+JOB_LIST_MATCH=-1
+fi
+echo ${JOB_LIST_MATCH}
+}
+
+function cleanup_cli_test() {
+  stop_cluster
+  $FLINK_DIR/bin/taskmanager.sh stop-all
--- End diff --

checking the clean up code seems like it doesn't explicitly call stopping 
all tm (similar to `start-cluster.sh`, I need to explicitly call `taskmanager 
start`). I should remove only the `stop-cluster` actually.


---

[GitHub] flink pull request #6072: [hotfix][doc] Clarify kinesis docs and fix debuggi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6072


---

[GitHub] flink issue #6072: [hotfix][doc] Clarify kinesis docs and fix debugging clas...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6072
  
Will only merge the Kinesis doc fix, as the classloading docs already has a 
substantial fix meanwhile in master.


---

[jira] [Commented] (FLINK-9429) Quickstart E2E not working locally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490200#comment-16490200
 ] 

ASF GitHub Bot commented on FLINK-9429:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/6074#discussion_r190786548
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -61,8 +61,9 @@ function verify_result {
 rm $TEST_DATA_DIR/output
 fi
 
-while : ; do
-  curl 'localhost:9200/index/_search?q=*=21' > 
$TEST_DATA_DIR/output
+# make sure can terminate properly with control-C.
+while [ $? -ne 130 ]; do
--- End diff --

I would prefer that we just wrap the Elasticsearch querying to another 
function, that improves readability.

Something along the lines of:
```
function fetch_elasticsearch {
curl 'localhost:9200/index3/_count?q=*' > $TEST_DATA_DIR/output
echo $(grep '\"count\"' $TEST_DATA_DIR/output | awk '{print $3}' | sed 
's/\(.*\),/\1 /')
}

function verify_result {
local numRecords=$1

if [ -f "$TEST_DATA_DIR/output" ]; then
rm $TEST_DATA_DIR/output
fi

while (( $(fetch_elasticsearch) < $numRecords )) ; do
  echo "Waiting for Elasticsearch records ..."
  sleep 1
done
}
```


> Quickstart E2E not working locally
> --
>
> Key: FLINK-9429
> URL: https://issues.apache.org/jira/browse/FLINK-9429
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>Priority: Critical
>  Labels: test-stability
>
> The quickstart e2e test is not working locally. It seems as if the job does 
> not produce anything into Elasticsearch. Furthermore, the test does not 
> terminate with control-C.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6074: [FLINK-9429] [test] Fix does not properly terminat...

[tzulitai]

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/6074#discussion_r190786548
  
--- Diff: flink-end-to-end-tests/test-scripts/elasticsearch-common.sh ---
@@ -61,8 +61,9 @@ function verify_result {
 rm $TEST_DATA_DIR/output
 fi
 
-while : ; do
-  curl 'localhost:9200/index/_search?q=*=21' > 
$TEST_DATA_DIR/output
+# make sure can terminate properly with control-C.
+while [ $? -ne 130 ]; do
--- End diff --

I would prefer that we just wrap the Elasticsearch querying to another 
function, that improves readability.

Something along the lines of:
```
function fetch_elasticsearch {
curl 'localhost:9200/index3/_count?q=*' > $TEST_DATA_DIR/output
echo $(grep '\"count\"' $TEST_DATA_DIR/output | awk '{print $3}' | sed 
's/\(.*\),/\1 /')
}

function verify_result {
local numRecords=$1

if [ -f "$TEST_DATA_DIR/output" ]; then
rm $TEST_DATA_DIR/output
fi

while (( $(fetch_elasticsearch) < $numRecords )) ; do
  echo "Waiting for Elasticsearch records ..."
  sleep 1
done
}
```


---

[GitHub] flink issue #6043: [FLINK-7386] evolve RequestIndexer API to make it working...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6043
  
One more thing to clarify:
When planning to switch to REST, are we speaking of an implementation that 
works directly against Elasticsearch's REST API? Or are we thinking of using 
Elasticsearch's 
[RestHighLevelClient](https://snapshots.elastic.co/javadoc/org/elasticsearch/client/elasticsearch-rest-high-level-client/7.0.0-alpha1-SNAPSHOT/org/elasticsearch/client/RestHighLevelClient.html)?

I would assume the latter, but IMO we would not be able to avoid yet again 
having a common base module across future versions (e.g. across ES 6.x, 7.x, 
and so on), even if we make a clean cut now.
So, I have the feeling that the main problem here isn't that we are sharing 
code between versions, but the fact that our base shared code isn't 
future-proof enough for potential 3rd party API breaks.

That's the main reason why I'm proposing not to expose Elasticsearch 
classes anymore through base class APIs in the shared module.


---

[jira] [Commented] (FLINK-7386) Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490198#comment-16490198
 ] 

ASF GitHub Bot commented on FLINK-7386:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6043
  
One more thing to clarify:
When planning to switch to REST, are we speaking of an implementation that 
works directly against Elasticsearch's REST API? Or are we thinking of using 
Elasticsearch's 
[RestHighLevelClient](https://snapshots.elastic.co/javadoc/org/elasticsearch/client/elasticsearch-rest-high-level-client/7.0.0-alpha1-SNAPSHOT/org/elasticsearch/client/RestHighLevelClient.html)?

I would assume the latter, but IMO we would not be able to avoid yet again 
having a common base module across future versions (e.g. across ES 6.x, 7.x, 
and so on), even if we make a clean cut now.
So, I have the feeling that the main problem here isn't that we are sharing 
code between versions, but the fact that our base shared code isn't 
future-proof enough for potential 3rd party API breaks.

That's the main reason why I'm proposing not to expose Elasticsearch 
classes anymore through base class APIs in the shared module.


> Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ 
> client
> 
>
> Key: FLINK-7386
> URL: https://issues.apache.org/jira/browse/FLINK-7386
> Project: Flink
>  Issue Type: Improvement
>  Components: ElasticSearch Connector
>Reporter: Dawid Wysakowicz
>Assignee: Fang Yong
>Priority: Critical
> Fix For: 1.5.0
>
>
> In Elasticsearch 5.2.0 client the class {{BulkProcessor}} was refactored and 
> has no longer the method {{add(ActionRequest)}}.
> For more info see: https://github.com/elastic/elasticsearch/pull/20109



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6043: [FLINK-7386] evolve RequestIndexer API to make it working...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6043
  
The main reason why the discussion leaned towards the current proposed 
change by this PR, was that only Elasticsearch 5.6+ supports REST.

Only working towards a clean-cut module that uses REST, would mean that we 
still wouldn't be able to support Elasticsearch 5.2+ up to Elasticsearch 5.5.


---

[jira] [Commented] (FLINK-7386) Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490193#comment-16490193
 ] 

ASF GitHub Bot commented on FLINK-7386:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6043
  
The main reason why the discussion leaned towards the current proposed 
change by this PR, was that only Elasticsearch 5.6+ supports REST.

Only working towards a clean-cut module that uses REST, would mean that we 
still wouldn't be able to support Elasticsearch 5.2+ up to Elasticsearch 5.5.


> Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ 
> client
> 
>
> Key: FLINK-7386
> URL: https://issues.apache.org/jira/browse/FLINK-7386
> Project: Flink
>  Issue Type: Improvement
>  Components: ElasticSearch Connector
>Reporter: Dawid Wysakowicz
>Assignee: Fang Yong
>Priority: Critical
> Fix For: 1.5.0
>
>
> In Elasticsearch 5.2.0 client the class {{BulkProcessor}} was refactored and 
> has no longer the method {{add(ActionRequest)}}.
> For more info see: https://github.com/elastic/elasticsearch/pull/20109



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9352) In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure

[vinoyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490187#comment-16490187
 ] 

vinoyang commented on FLINK-9352:
-

[~StephanEwen] delay random is a way to handle this issue. We provide a 
streaming API to let the user control the initial time, if not setting, use the 
default checkpoint interval. what do you think about this way?

> In Standalone checkpoint recover mode many jobs with same checkpoint interval 
> cause IO pressure
> ---
>
> Key: FLINK-9352
> URL: https://issues.apache.org/jira/browse/FLINK-9352
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Critical
>
> currently, the periodic checkpoint coordinator startCheckpointScheduler uses 
> *baseInterval* as the initialDelay parameter. the *baseInterval* is also the 
> checkpoint interval. 
> In standalone checkpoint mode, many jobs config the same checkpoint interval. 
> When all jobs being recovered (the cluster restart or jobmanager leadership 
> switched), all jobs' checkpoint period will tend to accordance. All jobs' 
> CheckpointCoordinator would start and trigger in a approximate time point.
> This caused the high IO cost in the same time period in our production 
> scenario.
> I suggest let the scheduleAtFixedRate's initial delay parameter as a API 
> config which can let user scatter checkpoint in this scenario.
>  
> cc [~StephanEwen] [~Zentol]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8836) Duplicating a KryoSerializer does not duplicate registered default serializers

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-8836:
---
Fix Version/s: (was: 1.4.0)
   1.4.3

> Duplicating a KryoSerializer does not duplicate registered default serializers
> --
>
> Key: FLINK-8836
> URL: https://issues.apache.org/jira/browse/FLINK-8836
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.5.0, 1.6.0, 1.4.3
>
>
> The {{duplicate()}} method of the {{KryoSerializer}} is as following:
> {code:java}
> public KryoSerializer duplicate() {
>     return new KryoSerializer<>(this);
> }
> protected KryoSerializer(KryoSerializer toCopy) {
>     defaultSerializers = toCopy.defaultSerializers;
>     defaultSerializerClasses = toCopy.defaultSerializerClasses;
>     kryoRegistrations = toCopy.kryoRegistrations;
>     ...
> }
> {code}
> Shortly put, when duplicating a {{KryoSerializer}}, the 
> {{defaultSerializers}} serializer instances are directly provided to the new 
> {{KryoSerializer}} instance.
>  This causes the fact that those default serializers are shared across two 
> different {{KryoSerializer}} instances, and therefore not a correct duplicate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8836) Duplicating a KryoSerializer does not duplicate registered default serializers

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai closed FLINK-8836.
--
Resolution: Fixed

> Duplicating a KryoSerializer does not duplicate registered default serializers
> --
>
> Key: FLINK-8836
> URL: https://issues.apache.org/jira/browse/FLINK-8836
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.5.0, 1.6.0, 1.4.3
>
>
> The {{duplicate()}} method of the {{KryoSerializer}} is as following:
> {code:java}
> public KryoSerializer duplicate() {
>     return new KryoSerializer<>(this);
> }
> protected KryoSerializer(KryoSerializer toCopy) {
>     defaultSerializers = toCopy.defaultSerializers;
>     defaultSerializerClasses = toCopy.defaultSerializerClasses;
>     kryoRegistrations = toCopy.kryoRegistrations;
>     ...
> }
> {code}
> Shortly put, when duplicating a {{KryoSerializer}}, the 
> {{defaultSerializers}} serializer instances are directly provided to the new 
> {{KryoSerializer}} instance.
>  This causes the fact that those default serializers are shared across two 
> different {{KryoSerializer}} instances, and therefore not a correct duplicate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (FLINK-8836) Duplicating a KryoSerializer does not duplicate registered default serializers

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai reopened FLINK-8836:


> Duplicating a KryoSerializer does not duplicate registered default serializers
> --
>
> Key: FLINK-8836
> URL: https://issues.apache.org/jira/browse/FLINK-8836
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.5.0, 1.6.0, 1.4.3
>
>
> The {{duplicate()}} method of the {{KryoSerializer}} is as following:
> {code:java}
> public KryoSerializer duplicate() {
>     return new KryoSerializer<>(this);
> }
> protected KryoSerializer(KryoSerializer toCopy) {
>     defaultSerializers = toCopy.defaultSerializers;
>     defaultSerializerClasses = toCopy.defaultSerializerClasses;
>     kryoRegistrations = toCopy.kryoRegistrations;
>     ...
> }
> {code}
> Shortly put, when duplicating a {{KryoSerializer}}, the 
> {{defaultSerializers}} serializer instances are directly provided to the new 
> {{KryoSerializer}} instance.
>  This causes the fact that those default serializers are shared across two 
> different {{KryoSerializer}} instances, and therefore not a correct duplicate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8836) Duplicating a KryoSerializer does not duplicate registered default serializers

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490185#comment-16490185
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-8836:


Reopening to fix incorrect fix version.
1.4.0 should be 1.4.3.

> Duplicating a KryoSerializer does not duplicate registered default serializers
> --
>
> Key: FLINK-8836
> URL: https://issues.apache.org/jira/browse/FLINK-8836
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.5.0, 1.6.0, 1.4.3
>
>
> The {{duplicate()}} method of the {{KryoSerializer}} is as following:
> {code:java}
> public KryoSerializer duplicate() {
>     return new KryoSerializer<>(this);
> }
> protected KryoSerializer(KryoSerializer toCopy) {
>     defaultSerializers = toCopy.defaultSerializers;
>     defaultSerializerClasses = toCopy.defaultSerializerClasses;
>     kryoRegistrations = toCopy.kryoRegistrations;
>     ...
> }
> {code}
> Shortly put, when duplicating a {{KryoSerializer}}, the 
> {{defaultSerializers}} serializer instances are directly provided to the new 
> {{KryoSerializer}} instance.
>  This causes the fact that those default serializers are shared across two 
> different {{KryoSerializer}} instances, and therefore not a correct duplicate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9429) Quickstart E2E not working locally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490174#comment-16490174
 ] 

ASF GitHub Bot commented on FLINK-9429:
---

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/6074

[FLINK-9429] [test] Fix does not properly terminate with control-C

## What is the purpose of the change

Fix the test does not terminate with control-C.

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6074


commit 7a7e275f15622e54946e53667a58233d21187d11
Author: zhangminglei 
Date:   2018-05-25T03:19:52Z

[FLINK-9429] [test] Fix does not properly terminate with control-C




> Quickstart E2E not working locally
> --
>
> Key: FLINK-9429
> URL: https://issues.apache.org/jira/browse/FLINK-9429
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>Priority: Critical
>  Labels: test-stability
>
> The quickstart e2e test is not working locally. It seems as if the job does 
> not produce anything into Elasticsearch. Furthermore, the test does not 
> terminate with control-C.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6074: [FLINK-9429] [test] Fix does not properly terminat...

[zhangminglei]

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/6074

[FLINK-9429] [test] Fix does not properly terminate with control-C

## What is the purpose of the change

Fix the test does not terminate with control-C.

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-9429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6074


commit 7a7e275f15622e54946e53667a58233d21187d11
Author: zhangminglei 
Date:   2018-05-25T03:19:52Z

[FLINK-9429] [test] Fix does not properly terminate with control-C




---

[jira] [Commented] (FLINK-1044) Website: Offer a zip archive with a pre-setup user project

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490157#comment-16490157
 ] 

ASF GitHub Bot commented on FLINK-1044:
---

Github user medcv commented on the issue:

https://github.com/apache/flink/pull/6044
  
@zentol Thanks for the review. 
Make sense to build this `quickstarts.zip` package on Jenkins but the main 
concern was test the `.zip` package (included all dependencies) before 
uploading it on site. As end2end test for quickstarts already got merged then 
it should be fine.

I will close this PR. 

Quick side note:
I looked at the end2end test for quickstarts (`test-quickstarts.sh`) and it 
has a test for Java version. Do you think it make sense we add a ent2end test 
for Scala version of quickstarts also?


> Website: Offer a zip archive with a pre-setup user project
> --
>
> Key: FLINK-1044
> URL: https://issues.apache.org/jira/browse/FLINK-1044
> Project: Flink
>  Issue Type: Improvement
>  Components: Project Website
>Reporter: Stephan Ewen
>Priority: Minor
>  Labels: starter
> Attachments: flink-project.zip
>
>
> This is basically a shortcut for those that are not familiar with maven 
> archetypes or do not have maven installed (other then as part of the Eclipse 
> IDE or so).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6044: [FLINK-1044] Website: Offer a zip archive with a pre-setu...

[medcv]

Github user medcv commented on the issue:

https://github.com/apache/flink/pull/6044
  
@zentol Thanks for the review. 
Make sense to build this `quickstarts.zip` package on Jenkins but the main 
concern was test the `.zip` package (included all dependencies) before 
uploading it on site. As end2end test for quickstarts already got merged then 
it should be fine.

I will close this PR. 

Quick side note:
I looked at the end2end test for quickstarts (`test-quickstarts.sh`) and it 
has a test for Java version. Do you think it make sense we add a ent2end test 
for Scala version of quickstarts also?


---

[jira] [Commented] (FLINK-9429) Quickstart E2E not working locally

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490142#comment-16490142
 ] 

mingleizhang commented on FLINK-9429:
-

I will give a PR to fix this soon.

> Quickstart E2E not working locally
> --
>
> Key: FLINK-9429
> URL: https://issues.apache.org/jira/browse/FLINK-9429
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: mingleizhang
>Priority: Critical
>  Labels: test-stability
>
> The quickstart e2e test is not working locally. It seems as if the job does 
> not produce anything into Elasticsearch. Furthermore, the test does not 
> terminate with control-C.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9430) Support Casting of Object to Primitive types for Flink SQL UDF

[Shuyi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490026#comment-16490026
 ] 

Shuyi Chen commented on FLINK-9430:
---

The JSON object can be a int, float, double, boolean, array, map or etc.. Are 
you suggesting serialize the JSON object to String? But in that case, I need to 
write more UDFs to parse the String into desired type again, not cast, and also 
that is inefficient. What are the issues with returning Object?

> Support Casting of Object to Primitive types for Flink SQL UDF
> --
>
> Key: FLINK-9430
> URL: https://issues.apache.org/jira/browse/FLINK-9430
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API  SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We want to add a SQL UDF to access specific element in a JSON string using 
> JSON path. However, the JSON element can be of different types, e.g. Int, 
> Float, Double, String, Boolean and etc.. Since return type is not part of the 
> method signature, we can not use overload. So we will end up writing a UDF 
> for each type, e.g. GetFloatFromJSON, GetIntFromJSON and etc., which has a 
> lot of duplication. 
> One way to unify all these UDF functions is to implement one UDF and return 
> java.lang.Object, and in the SQL statement, use CAST AS to cast the returned 
> Object into the correct type. Below is an example:
>  
> {code:java}
> object JsonPathUDF extends ScalarFunction {
>  def eval(jsonStr: String, path: String): Object = {
>JSONParser.parse(jsonStr).read(path)
>  }
> }{code}
> {code:java}
>  SELECT CAST(jsonpath(json, "$.store.book.title") AS VARCHAR(32)) as 
> bookTitle FROM table1{code}
> The current Flink SQL cast implementation does not support casting from 
> GenericTypeInfo to another type, I have already got a local 
> branch to fix this. Please comment if there are alternatives to the problem 
> above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8933) Avoid calling Class#newInstance

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-8933.
---
Resolution: Later

> Avoid calling Class#newInstance
> ---
>
> Key: FLINK-8933
> URL: https://issues.apache.org/jira/browse/FLINK-8933
> Project: Flink
>  Issue Type: Task
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9187) add prometheus pushgateway reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489803#comment-16489803
 ] 

ASF GitHub Bot commented on FLINK-9187:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5857
  
I think it's fine to be in the same project.


> add prometheus pushgateway reporter
> ---
>
> Key: FLINK-9187
> URL: https://issues.apache.org/jira/browse/FLINK-9187
> Project: Flink
>  Issue Type: New Feature
>  Components: Metrics
>Affects Versions: 1.4.2
>Reporter: lamber-ken
>Priority: Minor
>  Labels: features
> Fix For: 1.5.0
>
>
> make flink system can send metrics to prometheus via pushgateway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5857: [FLINK-9187][METRICS] add prometheus pushgateway reporter

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5857
  
I think it's fine to be in the same project.


---

[jira] [Commented] (FLINK-8744) Add annotations for documentation common/advanced options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489735#comment-16489735
 ] 

ASF GitHub Bot commented on FLINK-8744:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5843
  
For `env.java.opts` we have a `ConfigOption` in the `CoreOptions` class. 
They are documented like any other option (and technically they are also usable 
like any other option).


> Add annotations for documentation common/advanced options
> -
>
> Key: FLINK-8744
> URL: https://issues.apache.org/jira/browse/FLINK-8744
> Project: Flink
>  Issue Type: New Feature
>  Components: Configuration, Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} only generates [the full configuration 
> reference|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#full-reference].
>  The 
> [common|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#common-options]
>  and 
> [advanced|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#advanced-options]
>  sections are still manually defined in {{config.md}} and thus prone to being 
> outdated / out-of-sync with the full reference.
> I propose adding {{@Common}}/{{@Advanced}} annotations based on which we 
> could generate these sections as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5843: [FLINK-8744][docs] Generate "Common Option" section

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5843
  
For `env.java.opts` we have a `ConfigOption` in the `CoreOptions` class. 
They are documented like any other option (and technically they are also usable 
like any other option).


---

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489646#comment-16489646
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r190705206
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -247,7 +249,15 @@ public void shutdown(Time timeout) {
 
@Override
protected void channelRead0(ChannelHandlerContext ctx, Object 
msg) {
-   if (msg instanceof FullHttpResponse) {
+   // TODO: should this check for status OK (200) and 
treat all other as errors?
--- End diff --

message semantics are handled later when parsing the payload, which 
effectively does what you're suggesting. The else branch for debugging; our 
rest servers always return `FullHttpResponses`.


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6071: [FLINK-3952][runtine] Upgrade to Netty 4.1

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/6071#discussion_r190705206
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -247,7 +249,15 @@ public void shutdown(Time timeout) {
 
@Override
protected void channelRead0(ChannelHandlerContext ctx, Object 
msg) {
-   if (msg instanceof FullHttpResponse) {
+   // TODO: should this check for status OK (200) and 
treat all other as errors?
--- End diff --

message semantics are handled later when parsing the payload, which 
effectively does what you're suggesting. The else branch for debugging; our 
rest servers always return `FullHttpResponses`.


---

[jira] [Commented] (FLINK-9352) In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489529#comment-16489529
 ] 

Stephan Ewen commented on FLINK-9352:
-

Okay, I see that this can be a problem.

Would it help to make the initial checkpoint delay random, within the 
checkpoint interval (but at least "min pause between checkpoints")?

> In Standalone checkpoint recover mode many jobs with same checkpoint interval 
> cause IO pressure
> ---
>
> Key: FLINK-9352
> URL: https://issues.apache.org/jira/browse/FLINK-9352
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Critical
>
> currently, the periodic checkpoint coordinator startCheckpointScheduler uses 
> *baseInterval* as the initialDelay parameter. the *baseInterval* is also the 
> checkpoint interval. 
> In standalone checkpoint mode, many jobs config the same checkpoint interval. 
> When all jobs being recovered (the cluster restart or jobmanager leadership 
> switched), all jobs' checkpoint period will tend to accordance. All jobs' 
> CheckpointCoordinator would start and trigger in a approximate time point.
> This caused the high IO cost in the same time period in our production 
> scenario.
> I suggest let the scheduleAtFixedRate's initial delay parameter as a API 
> config which can let user scatter checkpoint in this scenario.
>  
> cc [~StephanEwen] [~Zentol]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8744) Add annotations for documentation common/advanced options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489516#comment-16489516
 ] 

ASF GitHub Bot commented on FLINK-8744:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5843
  
One could add an int to the annotation, as "priority / position" and sort 
by that.
Not sure nice, but could be okay.

I think it was nice for users that the most common options (the ones you 
need first) were at the top of the list.

Out of curiosity, what happens to options like `env.java.opts` which are 
shell script only options, but very common?




> Add annotations for documentation common/advanced options
> -
>
> Key: FLINK-8744
> URL: https://issues.apache.org/jira/browse/FLINK-8744
> Project: Flink
>  Issue Type: New Feature
>  Components: Configuration, Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} only generates [the full configuration 
> reference|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#full-reference].
>  The 
> [common|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#common-options]
>  and 
> [advanced|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#advanced-options]
>  sections are still manually defined in {{config.md}} and thus prone to being 
> outdated / out-of-sync with the full reference.
> I propose adding {{@Common}}/{{@Advanced}} annotations based on which we 
> could generate these sections as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5843: [FLINK-8744][docs] Generate "Common Option" section

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5843
  
One could add an int to the annotation, as "priority / position" and sort 
by that.
Not sure nice, but could be okay.

I think it was nice for users that the most common options (the ones you 
need first) were at the top of the list.

Out of curiosity, what happens to options like `env.java.opts` which are 
shell script only options, but very common?




---

[jira] [Closed] (FLINK-9088) Upgrade Nifi connector dependency to 1.6.0

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-9088.
---

> Upgrade Nifi connector dependency to 1.6.0
> --
>
> Key: FLINK-9088
> URL: https://issues.apache.org/jira/browse/FLINK-9088
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Ted Yu
>Assignee: Hai Zhou
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently dependency of Nifi is 0.6.1
> We should upgrade to 1.6.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9286) Update classloading debugging page

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-9286.
---

> Update classloading debugging page
> --
>
> Key: FLINK-9286
> URL: https://issues.apache.org/jira/browse/FLINK-9286
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0
>
>
> The [classloading 
> debugging|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  page must be updated for 1.5.
> The [parent-first pattern 
> section|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  refers to deprecated keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-9088) Upgrade Nifi connector dependency to 1.6.0

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-9088.
-
   Resolution: Fixed
Fix Version/s: 1.6.0

Fixed via a202d747cbd7c49ecd4f8f01b4007350c4327b94

> Upgrade Nifi connector dependency to 1.6.0
> --
>
> Key: FLINK-9088
> URL: https://issues.apache.org/jira/browse/FLINK-9088
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Ted Yu
>Assignee: Hai Zhou
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently dependency of Nifi is 0.6.1
> We should upgrade to 1.6.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-9305) Register flink-s3-fs-hadoop for the s3a:// scheme as well

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-9305.
-
   Resolution: Fixed
Fix Version/s: 1.6.0

Fixed in 1.6.0 via 33aee80694aeeb258cc47aa112505e9ceb993bc2

> Register flink-s3-fs-hadoop for the s3a:// scheme as well
> -
>
> Key: FLINK-9305
> URL: https://issues.apache.org/jira/browse/FLINK-9305
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.6.0
>
>
> For enhanced user experience, we should also register our Hadoop S3A-based 
> shaded S3 file system implementation for the {{s3a://}} file system scheme, 
> not just {{s3://}}. This way, the user can easily switch from the manual S3 
> integration to the shaded one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9305) Register flink-s3-fs-hadoop for the s3a:// scheme as well

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489501#comment-16489501
 ] 

ASF GitHub Bot commented on FLINK-9305:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5963


> Register flink-s3-fs-hadoop for the s3a:// scheme as well
> -
>
> Key: FLINK-9305
> URL: https://issues.apache.org/jira/browse/FLINK-9305
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.6.0
>
>
> For enhanced user experience, we should also register our Hadoop S3A-based 
> shaded S3 file system implementation for the {{s3a://}} file system scheme, 
> not just {{s3://}}. This way, the user can easily switch from the manual S3 
> integration to the shaded one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-9286) Update classloading debugging page

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-9286.
-
Resolution: Fixed

Fixed in
  - 1.4.3 via 5cc869c353619388c199fb7b13813d24e3b5bcfa and 
3166e6722a8c0f2bc99b3d312eaf92e3e4bd3502

  - 1.5.0 via a1b0a6fd511b9a051668747f97bbf6f3dd69c7d2 and 
067047e32257452f994656330797faa58bb3fec6

  - 1.6.0 via 428a900ce518297bb2fbf3421110594bc1e861c9 and 
c07b693fe3e39ed11deef89151aff83729a74268

> Update classloading debugging page
> --
>
> Key: FLINK-9286
> URL: https://issues.apache.org/jira/browse/FLINK-9286
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0
>
>
> The [classloading 
> debugging|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  page must be updated for 1.5.
> The [parent-first pattern 
> section|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  refers to deprecated keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9286) Update classloading debugging page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489503#comment-16489503
 ] 

ASF GitHub Bot commented on FLINK-9286:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5948


> Update classloading debugging page
> --
>
> Key: FLINK-9286
> URL: https://issues.apache.org/jira/browse/FLINK-9286
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0
>
>
> The [classloading 
> debugging|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  page must be updated for 1.5.
> The [parent-first pattern 
> section|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html#configuring-classloader-resolution-order]
>  refers to deprecated keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9088) Upgrade Nifi connector dependency to 1.6.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489502#comment-16489502
 ] 

ASF GitHub Bot commented on FLINK-9088:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5891


> Upgrade Nifi connector dependency to 1.6.0
> --
>
> Key: FLINK-9088
> URL: https://issues.apache.org/jira/browse/FLINK-9088
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Ted Yu
>Assignee: Hai Zhou
>Priority: Major
>
> Currently dependency of Nifi is 0.6.1
> We should upgrade to 1.6.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9305) Register flink-s3-fs-hadoop for the s3a:// scheme as well

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-9305.
---

> Register flink-s3-fs-hadoop for the s3a:// scheme as well
> -
>
> Key: FLINK-9305
> URL: https://issues.apache.org/jira/browse/FLINK-9305
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.6.0
>
>
> For enhanced user experience, we should also register our Hadoop S3A-based 
> shaded S3 file system implementation for the {{s3a://}} file system scheme, 
> not just {{s3://}}. This way, the user can easily switch from the manual S3 
> integration to the shaded one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5963: [FLINK-9305][s3] also register flink-s3-fs-hadoop'...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5963


---

[GitHub] flink pull request #5891: [FLINK-9088][nifi-connector][build] Bump nifi-site...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5891


---

[GitHub] flink pull request #5948: [FLINK-9286][docs] Update classloading docs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5948


---

[jira] [Commented] (FLINK-9349) KafkaConnector Exception while fetching from multiple kafka topics

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489409#comment-16489409
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-9349:


[~pnowojski] thanks for the reminder. Yes, I will test this with RC6.

> KafkaConnector Exception  while fetching from multiple kafka topics
> ---
>
> Key: FLINK-9349
> URL: https://issues.apache.org/jira/browse/FLINK-9349
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.4.0
>Reporter: Vishal Santoshi
>Assignee: Sergey Nuyanzin
>Priority: Critical
> Fix For: 1.5.0, 1.4.3
>
> Attachments: Flink9349Test.java
>
>
> ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java
>  
> It seems the List subscribedPartitionStates was being modified when 
> runFetchLoop iterated the List.
> This can happen if, e.g., FlinkKafkaConsumer runs the following code 
> concurrently:
>                 kafkaFetcher.addDiscoveredPartitions(discoveredPartitions);
>  
> {code:java}
>  java.util.ConcurrentModificationException
>   at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
>   at java.util.LinkedList$ListItr.next(LinkedList.java:888)
>   at 
> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8744) Add annotations for documentation common/advanced options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489403#comment-16489403
 ] 

ASF GitHub Bot commented on FLINK-8744:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5843
  
they _should_ be sorted alphabetically based on the key, but i now see that 
this isn't actually the case.

I'm not sure if I can easily add the kind of sorting you're asking for as 
that would require the generator to know about the semantics of an option ("oh, 
this is a port option, but that one is about memory!").


> Add annotations for documentation common/advanced options
> -
>
> Key: FLINK-8744
> URL: https://issues.apache.org/jira/browse/FLINK-8744
> Project: Flink
>  Issue Type: New Feature
>  Components: Configuration, Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} only generates [the full configuration 
> reference|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#full-reference].
>  The 
> [common|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#common-options]
>  and 
> [advanced|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#advanced-options]
>  sections are still manually defined in {{config.md}} and thus prone to being 
> outdated / out-of-sync with the full reference.
> I propose adding {{@Common}}/{{@Advanced}} annotations based on which we 
> could generate these sections as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5843: [FLINK-8744][docs] Generate "Common Option" section

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5843
  
they _should_ be sorted alphabetically based on the key, but i now see that 
this isn't actually the case.

I'm not sure if I can easily add the kind of sorting you're asking for as 
that would require the generator to know about the semantics of an option ("oh, 
this is a port option, but that one is about memory!").


---

[GitHub] flink issue #6007: [FLINK-8518][Table API & SQL] Support DOW for EXTRACT

[snuyanzin]

Github user snuyanzin commented on the issue:

https://github.com/apache/flink/pull/6007
  
> Just because DOW is in the reserved keywords section doesn't mean that we 
support it.
Definitely agree however my point was that the same situation is for DOY, 
CENTURY and others...
Now I added information about date/time functions synonyms which currently 
work in flink. I was surprised but they work without any efforts => I just 
added more tests related to them and documentation.
@twalthr could you please have a look?


---

[jira] [Commented] (FLINK-8518) Support DOW, EPOCH, DECADE for EXTRACT

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489399#comment-16489399
 ] 

ASF GitHub Bot commented on FLINK-8518:
---

Github user snuyanzin commented on the issue:

https://github.com/apache/flink/pull/6007
  
> Just because DOW is in the reserved keywords section doesn't mean that we 
support it.
Definitely agree however my point was that the same situation is for DOY, 
CENTURY and others...
Now I added information about date/time functions synonyms which currently 
work in flink. I was surprised but they work without any efforts => I just 
added more tests related to them and documentation.
@twalthr could you please have a look?


> Support DOW, EPOCH, DECADE for EXTRACT
> --
>
> Key: FLINK-8518
> URL: https://issues.apache.org/jira/browse/FLINK-8518
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: Sergey Nuyanzin
>Priority: Major
>
> We upgraded Calcite to version 1.15 in FLINK-7934. The EXTRACT method 
> supports more conversion targets. The targets DOW, EPOCH, DECADE should be 
> implemented and tested for different datatypes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-3952) Bump Netty to 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489387#comment-16489387
 ] 

ASF GitHub Bot commented on FLINK-3952:
---

zentol closed pull request #41: [FLINK-3952] Bump Netty to 4.1.24 and drop 
netty-router
URL: https://github.com/apache/flink-shaded/pull/41
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/flink-shaded-netty-4/pom.xml b/flink-shaded-netty-4/pom.xml
index 3ac5c25..0aa734b 100644
--- a/flink-shaded-netty-4/pom.xml
+++ b/flink-shaded-netty-4/pom.xml
@@ -34,7 +34,7 @@ under the License.
 ${netty.version}-4.0
 
 
-4.0.56.Final
+4.1.24.Final
 
 
 
@@ -43,12 +43,6 @@ under the License.
 netty-all
 ${netty.version}
 
-
-
-tv.cntt
-netty-router
-1.10
-
 
 
 
@@ -101,4 +95,4 @@ under the License.
 
 
 
-
\ No newline at end of file
+


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Bump Netty to 4.1
> -
>
> Key: FLINK-3952
> URL: https://issues.apache.org/jira/browse/FLINK-3952
> Project: Flink
>  Issue Type: Improvement
>  Components: Core, Network
>Reporter: rektide de la fey
>Assignee: Piotr Nowojski
>Priority: Major
>  Labels: netty
>
> Netty 4.1 is about to release final. This release has [a number of 
> significant 
> enhancements|http://netty.io/wiki/new-and-noteworthy-in-4.1.html], and in 
> particular I find HTTP/2 codecs to be incredibly desirable to have. 
> Additionally, hopefully, the [Hadoop patches for Netty 
> 4.1|https://issues.apache.org/jira/browse/HADOOP-11716] get some tests and 
> get merged, & I believe if/when that happens it'll be important for Flink to 
> also be using the new Netty minor version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9420) Add tests for SQL IN sub-query operator in streaming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489330#comment-16489330
 ] 

ASF GitHub Bot commented on FLINK-9420:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6065


> Add tests for SQL IN sub-query operator in streaming
> 
>
> Key: FLINK-9420
> URL: https://issues.apache.org/jira/browse/FLINK-9420
> Project: Flink
>  Issue Type: Test
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: Ruidong Li
>Priority: Major
> Fix For: 1.6.0
>
>
> In FLINK-6094 we implemented non-windowed inner joins. The Table API & SQL 
> should now  support the {{IN}} operator for sub-queries in streaming. Batch 
> support has been added in FLINK-4565. We need to add unit tests, an IT case, 
> and update the docs about that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6065: [FLINK-9420][TableAPI & SQL] Add tests for SQL IN ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6065


---

[jira] [Resolved] (FLINK-9420) Add tests for SQL IN sub-query operator in streaming

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-9420.
-
   Resolution: Fixed
Fix Version/s: 1.6.0

Fixed in 1.6.0: c47cc87e507f6cb9276c8414c32114ef512cf917

> Add tests for SQL IN sub-query operator in streaming
> 
>
> Key: FLINK-9420
> URL: https://issues.apache.org/jira/browse/FLINK-9420
> Project: Flink
>  Issue Type: Test
>  Components: Table API  SQL
>Reporter: Timo Walther
>Assignee: Ruidong Li
>Priority: Major
> Fix For: 1.6.0
>
>
> In FLINK-6094 we implemented non-windowed inner joins. The Table API & SQL 
> should now  support the {{IN}} operator for sub-queries in streaming. Batch 
> support has been added in FLINK-4565. We need to add unit tests, an IT case, 
> and update the docs about that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6072: [hotfix][doc] Clarify kinesis docs and fix debugging clas...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/6072
  
+1, thanks for the fixes @pnowojski.
Will merge this ..


---

[jira] [Assigned] (FLINK-8468) Make the connector to take advantage of AMQP features (routing key, exchange and message properties)

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reassigned FLINK-8468:
---

Assignee: (was: Gary Yao)

> Make the connector to take advantage of AMQP features (routing key, exchange 
> and message properties)
> 
>
> Key: FLINK-8468
> URL: https://issues.apache.org/jira/browse/FLINK-8468
> Project: Flink
>  Issue Type: Improvement
>  Components: RabbitMQ Connector
>Affects Versions: 1.4.0
>Reporter: Ph.Duveau
>Priority: Major
>
> Make the connector to take advantage of AMQP features by adding a constructor 
> and an interface to implement



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8468) Make the connector to take advantage of AMQP features (routing key, exchange and message properties)

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reassigned FLINK-8468:
---

Assignee: Gary Yao

> Make the connector to take advantage of AMQP features (routing key, exchange 
> and message properties)
> 
>
> Key: FLINK-8468
> URL: https://issues.apache.org/jira/browse/FLINK-8468
> Project: Flink
>  Issue Type: Improvement
>  Components: RabbitMQ Connector
>Affects Versions: 1.4.0
>Reporter: Ph.Duveau
>Assignee: Gary Yao
>Priority: Major
>
> Make the connector to take advantage of AMQP features by adding a constructor 
> and an interface to implement



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9422) Dedicated operator for UNION on streaming tables with time attributes

[Hequn Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489261#comment-16489261
 ] 

Hequn Cheng commented on FLINK-9422:


Thanks for your explanation. :)

> Dedicated operator for UNION on streaming tables with time attributes
> -
>
> Key: FLINK-9422
> URL: https://issues.apache.org/jira/browse/FLINK-9422
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: Fabian Hueske
>Assignee: Ruidong Li
>Priority: Minor
>
> We can implement a dedicated operator for a {{UNION}} operator on tables with 
> time attributes. Currently, {{UNION}} is translated into a {{UNION ALL}} and 
> a subsequent {{GROUP BY}} on all attributes without aggregation functions. 
> The state of the grouping operator is only clean up using state retention 
> timers. 
> The dedicated operator would leverage the monotonicity property of the time 
> attribute and watermarks to automatically clean up its state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (FLINK-9435) Remove per-key selection Tuple instantiation via reflection in KeySelectorUtil#ComparableKeySelector

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-9435:

Comment: was deleted

(was: I think an even more performant way would be a set of Java MethodHandles.)

> Remove per-key selection Tuple instantiation via reflection in 
> KeySelectorUtil#ComparableKeySelector
> 
>
> Key: FLINK-9435
> URL: https://issues.apache.org/jira/browse/FLINK-9435
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 1.3.0, 1.3.1, 1.4.0, 1.3.2, 1.3.3, 1.5.0, 1.4.1, 1.4.2, 
> 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
>
> Every {{ComparableKeySelector#getKey()}} call currently creates a new tuple 
> from {{Tuple.getTupleClass(keyLength).newInstance();}} which seems expensive. 
> Instead, we could get a template tuple and use {{Tuple#copy()}} which copies 
> the right sub-class in a more optimal way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8554) Upgrade AWS SDK

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489226#comment-16489226
 ] 

Ted Yu commented on FLINK-8554:
---

We can resolve this JIRA as duplicate of FLINK-9266

> Upgrade AWS SDK
> ---
>
> Key: FLINK-8554
> URL: https://issues.apache.org/jira/browse/FLINK-8554
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> AWS SDK 1.11.271 fixes a lot of bugs.
> One of which would exhibit the following:
> {code}
> Caused by: java.lang.NullPointerException
>   at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
>   at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
>   at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9435) Remove per-key selection Tuple instantiation via reflection in KeySelectorUtil#ComparableKeySelector

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489222#comment-16489222
 ] 

Timo Walther commented on FLINK-9435:
-

I think an even more performant way would be a set of Java MethodHandles.

> Remove per-key selection Tuple instantiation via reflection in 
> KeySelectorUtil#ComparableKeySelector
> 
>
> Key: FLINK-9435
> URL: https://issues.apache.org/jira/browse/FLINK-9435
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 1.3.0, 1.3.1, 1.4.0, 1.3.2, 1.3.3, 1.5.0, 1.4.1, 1.4.2, 
> 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
>
> Every {{ComparableKeySelector#getKey()}} call currently creates a new tuple 
> from {{Tuple.getTupleClass(keyLength).newInstance();}} which seems expensive. 
> Instead, we could get a template tuple and use {{Tuple#copy()}} which copies 
> the right sub-class in a more optimal way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8554) Upgrade AWS SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489218#comment-16489218
 ] 

ASF GitHub Bot commented on FLINK-8554:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5978
  
@StephanEwen close this or not?


> Upgrade AWS SDK
> ---
>
> Key: FLINK-8554
> URL: https://issues.apache.org/jira/browse/FLINK-8554
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> AWS SDK 1.11.271 fixes a lot of bugs.
> One of which would exhibit the following:
> {code}
> Caused by: java.lang.NullPointerException
>   at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
>   at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
>   at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5978: [FLINK-8554] Upgrade AWS SDK

[yanghua]

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/5978
  
@StephanEwen close this or not?


---

[jira] [Commented] (FLINK-9091) Failure while enforcing releasability in building flink-json module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489213#comment-16489213
 ] 

ASF GitHub Bot commented on FLINK-9091:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6073
  
Ok, I see it now. This pull request's title is just a tad misleading (it 
doesn't fix all of the convergence for this module, just some selection of them)


> Failure while enforcing releasability in building flink-json module
> ---
>
> Key: FLINK-9091
> URL: https://issues.apache.org/jira/browse/FLINK-9091
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Reporter: Ted Yu
>Assignee: Hai Zhou
>Priority: Major
> Attachments: f-json.out
>
>
> Got the following when building flink-json module:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> ...
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce 
> (dependency-convergence) on project flink-json: Some Enforcer rules have 
> failed.   Look above for specific messages explaining why the rule failed. -> 
> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7386) Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489214#comment-16489214
 ] 

ASF GitHub Bot commented on FLINK-7386:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/6043
  
@cjolif I agree, let's do something here.

@tzulitai what do you think about trying to use the switch to REST to make 
a clean cut and start a new connector project (without dependency on 
`flink-connector-elasticsearch-base`). As an experiment, we could try how much 
code we would actually need to copy into the new project.

@aljoscha and @patricklucas I remember you also had some thoughts on the 
elasticsearch connectors.

I am +1 for seeing if we can drop ElasticSearch 1.x and 2.x support, but 
that should be a separate thread.


> Flink Elasticsearch 5 connector is not compatible with Elasticsearch 5.2+ 
> client
> 
>
> Key: FLINK-7386
> URL: https://issues.apache.org/jira/browse/FLINK-7386
> Project: Flink
>  Issue Type: Improvement
>  Components: ElasticSearch Connector
>Reporter: Dawid Wysakowicz
>Assignee: Fang Yong
>Priority: Critical
> Fix For: 1.5.0
>
>
> In Elasticsearch 5.2.0 client the class {{BulkProcessor}} was refactored and 
> has no longer the method {{add(ActionRequest)}}.
> For more info see: https://github.com/elastic/elasticsearch/pull/20109



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6043: [FLINK-7386] evolve RequestIndexer API to make it working...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/6043
  
@cjolif I agree, let's do something here.

@tzulitai what do you think about trying to use the switch to REST to make 
a clean cut and start a new connector project (without dependency on 
`flink-connector-elasticsearch-base`). As an experiment, we could try how much 
code we would actually need to copy into the new project.

@aljoscha and @patricklucas I remember you also had some thoughts on the 
elasticsearch connectors.

I am +1 for seeing if we can drop ElasticSearch 1.x and 2.x support, but 
that should be a separate thread.


---

[GitHub] flink issue #6073: [FLINK-9091] [table] Fix dependency convergence for flink...

[pnowojski]

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/6073
  
Ok, I see it now. This pull request's title is just a tad misleading (it 
doesn't fix all of the convergence for this module, just some selection of them)


---

[jira] [Created] (FLINK-9435) Remove per-key selection Tuple instantiation via reflection in KeySelectorUtil#ComparableKeySelector

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-9435:
--

 Summary: Remove per-key selection Tuple instantiation via 
reflection in KeySelectorUtil#ComparableKeySelector
 Key: FLINK-9435
 URL: https://issues.apache.org/jira/browse/FLINK-9435
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 1.4.2, 1.4.1, 1.3.3, 1.3.2, 1.4.0, 1.3.1, 1.3.0, 1.5.0, 
1.6.0
Reporter: Nico Kruber
Assignee: Nico Kruber


Every {{ComparableKeySelector#getKey()}} call currently creates a new tuple 
from {{Tuple.getTupleClass(keyLength).newInstance();}} which seems expensive. 
Instead, we could get a template tuple and use {{Tuple#copy()}} which copies 
the right sub-class in a more optimal way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9352) In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure

[vinoyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489191#comment-16489191
 ] 

vinoyang commented on FLINK-9352:
-

[~StephanEwen] can you look at this?

> In Standalone checkpoint recover mode many jobs with same checkpoint interval 
> cause IO pressure
> ---
>
> Key: FLINK-9352
> URL: https://issues.apache.org/jira/browse/FLINK-9352
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Critical
>
> currently, the periodic checkpoint coordinator startCheckpointScheduler uses 
> *baseInterval* as the initialDelay parameter. the *baseInterval* is also the 
> checkpoint interval. 
> In standalone checkpoint mode, many jobs config the same checkpoint interval. 
> When all jobs being recovered (the cluster restart or jobmanager leadership 
> switched), all jobs' checkpoint period will tend to accordance. All jobs' 
> CheckpointCoordinator would start and trigger in a approximate time point.
> This caused the high IO cost in the same time period in our production 
> scenario.
> I suggest let the scheduleAtFixedRate's initial delay parameter as a API 
> config which can let user scatter checkpoint in this scenario.
>  
> cc [~StephanEwen] [~Zentol]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5963: [FLINK-9305][s3] also register flink-s3-fs-hadoop's facto...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5963
  
Merging this...


---

[jira] [Commented] (FLINK-9305) Register flink-s3-fs-hadoop for the s3a:// scheme as well

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489182#comment-16489182
 ] 

ASF GitHub Bot commented on FLINK-9305:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5963
  
Merging this...


> Register flink-s3-fs-hadoop for the s3a:// scheme as well
> -
>
> Key: FLINK-9305
> URL: https://issues.apache.org/jira/browse/FLINK-9305
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
>
> For enhanced user experience, we should also register our Hadoop S3A-based 
> shaded S3 file system implementation for the {{s3a://}} file system scheme, 
> not just {{s3://}}. This way, the user can easily switch from the manual S3 
> integration to the shaded one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9276) Improve error message when TaskManager fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489180#comment-16489180
 ] 

ASF GitHub Bot commented on FLINK-9276:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5954
  
This looks fine from my side.

Would like to get a +1 from @tillrohrmann before merging this...


> Improve error message when TaskManager fails
> 
>
> Key: FLINK-9276
> URL: https://issues.apache.org/jira/browse/FLINK-9276
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Critical
>
> When a TaskManager fails, we frequently get a message
> {code}
> org.apache.flink.util.FlinkException: Releasing TaskManager 
> container_1524853016208_0001_01_000102
> {code}
> This message is misleading in that it sounds like an intended operation, when 
> it really is a failure of a container that the {{ResourceManager}} reports to 
> the {{JobManager}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5954: [FLINK-9276] Improve error message when TaskManager fails

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5954
  
This looks fine from my side.

Would like to get a +1 from @tillrohrmann before merging this...


---

[jira] [Commented] (FLINK-8933) Avoid calling Class#newInstance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489157#comment-16489157
 ] 

ASF GitHub Bot commented on FLINK-8933:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/6016
  
agree, closing this...


> Avoid calling Class#newInstance
> ---
>
> Key: FLINK-8933
> URL: https://issues.apache.org/jira/browse/FLINK-8933
> Project: Flink
>  Issue Type: Task
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8933) Avoid calling Class#newInstance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489158#comment-16489158
 ] 

ASF GitHub Bot commented on FLINK-8933:
---

Github user yanghua closed the pull request at:

https://github.com/apache/flink/pull/6016


> Avoid calling Class#newInstance
> ---
>
> Key: FLINK-8933
> URL: https://issues.apache.org/jira/browse/FLINK-8933
> Project: Flink
>  Issue Type: Task
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6016: [FLINK-8933] Avoid calling Class#newInstance(part 2)

[yanghua]

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/6016
  
agree, closing this...


---

[GitHub] flink pull request #6016: [FLINK-8933] Avoid calling Class#newInstance(part ...

[yanghua]

Github user yanghua closed the pull request at:

https://github.com/apache/flink/pull/6016


---

[jira] [Updated] (FLINK-9434) Test instability in YARNSessionCapacitySchedulerITCase#

[Nico Kruber (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber updated FLINK-9434:
---
Labels: test-stability  (was: )

> Test instability in YARNSessionCapacitySchedulerITCase#
> ---
>
> Key: FLINK-9434
> URL: https://issues.apache.org/jira/browse/FLINK-9434
> Project: Flink
>  Issue Type: Bug
>  Components: Tests, YARN
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Priority: Major
>  Labels: test-stability
>
> {code}
> Test 
> testNonexistingQueueWARNmessage(org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase)
>  failed with:
> java.lang.AssertionError: There is at least one application on the cluster is 
> not finished.App application_1527164710351_0007 is in state RUNNING
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.flink.yarn.YarnTestBase.checkClusterEmpty(YarnTestBase.java:217)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
> https://travis-ci.org/apache/flink/jobs/383117221



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9434) Test instability in YARNSessionCapacitySchedulerITCase#

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-9434:
--

 Summary: Test instability in YARNSessionCapacitySchedulerITCase#
 Key: FLINK-9434
 URL: https://issues.apache.org/jira/browse/FLINK-9434
 Project: Flink
  Issue Type: Bug
  Components: Tests, YARN
Affects Versions: 1.5.0, 1.6.0
Reporter: Nico Kruber


{code}
Test 
testNonexistingQueueWARNmessage(org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase)
 failed with:
java.lang.AssertionError: There is at least one application on the cluster is 
not finished.App application_1527164710351_0007 is in state RUNNING
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.flink.yarn.YarnTestBase.checkClusterEmpty(YarnTestBase.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}

https://travis-ci.org/apache/flink/jobs/383117221



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-9427) Cannot download from BlobServer, because the server address is unknown.

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-9427.
--
Resolution: Fixed

Fixed via
master: 47dc69970996aa2a57c0dbfde866d3dac6d53001
1.5.0: 5c35e04cecabdb56ffe5dc453dff3a748df874b3

> Cannot download from BlobServer, because the server address is unknown.
> ---
>
> Key: FLINK-9427
> URL: https://issues.apache.org/jira/browse/FLINK-9427
> Project: Flink
>  Issue Type: Bug
>  Components: TaskManager
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: failure
>
>
> Setup: 6 + 1 nodes EMR cluster with m4.4xlarge instances
> Job submission fails in most cases (but not all of them):
> {noformat}
> [hadoop@ip-172-31-28-17 flink-1.5.0]$ HADOOP_CONF_DIR=/etc/hadoop/conf 
> ./bin/flink run -m yarn-cluster -p 80 -yn 80 examples/batch/WordCount.jar 
> --input hdfs:///user/hadoop/enwiki-latest-abstract.xml --output 
> hdfs:///user/hadoop/output
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/hadoop/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2018-05-23 15:07:46,062 INFO  org.apache.hadoop.yarn.client.RMProxy           
>               - Connecting to ResourceManager at 
> ip-172-31-28-17.eu-central-1.compute.internal/172.31.28.17:8032
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli   
>               - No path for the flink jar passed. Using the location of class 
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli   
>               - No path for the flink jar passed. Using the location of class 
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-05-23 15:07:46,339 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster 
> specification: ClusterSpecification{masterMemoryMB=1024, 
> taskManagerMemoryMB=4096, numberTaskManagers=80, slotsPerTaskManager=1}
> 2018-05-23 15:07:46,596 WARN  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The 
> configuration directory ('/home/hadoop/flink-1.5.0/conf') contains both LOG4J 
> and Logback configuration files. Please delete or rename one of them.
> 2018-05-23 15:07:47,318 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting 
> application master application_1526561166266_0049
> 2018-05-23 15:07:47,336 INFO  
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted 
> application application_1526561166266_0049
> 2018-05-23 15:07:47,337 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for 
> the cluster to be allocated
> 2018-05-23 15:07:47,338 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying 
> cluster, current state ACCEPTED
> 2018-05-23 15:07:51,101 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN 
> application has been deployed successfully.
> Starting execution of program
> 
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: 
> java.io.IOException: Cannot download from BlobServer, because the server 
> address is unknown.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:264)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:452)
> at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
> at 
> 

[jira] [Closed] (FLINK-9416) Make job submission retriable operation in case of a ongoing leader election

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-9416.

   Resolution: Fixed
Fix Version/s: (was: 1.5.1)
   (was: 1.6.0)
   1.5.0

Fixed via
master: 88987ebb93b4d56cc5a2efd7ef47dedab9b9d3d9
1.5.0: 44467239ab264039b37fe5e608531ae1d60cb8a1

> Make job submission retriable operation in case of a ongoing leader election
> 
>
> Key: FLINK-9416
> URL: https://issues.apache.org/jira/browse/FLINK-9416
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Fix For: 1.5.0
>
>
> When starting a session cluster, it can happen that the job submission fails 
> if the REST server endpoint has already gained leadership but if the 
> leadership election for the {{Dispatcher}} is still ongoing. In such a case, 
> we receive a error response saying that the leader election is still ongoing 
> and fail the job submission. I think it would be nicer to also make the 
> submission step a retriable operation in order to avoid this race condition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9421) RunningJobsRegistry entries are not cleaned up after job termination

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-9421.

   Resolution: Fixed
Fix Version/s: 1.5.0

Fixed via
master: 5ad0ca2392a9672a92756337243834d1b466a24d
1.5.0: 2307102fb3cd733b453438877b179043dfeaae9b

> RunningJobsRegistry entries are not cleaned up after job termination
> 
>
> Key: FLINK-9421
> URL: https://issues.apache.org/jira/browse/FLINK-9421
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Fix For: 1.5.0
>
>
> Currently, the {{Dispatcher}} does not clean up the {{RunningJobsRegistry}} 
> after the job has finished. The consequence is that a ZNode with the JobID 
> and a state num per job remains in ZooKeeper.
> We should clean up these ZNodes to avoid a resource leak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6015: [FLINK-8933] Avoid calling Class#newInstance(part 1)

[yanghua]

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/6015
  
OK, agree, closing this PR~


---

[jira] [Commented] (FLINK-9421) RunningJobsRegistry entries are not cleaned up after job termination

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489140#comment-16489140
 ] 

ASF GitHub Bot commented on FLINK-9421:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6068


> RunningJobsRegistry entries are not cleaned up after job termination
> 
>
> Key: FLINK-9421
> URL: https://issues.apache.org/jira/browse/FLINK-9421
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Currently, the {{Dispatcher}} does not clean up the {{RunningJobsRegistry}} 
> after the job has finished. The consequence is that a ZNode with the JobID 
> and a state num per job remains in ZooKeeper.
> We should clean up these ZNodes to avoid a resource leak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8933) Avoid calling Class#newInstance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489144#comment-16489144
 ] 

ASF GitHub Bot commented on FLINK-8933:
---

Github user yanghua closed the pull request at:

https://github.com/apache/flink/pull/6015


> Avoid calling Class#newInstance
> ---
>
> Key: FLINK-8933
> URL: https://issues.apache.org/jira/browse/FLINK-8933
> Project: Flink
>  Issue Type: Task
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9427) Cannot download from BlobServer, because the server address is unknown.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489139#comment-16489139
 ] 

ASF GitHub Bot commented on FLINK-9427:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6067


> Cannot download from BlobServer, because the server address is unknown.
> ---
>
> Key: FLINK-9427
> URL: https://issues.apache.org/jira/browse/FLINK-9427
> Project: Flink
>  Issue Type: Bug
>  Components: TaskManager
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: failure
>
>
> Setup: 6 + 1 nodes EMR cluster with m4.4xlarge instances
> Job submission fails in most cases (but not all of them):
> {noformat}
> [hadoop@ip-172-31-28-17 flink-1.5.0]$ HADOOP_CONF_DIR=/etc/hadoop/conf 
> ./bin/flink run -m yarn-cluster -p 80 -yn 80 examples/batch/WordCount.jar 
> --input hdfs:///user/hadoop/enwiki-latest-abstract.xml --output 
> hdfs:///user/hadoop/output
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/hadoop/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2018-05-23 15:07:46,062 INFO  org.apache.hadoop.yarn.client.RMProxy           
>               - Connecting to ResourceManager at 
> ip-172-31-28-17.eu-central-1.compute.internal/172.31.28.17:8032
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli   
>               - No path for the flink jar passed. Using the location of class 
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli   
>               - No path for the flink jar passed. Using the location of class 
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-05-23 15:07:46,339 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster 
> specification: ClusterSpecification{masterMemoryMB=1024, 
> taskManagerMemoryMB=4096, numberTaskManagers=80, slotsPerTaskManager=1}
> 2018-05-23 15:07:46,596 WARN  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The 
> configuration directory ('/home/hadoop/flink-1.5.0/conf') contains both LOG4J 
> and Logback configuration files. Please delete or rename one of them.
> 2018-05-23 15:07:47,318 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting 
> application master application_1526561166266_0049
> 2018-05-23 15:07:47,336 INFO  
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted 
> application application_1526561166266_0049
> 2018-05-23 15:07:47,337 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for 
> the cluster to be allocated
> 2018-05-23 15:07:47,338 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying 
> cluster, current state ACCEPTED
> 2018-05-23 15:07:51,101 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN 
> application has been deployed successfully.
> Starting execution of program
> 
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: 
> java.io.IOException: Cannot download from BlobServer, because the server 
> address is unknown.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:264)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:452)
> at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
> at 
> 

[GitHub] flink pull request #6068: [FLINK-9421] Remove job from RunningJobsRegistry w...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6068


---

[jira] [Commented] (FLINK-8933) Avoid calling Class#newInstance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489143#comment-16489143
 ] 

ASF GitHub Bot commented on FLINK-8933:
---

Github user yanghua commented on the issue:

https://github.com/apache/flink/pull/6015
  
OK, agree, closing this PR~


> Avoid calling Class#newInstance
> ---
>
> Key: FLINK-8933
> URL: https://issues.apache.org/jira/browse/FLINK-8933
> Project: Flink
>  Issue Type: Task
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #6015: [FLINK-8933] Avoid calling Class#newInstance(part ...

[yanghua]

Github user yanghua closed the pull request at:

https://github.com/apache/flink/pull/6015


---

[GitHub] flink pull request #6069: [FLINK-9416] Make all RestClusterClient calls retr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6069


---

[GitHub] flink pull request #6067: [FLINK-9427] Fix registration and request slot rac...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6067


---

[jira] [Commented] (FLINK-9416) Make job submission retriable operation in case of a ongoing leader election

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489141#comment-16489141
 ] 

ASF GitHub Bot commented on FLINK-9416:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6069


> Make job submission retriable operation in case of a ongoing leader election
> 
>
> Key: FLINK-9416
> URL: https://issues.apache.org/jira/browse/FLINK-9416
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Fix For: 1.6.0, 1.5.1
>
>
> When starting a session cluster, it can happen that the job submission fails 
> if the REST server endpoint has already gained leadership but if the 
> leadership election for the {{Dispatcher}} is still ongoing. In such a case, 
> we receive a error response saying that the leader election is still ongoing 
> and fail the job submission. I think it would be nicer to also make the 
> submission step a retriable operation in order to avoid this race condition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9091) Failure while enforcing releasability in building flink-json module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489136#comment-16489136
 ] 

ASF GitHub Bot commented on FLINK-9091:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/6073
  
It only fixes the issues discussed in the corresponding JIRA. Dependency 
convergence issues with `flink-runtime` are a different topic.


> Failure while enforcing releasability in building flink-json module
> ---
>
> Key: FLINK-9091
> URL: https://issues.apache.org/jira/browse/FLINK-9091
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Reporter: Ted Yu
>Assignee: Hai Zhou
>Priority: Major
> Attachments: f-json.out
>
>
> Got the following when building flink-json module:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> ...
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce 
> (dependency-convergence) on project flink-json: Some Enforcer rules have 
> failed.   Look above for specific messages explaining why the rule failed. -> 
> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #6073: [FLINK-9091] [table] Fix dependency convergence for flink...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/6073
  
It only fixes the issues discussed in the corresponding JIRA. Dependency 
convergence issues with `flink-runtime` are a different topic.


---

[jira] [Closed] (FLINK-7130) Remove eventSerializer from NFA

[Dawid Wysakowicz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz closed FLINK-7130.
---
Resolution: Won't Fix

> Remove eventSerializer from NFA
> ---
>
> Key: FLINK-7130
> URL: https://issues.apache.org/jira/browse/FLINK-7130
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>
> Right now eventSerializer is serialized within NFA. It should be present only 
> in NFASerializer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #4610: [FLINK-7130] Removed event serializer from NFA and...

[dawidwys]

Github user dawidwys closed the pull request at:

https://github.com/apache/flink/pull/4610


---

[jira] [Commented] (FLINK-7130) Remove eventSerializer from NFA

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489128#comment-16489128
 ] 

ASF GitHub Bot commented on FLINK-7130:
---

Github user dawidwys closed the pull request at:

https://github.com/apache/flink/pull/4610


> Remove eventSerializer from NFA
> ---
>
> Key: FLINK-7130
> URL: https://issues.apache.org/jira/browse/FLINK-7130
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>
> Right now eventSerializer is serialized within NFA. It should be present only 
> in NFASerializer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9427) Cannot download from BlobServer, because the server address is unknown.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489121#comment-16489121
 ] 

ASF GitHub Bot commented on FLINK-9427:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/6067#discussion_r190611660
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskExecutorTest.java
 ---
@@ -1483,6 +1485,216 @@ public void 
testMaximumRegistrationDurationAfterConnectionLoss() throws Exceptio
}
}
 
+   /**
+* Tests that we ignore slot requests if the TaskExecutor is not
+* registered at a ResourceManager.
+*/
+   @Test
+   public void testIgnoringSlotRequestsIfNotRegistered() throws Exception {
+   final TaskSlotTable taskSlotTable = new 
TaskSlotTable(Collections.singleton(ResourceProfile.UNKNOWN), timerService);
+   final TaskManagerServices taskManagerServices = new 
TaskManagerServicesBuilder().setTaskSlotTable(taskSlotTable).build();
+
+   final TaskExecutor taskExecutor = 
createTaskExecutor(taskManagerServices);
+
+   taskExecutor.start();
+
+   try {
+   final TestingResourceManagerGateway 
testingResourceManagerGateway = new TestingResourceManagerGateway();
+
+   final CompletableFuture 
registrationFuture = new CompletableFuture<>();
+   final CompletableFuture 
taskExecutorResourceIdFuture = new CompletableFuture<>();
+
+   
testingResourceManagerGateway.setRegisterTaskExecutorFunction(stringResourceIDSlotReportIntegerHardwareDescriptionTuple5
 -> {
+
taskExecutorResourceIdFuture.complete(stringResourceIDSlotReportIntegerHardwareDescriptionTuple5.f1);
+return registrationFuture;
+});
+
+   
rpc.registerGateway(testingResourceManagerGateway.getAddress(), 
testingResourceManagerGateway);
+   
resourceManagerLeaderRetriever.notifyListener(testingResourceManagerGateway.getAddress(),
 testingResourceManagerGateway.getFencingToken().toUUID());
+
+   final TaskExecutorGateway taskExecutorGateway = 
taskExecutor.getSelfGateway(TaskExecutorGateway.class);
+
+   final ResourceID resourceId = 
taskExecutorResourceIdFuture.get();
+
+   final SlotID slotId = new SlotID(resourceId, 0);
+   final CompletableFuture 
slotRequestResponse = taskExecutorGateway.requestSlot(slotId, jobId, new 
AllocationID(), "foobar", testingResourceManagerGateway.getFencingToken(), 
timeout);
+
+   try {
+   slotRequestResponse.get();
+   fail("We should not be able to request slots 
before the TaskExecutor is registered at the ResourceManager.");
+   } catch (ExecutionException ee) {
+   
assertThat(ExceptionUtils.stripExecutionException(ee), 
instanceOf(TaskManagerException.class));
+   }
+   } finally {
+   RpcUtils.terminateRpcEndpoint(taskExecutor, timeout);
+   }
+   }
+
+   /**
+* Tests that the TaskExecutor tries to reconnect to a ResourceManager 
from which it
+* was explicitly disconnected.
+*/
+   @Test
+   public void testReconnectionAttemptIfExplicitlyDisconnected() throws 
Exception {
+   final long heartbeatInterval = 1000L;
+   final TaskSlotTable taskSlotTable = new 
TaskSlotTable(Collections.singleton(ResourceProfile.UNKNOWN), timerService);
+   final TaskManagerLocation taskManagerLocation = new 
LocalTaskManagerLocation();
+   final TaskExecutor taskExecutor = new TaskExecutor(
+   rpc,
+   
TaskManagerConfiguration.fromConfiguration(configuration),
+   haServices,
+   new TaskManagerServicesBuilder()
+   .setTaskSlotTable(taskSlotTable)
+   .setTaskManagerLocation(taskManagerLocation)
+   .build(),
+   new HeartbeatServices(heartbeatInterval, 1000L),
+   
UnregisteredMetricGroups.createUnregisteredTaskManagerMetricGroup(),
+   dummyBlobCacheService,
+   testingFatalErrorHandler);
+
+   taskExecutor.start();
+
+   try {
+   final TestingResourceManagerGateway 
testingResourceManagerGateway = new TestingResourceManagerGateway();
+   final ClusterInformation clusterInformation = new 
ClusterInformation("foobar", 1234);