[jira] [Commented] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205992#comment-17205992
]
Arvid Heise commented on FLINK-19027:
-
Thanks, [~dian.fu] , I'm digging in.
> UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel
> failed because of test timeout
>
>
> Key: FLINK-19027
> URL: https://issues.apache.org/jira/browse/FLINK-19027
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Dian Fu
>Assignee: Arvid Heise
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0, 1.11.3
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5789=logs=119bbba7-f5e3-5e08-e72d-09f1529665de=ec103906-d047-5b8a-680e-05fc000dfca9]
> {code}
> 2020-08-22T21:13:05.5315459Z [ERROR]
> shouldPerformUnalignedCheckpointOnParallelRemoteChannel(org.apache.flink.test.checkpointing.UnalignedCheckpointITCase)
> Time elapsed: 300.075 s <<< ERROR!
> 2020-08-22T21:13:05.5316451Z org.junit.runners.model.TestTimedOutException:
> test timed out after 300 seconds
> 2020-08-22T21:13:05.5317432Z at sun.misc.Unsafe.park(Native Method)
> 2020-08-22T21:13:05.5317799Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-08-22T21:13:05.5318247Z at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2020-08-22T21:13:05.5318885Z at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> 2020-08-22T21:13:05.5327035Z at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2020-08-22T21:13:05.5328114Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-08-22T21:13:05.5328869Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1719)
> 2020-08-22T21:13:05.5329482Z at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> 2020-08-22T21:13:05.5330138Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1699)
> 2020-08-22T21:13:05.5330771Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1681)
> 2020-08-22T21:13:05.5331351Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.execute(UnalignedCheckpointITCase.java:158)
> 2020-08-22T21:13:05.5332015Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel(UnalignedCheckpointITCase.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-19027:
---
Assignee: Arvid Heise
> UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel
> failed because of test timeout
>
>
> Key: FLINK-19027
> URL: https://issues.apache.org/jira/browse/FLINK-19027
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Dian Fu
>Assignee: Arvid Heise
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0, 1.11.3
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5789=logs=119bbba7-f5e3-5e08-e72d-09f1529665de=ec103906-d047-5b8a-680e-05fc000dfca9]
> {code}
> 2020-08-22T21:13:05.5315459Z [ERROR]
> shouldPerformUnalignedCheckpointOnParallelRemoteChannel(org.apache.flink.test.checkpointing.UnalignedCheckpointITCase)
> Time elapsed: 300.075 s <<< ERROR!
> 2020-08-22T21:13:05.5316451Z org.junit.runners.model.TestTimedOutException:
> test timed out after 300 seconds
> 2020-08-22T21:13:05.5317432Z at sun.misc.Unsafe.park(Native Method)
> 2020-08-22T21:13:05.5317799Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-08-22T21:13:05.5318247Z at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2020-08-22T21:13:05.5318885Z at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> 2020-08-22T21:13:05.5327035Z at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2020-08-22T21:13:05.5328114Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-08-22T21:13:05.5328869Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1719)
> 2020-08-22T21:13:05.5329482Z at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> 2020-08-22T21:13:05.5330138Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1699)
> 2020-08-22T21:13:05.5330771Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1681)
> 2020-08-22T21:13:05.5331351Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.execute(UnalignedCheckpointITCase.java:158)
> 2020-08-22T21:13:05.5332015Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel(UnalignedCheckpointITCase.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gm7y8 commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
gm7y8 commented on a change in pull request #13458: URL: https://github.com/apache/flink/pull/13458#discussion_r498619466 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointStatistics.java ## @@ -229,8 +242,8 @@ public static CheckpointStatistics generateCheckpointStatistics(AbstractCheckpoi checkpointStatisticsPerTask.put( taskStateStat.getJobVertexId(), new TaskCheckpointStatistics( - checkpointStats.getCheckpointId(), - checkpointStats.getStatus(), + checkpointStats.getCheckpointId(), + checkpointStats.getStatus(), Review comment: Thanks for pointing out. let me fix this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * ce25c6bbc562da022b86aaa99d24f40c1d159b41 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7160) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gm7y8 commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
gm7y8 commented on a change in pull request #13458:
URL: https://github.com/apache/flink/pull/13458#discussion_r498611476
##
File path: flink-runtime-web/src/test/resources/rest_api_v1.snapshot
##
@@ -557,7 +557,7 @@
},
"status" : {
"type" : "string",
-"enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED",
"CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ]
+"enum" : [ "INITIALIZING", "CREATED", "RUNNING", "FAILING",
"FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED",
"RECONCILING" ]
Review comment:
this was auto-generated code.. not related to this change.. will open
jira issue and commit this change separately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-playgrounds] shuiqiangchen commented on a change in pull request #16: [FLINK-19145][walkthroughs] Add PyFlink-walkthrough to Flink playground.
shuiqiangchen commented on a change in pull request #16:
URL: https://github.com/apache/flink-playgrounds/pull/16#discussion_r498607377
##
File path: pyflink-walkthrough/README.md
##
@@ -0,0 +1,140 @@
+# pyflink-walkthrough
+
+## Background
+
+In this playground, you will learn how to build and run an end-to-end PyFlink
pipeline for data analytics, covering the following steps:
+
+* Reading data from a Kafka source;
+* Creating data using a
[UDF](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/python/table-api-users-guide/udfs/python_udfs.html);
+* Performing a simple aggregation over the source data;
+* Writing the results to Elasticsearch and visualizing them in Kibana.
+
+The environment is based on Docker Compose, so the only requirement is that
you have [Docker](https://docs.docker.com/get-docker/)
+installed in your machine.
+
+### Kafka
+You will be using Kafka to store sample input data about payment transactions.
A simple data generator
[generate_source_data.py](generator/generate_source_data.py) is provided to
+continuously write new records to the `payment_msg` Kafka topic. Each record
is structured as follows:
+
+`{"createTime": "2020-08-12 06:29:02", "orderId": 1597213797, "payAmount":
28306.44976403719, "payPlatform": 0, "provinceId": 4}`
+
+* `createTime`: The creation time of the transaction.
+* `orderId`: The id of the current transaction.
+* `payAmount`: The amount being paid with this transaction.
+* `payPlatform`: The platform used to create this payment: pc or mobile.
+* `provinceId`: The id of the province for the user.
+
+You can use the following command to read data from the Kafka topic and check
whether it's generated correctly:
+```shell script
+$ docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server
kafka:9092 --topic payment_msg
+{"createTime":"2020-07-27
09:25:32.77","orderId":1595841867217,"payAmount":7732.44,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.231","orderId":1595841867218,"payAmount":75774.05,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.72","orderId":1595841867219,"payAmount":65908.55,"payPlatform":0,"provinceId":0}
+{"createTime":"2020-07-27
09:25:34.216","orderId":1595841867220,"payAmount":15341.11,"payPlatform":0,"provinceId":1}
+{"createTime":"2020-07-27
09:25:34.698","orderId":1595841867221,"payAmount":37504.42,"payPlatform":0,"provinceId":0}
+```
+You can also create a new topic by executing the following command:
+```shell script
+$ docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092
--create --topic --partitions 8 --replication-factor 1
+```
+
+### PyFlink
+
+The transaction data will be processed with PyFlink using the Python script
[payment_msg_processing.py](payment_msg_proccessing.py).
+This script will first map the `provinceId` in the input records to its
corresponding province name using a Python UDF,
+and then compute the sum of the transaction amounts for each province.
+
+### ElasticSearch
+
+ElasticSearch is used to store the results and to provide an efficient query
service.
+
+### Kibana
+
+Kibana is an open source data visualization dashboard for ElasticSearch. You
will use it to visualize
+the results of your PyFlink pipeline.
+
+## Setup
+
+As mentioned, the environment for this walkthrough is based on Docker Compose;
It uses a custom image
+to spin up Flink (JobManager+TaskManager), Kafka+Zookeeper, the data
generator, and Elasticsearch+Kibana containers.
+
+You can find the [docker-compose.yaml](docker-compose.yml) file of the
pyflink-walkthrough in the `pyflink-walkthrough` root directory.
+
+### Building the Docker image
+
+First, build the Docker image by running:
+
+```bash
+docker-compose build
+```
+
+### Starting the Playground
+
+Once the Docker image build is complete, run the following command to start
the playground:
+
+```bash
+docker-compose up -d
+```
+
+One way of checking if the playground was successfully started is to access
some of the services that are exposed:
+
+1. visiting Flink Web UI [http://localhost:8081](http://localhost:8081).
+2. visiting Elasticsearch [http://localhost:9200](http://localhost:9200).
+3. visiting Kibana [http://localhost:5601](http://localhost:5601).
+
+**Note:** you may need to wait around 1 minute before all the services come up.
+
+### Stopping the Playground
+
+To stop the playground, run the following command:
+
+```bash
+docker-compose down
+```
+
+
+## Running the PyFlink job
+
+1. Submit the PyFlink job.
+```shell script
+$ docker-compose exec jobmanager ./bin/flink run -py
/opt/pyflink-walkthrough/payment_msg_proccessing.py -d
+```
+Navigate to the [Flink Web UI](http://localhost:8081) after the job is
submitted successfully. There should be a job in the running job list.
+Click the job to get more details. You should see that the `StreamGraph` of
the `payment_msg_proccessing` consists of two nodes, each with a parallelism of
1.
[GitHub] [flink-playgrounds] shuiqiangchen commented on a change in pull request #16: [FLINK-19145][walkthroughs] Add PyFlink-walkthrough to Flink playground.
shuiqiangchen commented on a change in pull request #16:
URL: https://github.com/apache/flink-playgrounds/pull/16#discussion_r498607377
##
File path: pyflink-walkthrough/README.md
##
@@ -0,0 +1,140 @@
+# pyflink-walkthrough
+
+## Background
+
+In this playground, you will learn how to build and run an end-to-end PyFlink
pipeline for data analytics, covering the following steps:
+
+* Reading data from a Kafka source;
+* Creating data using a
[UDF](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/python/table-api-users-guide/udfs/python_udfs.html);
+* Performing a simple aggregation over the source data;
+* Writing the results to Elasticsearch and visualizing them in Kibana.
+
+The environment is based on Docker Compose, so the only requirement is that
you have [Docker](https://docs.docker.com/get-docker/)
+installed in your machine.
+
+### Kafka
+You will be using Kafka to store sample input data about payment transactions.
A simple data generator
[generate_source_data.py](generator/generate_source_data.py) is provided to
+continuously write new records to the `payment_msg` Kafka topic. Each record
is structured as follows:
+
+`{"createTime": "2020-08-12 06:29:02", "orderId": 1597213797, "payAmount":
28306.44976403719, "payPlatform": 0, "provinceId": 4}`
+
+* `createTime`: The creation time of the transaction.
+* `orderId`: The id of the current transaction.
+* `payAmount`: The amount being paid with this transaction.
+* `payPlatform`: The platform used to create this payment: pc or mobile.
+* `provinceId`: The id of the province for the user.
+
+You can use the following command to read data from the Kafka topic and check
whether it's generated correctly:
+```shell script
+$ docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server
kafka:9092 --topic payment_msg
+{"createTime":"2020-07-27
09:25:32.77","orderId":1595841867217,"payAmount":7732.44,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.231","orderId":1595841867218,"payAmount":75774.05,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.72","orderId":1595841867219,"payAmount":65908.55,"payPlatform":0,"provinceId":0}
+{"createTime":"2020-07-27
09:25:34.216","orderId":1595841867220,"payAmount":15341.11,"payPlatform":0,"provinceId":1}
+{"createTime":"2020-07-27
09:25:34.698","orderId":1595841867221,"payAmount":37504.42,"payPlatform":0,"provinceId":0}
+```
+You can also create a new topic by executing the following command:
+```shell script
+$ docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092
--create --topic --partitions 8 --replication-factor 1
+```
+
+### PyFlink
+
+The transaction data will be processed with PyFlink using the Python script
[payment_msg_processing.py](payment_msg_proccessing.py).
+This script will first map the `provinceId` in the input records to its
corresponding province name using a Python UDF,
+and then compute the sum of the transaction amounts for each province.
+
+### ElasticSearch
+
+ElasticSearch is used to store the results and to provide an efficient query
service.
+
+### Kibana
+
+Kibana is an open source data visualization dashboard for ElasticSearch. You
will use it to visualize
+the results of your PyFlink pipeline.
+
+## Setup
+
+As mentioned, the environment for this walkthrough is based on Docker Compose;
It uses a custom image
+to spin up Flink (JobManager+TaskManager), Kafka+Zookeeper, the data
generator, and Elasticsearch+Kibana containers.
+
+You can find the [docker-compose.yaml](docker-compose.yml) file of the
pyflink-walkthrough in the `pyflink-walkthrough` root directory.
+
+### Building the Docker image
+
+First, build the Docker image by running:
+
+```bash
+docker-compose build
+```
+
+### Starting the Playground
+
+Once the Docker image build is complete, run the following command to start
the playground:
+
+```bash
+docker-compose up -d
+```
+
+One way of checking if the playground was successfully started is to access
some of the services that are exposed:
+
+1. visiting Flink Web UI [http://localhost:8081](http://localhost:8081).
+2. visiting Elasticsearch [http://localhost:9200](http://localhost:9200).
+3. visiting Kibana [http://localhost:5601](http://localhost:5601).
+
+**Note:** you may need to wait around 1 minute before all the services come up.
+
+### Stopping the Playground
+
+To stop the playground, run the following command:
+
+```bash
+docker-compose down
+```
+
+
+## Running the PyFlink job
+
+1. Submit the PyFlink job.
+```shell script
+$ docker-compose exec jobmanager ./bin/flink run -py
/opt/pyflink-walkthrough/payment_msg_proccessing.py -d
+```
+Navigate to the [Flink Web UI](http://localhost:8081) after the job is
submitted successfully. There should be a job in the running job list.
+Click the job to get more details. You should see that the `StreamGraph` of
the `payment_msg_proccessing` consists of two nodes, each with a parallelism of
1.
[GitHub] [flink-playgrounds] shuiqiangchen commented on a change in pull request #16: [FLINK-19145][walkthroughs] Add PyFlink-walkthrough to Flink playground.
shuiqiangchen commented on a change in pull request #16:
URL: https://github.com/apache/flink-playgrounds/pull/16#discussion_r498606570
##
File path: pyflink-walkthrough/README.md
##
@@ -0,0 +1,140 @@
+# pyflink-walkthrough
+
+## Background
+
+In this playground, you will learn how to build and run an end-to-end PyFlink
pipeline for data analytics, covering the following steps:
+
+* Reading data from a Kafka source;
+* Creating data using a
[UDF](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/python/table-api-users-guide/udfs/python_udfs.html);
+* Performing a simple aggregation over the source data;
+* Writing the results to Elasticsearch and visualizing them in Kibana.
+
+The environment is based on Docker Compose, so the only requirement is that
you have [Docker](https://docs.docker.com/get-docker/)
+installed in your machine.
+
+### Kafka
+You will be using Kafka to store sample input data about payment transactions.
A simple data generator
[generate_source_data.py](generator/generate_source_data.py) is provided to
+continuously write new records to the `payment_msg` Kafka topic. Each record
is structured as follows:
+
+`{"createTime": "2020-08-12 06:29:02", "orderId": 1597213797, "payAmount":
28306.44976403719, "payPlatform": 0, "provinceId": 4}`
+
+* `createTime`: The creation time of the transaction.
+* `orderId`: The id of the current transaction.
+* `payAmount`: The amount being paid with this transaction.
+* `payPlatform`: The platform used to create this payment: pc or mobile.
+* `provinceId`: The id of the province for the user.
+
+You can use the following command to read data from the Kafka topic and check
whether it's generated correctly:
+```shell script
+$ docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server
kafka:9092 --topic payment_msg
+{"createTime":"2020-07-27
09:25:32.77","orderId":1595841867217,"payAmount":7732.44,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.231","orderId":1595841867218,"payAmount":75774.05,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.72","orderId":1595841867219,"payAmount":65908.55,"payPlatform":0,"provinceId":0}
+{"createTime":"2020-07-27
09:25:34.216","orderId":1595841867220,"payAmount":15341.11,"payPlatform":0,"provinceId":1}
+{"createTime":"2020-07-27
09:25:34.698","orderId":1595841867221,"payAmount":37504.42,"payPlatform":0,"provinceId":0}
+```
+You can also create a new topic by executing the following command:
+```shell script
+$ docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092
--create --topic --partitions 8 --replication-factor 1
+```
+
+### PyFlink
+
+The transaction data will be processed with PyFlink using the Python script
[payment_msg_processing.py](payment_msg_proccessing.py).
+This script will first map the `provinceId` in the input records to its
corresponding province name using a Python UDF,
+and then compute the sum of the transaction amounts for each province.
+
+### ElasticSearch
+
+ElasticSearch is used to store the results and to provide an efficient query
service.
+
+### Kibana
+
+Kibana is an open source data visualization dashboard for ElasticSearch. You
will use it to visualize
+the results of your PyFlink pipeline.
+
+## Setup
+
+As mentioned, the environment for this walkthrough is based on Docker Compose;
It uses a custom image
+to spin up Flink (JobManager+TaskManager), Kafka+Zookeeper, the data
generator, and Elasticsearch+Kibana containers.
+
+You can find the [docker-compose.yaml](docker-compose.yml) file of the
pyflink-walkthrough in the `pyflink-walkthrough` root directory.
+
+### Building the Docker image
+
+First, build the Docker image by running:
+
+```bash
+docker-compose build
+```
+
+### Starting the Playground
+
+Once the Docker image build is complete, run the following command to start
the playground:
+
+```bash
+docker-compose up -d
+```
+
+One way of checking if the playground was successfully started is to access
some of the services that are exposed:
+
+1. visiting Flink Web UI [http://localhost:8081](http://localhost:8081).
+2. visiting Elasticsearch [http://localhost:9200](http://localhost:9200).
Review comment:
This simply indicates that the Elasticsearch service is successfully up
when you visit this link.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-playgrounds] shuiqiangchen commented on a change in pull request #16: [FLINK-19145][walkthroughs] Add PyFlink-walkthrough to Flink playground.
shuiqiangchen commented on a change in pull request #16:
URL: https://github.com/apache/flink-playgrounds/pull/16#discussion_r498605245
##
File path: pyflink-walkthrough/README.md
##
@@ -0,0 +1,140 @@
+# pyflink-walkthrough
+
+## Background
+
+In this playground, you will learn how to build and run an end-to-end PyFlink
pipeline for data analytics, covering the following steps:
+
+* Reading data from a Kafka source;
+* Creating data using a
[UDF](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/python/table-api-users-guide/udfs/python_udfs.html);
+* Performing a simple aggregation over the source data;
+* Writing the results to Elasticsearch and visualizing them in Kibana.
+
+The environment is based on Docker Compose, so the only requirement is that
you have [Docker](https://docs.docker.com/get-docker/)
+installed in your machine.
+
+### Kafka
+You will be using Kafka to store sample input data about payment transactions.
A simple data generator
[generate_source_data.py](generator/generate_source_data.py) is provided to
+continuously write new records to the `payment_msg` Kafka topic. Each record
is structured as follows:
+
+`{"createTime": "2020-08-12 06:29:02", "orderId": 1597213797, "payAmount":
28306.44976403719, "payPlatform": 0, "provinceId": 4}`
+
+* `createTime`: The creation time of the transaction.
+* `orderId`: The id of the current transaction.
+* `payAmount`: The amount being paid with this transaction.
+* `payPlatform`: The platform used to create this payment: pc or mobile.
+* `provinceId`: The id of the province for the user.
+
+You can use the following command to read data from the Kafka topic and check
whether it's generated correctly:
+```shell script
+$ docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server
kafka:9092 --topic payment_msg
+{"createTime":"2020-07-27
09:25:32.77","orderId":1595841867217,"payAmount":7732.44,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.231","orderId":1595841867218,"payAmount":75774.05,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.72","orderId":1595841867219,"payAmount":65908.55,"payPlatform":0,"provinceId":0}
+{"createTime":"2020-07-27
09:25:34.216","orderId":1595841867220,"payAmount":15341.11,"payPlatform":0,"provinceId":1}
+{"createTime":"2020-07-27
09:25:34.698","orderId":1595841867221,"payAmount":37504.42,"payPlatform":0,"provinceId":0}
+```
+You can also create a new topic by executing the following command:
+```shell script
+$ docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092
--create --topic --partitions 8 --replication-factor 1
+```
+
+### PyFlink
+
+The transaction data will be processed with PyFlink using the Python script
[payment_msg_processing.py](payment_msg_proccessing.py).
+This script will first map the `provinceId` in the input records to its
corresponding province name using a Python UDF,
+and then compute the sum of the transaction amounts for each province.
+
+### ElasticSearch
+
+ElasticSearch is used to store the results and to provide an efficient query
service.
+
+### Kibana
+
+Kibana is an open source data visualization dashboard for ElasticSearch. You
will use it to visualize
+the results of your PyFlink pipeline.
+
+## Setup
+
+As mentioned, the environment for this walkthrough is based on Docker Compose;
It uses a custom image
+to spin up Flink (JobManager+TaskManager), Kafka+Zookeeper, the data
generator, and Elasticsearch+Kibana containers.
+
+You can find the [docker-compose.yaml](docker-compose.yml) file of the
pyflink-walkthrough in the `pyflink-walkthrough` root directory.
+
+### Building the Docker image
+
+First, build the Docker image by running:
+
+```bash
+docker-compose build
+```
+
+### Starting the Playground
+
+Once the Docker image build is complete, run the following command to start
the playground:
+
+```bash
+docker-compose up -d
+```
+
+One way of checking if the playground was successfully started is to access
some of the services that are exposed:
+
+1. visiting Flink Web UI [http://localhost:8081](http://localhost:8081).
+2. visiting Elasticsearch [http://localhost:9200](http://localhost:9200).
+3. visiting Kibana [http://localhost:5601](http://localhost:5601).
+
+**Note:** you may need to wait around 1 minute before all the services come up.
+
+### Stopping the Playground
+
+To stop the playground, run the following command:
+
+```bash
+docker-compose down
+```
Review comment:
Yes, it should be placed in the end of the walkthrough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * 0e3916e68d38b432ebc33248d3a1ce7f7213b1f8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6968) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6572) * ce25c6bbc562da022b86aaa99d24f40c1d159b41 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7160) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19489) SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
[
https://issues.apache.org/jira/browse/FLINK-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19489:
Labels: test-stability (was: )
> SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
> ---
>
> Key: FLINK-19489
> URL: https://issues.apache.org/jira/browse/FLINK-19489
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f
> {code}
> 020-10-01T21:55:34.9982203Z "main" #1 prio=5 os_prio=0 cpu=1048.80ms
> elapsed=921.99s tid=0x7f8c00015800 nid=0xf6e in Object.wait()
> [0x7f8c06648000]
> 2020-10-01T21:55:34.9982807Zjava.lang.Thread.State: WAITING (on object
> monitor)
> 2020-10-01T21:55:34.9983177Z at
> java.lang.Object.wait(java.base@11.0.7/Native Method)
> 2020-10-01T21:55:34.9983871Z - waiting on <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9984581Z at
> java.lang.Thread.join(java.base@11.0.7/Thread.java:1305)
> 2020-10-01T21:55:34.9985433Z - waiting to re-lock in wait()
> <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9985998Z at
> org.apache.flink.core.testutils.CheckedThread.trySync(CheckedThread.java:112)
> 2020-10-01T21:55:34.9986511Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:100)
> 2020-10-01T21:55:34.9987004Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:89)
> 2020-10-01T21:55:34.9987707Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread.shutdown(SplitFetcherTest.java:301)
> 2020-10-01T21:55:34.9988427Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent(SplitFetcherTest.java:131)
> 2020-10-01T21:55:34.9989025Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.7/Native
> Method)
> 2020-10-01T21:55:34.9989531Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.7/NativeMethodAccessorImpl.java:62)
> 2020-10-01T21:55:34.9990117Z at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.7/DelegatingMethodAccessorImpl.java:43)
> 2020-10-01T21:55:34.9990626Z at
> java.lang.reflect.Method.invoke(java.base@11.0.7/Method.java:566)
> 2020-10-01T21:55:34.9991078Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-01T21:55:34.9991602Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-01T21:55:34.9992119Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-01T21:55:34.9992749Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-01T21:55:34.9993229Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-01T21:55:34.9993700Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-01T21:55:34.9994202Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-01T21:55:34.9994670Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9995098Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9995524Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.9995965Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-01T21:55:34.9996403Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-01T21:55:34.9996816Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-01T21:55:34.9997268Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-01T21:55:34.9997695Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-01T21:55:34.9998077Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9998510Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9998941Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.380Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-01T21:55:34.815Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>
[jira] [Created] (FLINK-19489) SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
Dian Fu created FLINK-19489:
---
Summary: SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets
stuck
Key: FLINK-19489
URL: https://issues.apache.org/jira/browse/FLINK-19489
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Affects Versions: 1.12.0
Reporter: Dian Fu
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f
{code}
020-10-01T21:55:34.9982203Z "main" #1 prio=5 os_prio=0 cpu=1048.80ms
elapsed=921.99s tid=0x7f8c00015800 nid=0xf6e in Object.wait()
[0x7f8c06648000]
2020-10-01T21:55:34.9982807Zjava.lang.Thread.State: WAITING (on object
monitor)
2020-10-01T21:55:34.9983177Zat
java.lang.Object.wait(java.base@11.0.7/Native Method)
2020-10-01T21:55:34.9983871Z- waiting on <0x8e0be190> (a
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
2020-10-01T21:55:34.9984581Zat
java.lang.Thread.join(java.base@11.0.7/Thread.java:1305)
2020-10-01T21:55:34.9985433Z- waiting to re-lock in wait()
<0x8e0be190> (a
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
2020-10-01T21:55:34.9985998Zat
org.apache.flink.core.testutils.CheckedThread.trySync(CheckedThread.java:112)
2020-10-01T21:55:34.9986511Zat
org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:100)
2020-10-01T21:55:34.9987004Zat
org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:89)
2020-10-01T21:55:34.9987707Zat
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread.shutdown(SplitFetcherTest.java:301)
2020-10-01T21:55:34.9988427Zat
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent(SplitFetcherTest.java:131)
2020-10-01T21:55:34.9989025Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.7/Native
Method)
2020-10-01T21:55:34.9989531Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.7/NativeMethodAccessorImpl.java:62)
2020-10-01T21:55:34.9990117Zat
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.7/DelegatingMethodAccessorImpl.java:43)
2020-10-01T21:55:34.9990626Zat
java.lang.reflect.Method.invoke(java.base@11.0.7/Method.java:566)
2020-10-01T21:55:34.9991078Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-10-01T21:55:34.9991602Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-10-01T21:55:34.9992119Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-10-01T21:55:34.9992749Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-10-01T21:55:34.9993229Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-10-01T21:55:34.9993700Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-10-01T21:55:34.9994202Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-10-01T21:55:34.9994670Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-10-01T21:55:34.9995098Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-10-01T21:55:34.9995524Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-10-01T21:55:34.9995965Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-10-01T21:55:34.9996403Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-10-01T21:55:34.9996816Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-10-01T21:55:34.9997268Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2020-10-01T21:55:34.9997695Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2020-10-01T21:55:34.9998077Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-10-01T21:55:34.9998510Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-10-01T21:55:34.9998941Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-10-01T21:55:34.380Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-10-01T21:55:34.815Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-10-01T21:55:35.226Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-10-01T21:55:35.662Zat
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2020-10-01T21:55:35.0001190Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2020-10-01T21:55:35.0001746Zat
[jira] [Updated] (FLINK-19489) SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
[
https://issues.apache.org/jira/browse/FLINK-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19489:
Issue Type: Bug (was: Improvement)
> SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent gets stuck
> ---
>
> Key: FLINK-19489
> URL: https://issues.apache.org/jira/browse/FLINK-19489
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=298e20ef-7951-5965-0e79-ea664ddc435e=b4cd3436-dbe8-556d-3bca-42f92c3cbf2f
> {code}
> 020-10-01T21:55:34.9982203Z "main" #1 prio=5 os_prio=0 cpu=1048.80ms
> elapsed=921.99s tid=0x7f8c00015800 nid=0xf6e in Object.wait()
> [0x7f8c06648000]
> 2020-10-01T21:55:34.9982807Zjava.lang.Thread.State: WAITING (on object
> monitor)
> 2020-10-01T21:55:34.9983177Z at
> java.lang.Object.wait(java.base@11.0.7/Native Method)
> 2020-10-01T21:55:34.9983871Z - waiting on <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9984581Z at
> java.lang.Thread.join(java.base@11.0.7/Thread.java:1305)
> 2020-10-01T21:55:34.9985433Z - waiting to re-lock in wait()
> <0x8e0be190> (a
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread)
> 2020-10-01T21:55:34.9985998Z at
> org.apache.flink.core.testutils.CheckedThread.trySync(CheckedThread.java:112)
> 2020-10-01T21:55:34.9986511Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:100)
> 2020-10-01T21:55:34.9987004Z at
> org.apache.flink.core.testutils.CheckedThread.sync(CheckedThread.java:89)
> 2020-10-01T21:55:34.9987707Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest$QueueDrainerThread.shutdown(SplitFetcherTest.java:301)
> 2020-10-01T21:55:34.9988427Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent(SplitFetcherTest.java:131)
> 2020-10-01T21:55:34.9989025Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.7/Native
> Method)
> 2020-10-01T21:55:34.9989531Z at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.7/NativeMethodAccessorImpl.java:62)
> 2020-10-01T21:55:34.9990117Z at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.7/DelegatingMethodAccessorImpl.java:43)
> 2020-10-01T21:55:34.9990626Z at
> java.lang.reflect.Method.invoke(java.base@11.0.7/Method.java:566)
> 2020-10-01T21:55:34.9991078Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-01T21:55:34.9991602Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-01T21:55:34.9992119Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-01T21:55:34.9992749Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-01T21:55:34.9993229Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-01T21:55:34.9993700Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-01T21:55:34.9994202Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-01T21:55:34.9994670Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9995098Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9995524Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.9995965Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-01T21:55:34.9996403Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-01T21:55:34.9996816Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-01T21:55:34.9997268Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-01T21:55:34.9997695Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-01T21:55:34.9998077Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-01T21:55:34.9998510Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-01T21:55:34.9998941Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-01T21:55:34.380Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-01T21:55:34.815Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-01T21:55:35.226Z
[jira] [Updated] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19027:
Affects Version/s: 1.12.0
> UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel
> failed because of test timeout
>
>
> Key: FLINK-19027
> URL: https://issues.apache.org/jira/browse/FLINK-19027
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0, 1.11.3
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5789=logs=119bbba7-f5e3-5e08-e72d-09f1529665de=ec103906-d047-5b8a-680e-05fc000dfca9]
> {code}
> 2020-08-22T21:13:05.5315459Z [ERROR]
> shouldPerformUnalignedCheckpointOnParallelRemoteChannel(org.apache.flink.test.checkpointing.UnalignedCheckpointITCase)
> Time elapsed: 300.075 s <<< ERROR!
> 2020-08-22T21:13:05.5316451Z org.junit.runners.model.TestTimedOutException:
> test timed out after 300 seconds
> 2020-08-22T21:13:05.5317432Z at sun.misc.Unsafe.park(Native Method)
> 2020-08-22T21:13:05.5317799Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-08-22T21:13:05.5318247Z at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2020-08-22T21:13:05.5318885Z at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> 2020-08-22T21:13:05.5327035Z at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2020-08-22T21:13:05.5328114Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-08-22T21:13:05.5328869Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1719)
> 2020-08-22T21:13:05.5329482Z at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> 2020-08-22T21:13:05.5330138Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1699)
> 2020-08-22T21:13:05.5330771Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1681)
> 2020-08-22T21:13:05.5331351Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.execute(UnalignedCheckpointITCase.java:158)
> 2020-08-22T21:13:05.5332015Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel(UnalignedCheckpointITCase.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205904#comment-17205904
]
Dian Fu edited comment on FLINK-19027 at 10/2/20, 12:43 AM:
Another instance on master:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=ae4f8708-9994-57d3-c2d7-b892156e7812=5c8e7682-d68f-54d1-16a2-a09310218a49
Found the following exceptions in the log, hope it could help to identify the
issue:
{code}
21:38:30,457 [ Map (1/1)] WARN
org.apache.flink.runtime.taskmanager.Task[] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:164)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:368)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:185)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:594)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:558)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
21:38:30,458 [ Map (1/1)] INFO
org.apache.flink.runtime.taskmanager.Task[] - Freeing task
resources for Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0).
21:38:30,458 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Un-registering task and sending final execution state FAILED to JobManager for
task Map (1/1)
cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0.
21:38:30,459 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@2535c61e.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
[jira] [Comment Edited] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205904#comment-17205904
]
Dian Fu edited comment on FLINK-19027 at 10/2/20, 12:38 AM:
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=ae4f8708-9994-57d3-c2d7-b892156e7812=5c8e7682-d68f-54d1-16a2-a09310218a49
Found the following exceptions in the log, hope it could help to identify the
issue:
{code}
21:38:30,457 [ Map (1/1)] WARN
org.apache.flink.runtime.taskmanager.Task[] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:164)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:368)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:185)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:594)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:558)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
21:38:30,458 [ Map (1/1)] INFO
org.apache.flink.runtime.taskmanager.Task[] - Freeing task
resources for Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0).
21:38:30,458 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Un-registering task and sending final execution state FAILED to JobManager for
task Map (1/1)
cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0.
21:38:30,459 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@2535c61e.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
[jira] [Commented] (FLINK-19027) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel failed because of test timeout
[
https://issues.apache.org/jira/browse/FLINK-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205904#comment-17205904
]
Dian Fu commented on FLINK-19027:
-
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7158=logs=ae4f8708-9994-57d3-c2d7-b892156e7812=5c8e7682-d68f-54d1-16a2-a09310218a49
{code}
21:38:30,457 [ Map (1/1)] WARN
org.apache.flink.runtime.taskmanager.Task[] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:164)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:368)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:185)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:594)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:558)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
21:38:30,458 [ Map (1/1)] INFO
org.apache.flink.runtime.taskmanager.Task[] - Freeing task
resources for Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0).
21:38:30,458 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Un-registering task and sending final execution state FAILED to JobManager for
task Map (1/1)
cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0.
21:38:30,459 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(cb2b6ee8f94a6516932dfa923bfede3f_0a448493b4782967b150582570326227_0_0)
switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@2535c61e.
java.lang.Exception: Failing map @ 4 (0 attempt); last value 391
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.failMapper(UnalignedCheckpointITCase.java:616)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.checkFail(UnalignedCheckpointITCase.java:611)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:605)
~[test-classes/:?]
at
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$FailingMapper.map(UnalignedCheckpointITCase.java:580)
~[test-classes/:?]
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-streaming-java_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:164)
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * 0e3916e68d38b432ebc33248d3a1ce7f7213b1f8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6968) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6572) * ce25c6bbc562da022b86aaa99d24f40c1d159b41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18366) Track E2E test durations centrally
[ https://issues.apache.org/jira/browse/FLINK-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205902#comment-17205902 ] Dian Fu commented on FLINK-18366: - slow run: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7153=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529 > Track E2E test durations centrally > -- > > Key: FLINK-18366 > URL: https://issues.apache.org/jira/browse/FLINK-18366 > Project: Flink > Issue Type: Improvement > Components: Build System / CI >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Major > > Every now and then, our E2E tests start timing out (see FLINK-16795), because > they hit the currently configured time-limit. > To better understand what the expected E2E time, and potential performance > regressions, we should track the test execution duration centrally. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] kezhuw commented on a change in pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
kezhuw commented on a change in pull request #13375:
URL: https://github.com/apache/flink/pull/13375#discussion_r498570772
##
File path:
flink-core/src/main/java/org/apache/flink/core/fs/SafetyNetCloseableRegistry.java
##
@@ -134,10 +135,17 @@ public void close() throws IOException {
--GLOBAL_SAFETY_NET_REGISTRY_COUNT;
if (0 == GLOBAL_SAFETY_NET_REGISTRY_COUNT) {
REAPER_THREAD.interrupt();
Review comment:
All reclamation related classes and methods, say,
`CloseableReaperThread`, `PhantomDelegatingCloseableRef` and
`SafetyNetCloseableRegistry(Supplier
reaperThreadSupplier)`, are private or visible for testing, normal clients
should care only about the reclamation of leaking resources guarantee exported
by `SafetyNetCloseableRegistry`, "join on the reaper thread" does not
contribute to that exported guarantee.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] kezhuw commented on a change in pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
kezhuw commented on a change in pull request #13375:
URL: https://github.com/apache/flink/pull/13375#discussion_r498570772
##
File path:
flink-core/src/main/java/org/apache/flink/core/fs/SafetyNetCloseableRegistry.java
##
@@ -134,10 +135,17 @@ public void close() throws IOException {
--GLOBAL_SAFETY_NET_REGISTRY_COUNT;
if (0 == GLOBAL_SAFETY_NET_REGISTRY_COUNT) {
REAPER_THREAD.interrupt();
Review comment:
All reclamation related classes and methods, say,
`CloseableReaperThread`, `PhantomDelegatingCloseableRef` and
`SafetyNetCloseableRegistry(Supplier
reaperThreadSupplier)`, are private or visible for testing, normal clients
should care only about the reclamation of leaking resources guarantee exported
by `SafetyNetCloseableRegistry`, "join on the reaper thread" does not belong to
that exported guarantee.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * b7162b6109b03f905f2c52cc22002b198bc9438e UNKNOWN * d83bf413dd7661c4ee87a692b80147f2a6750f46 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7157) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13529: [FLINK-19473] Implement multi inputs sorting DataInput
flinkbot edited a comment on pull request #13529: URL: https://github.com/apache/flink/pull/13529#issuecomment-702193043 ## CI report: * 6270bf09d731ac358ca621e4150c817153882eb9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7155) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * c5164bceb05c9e11c652fdb2a0cefe289af0ec1b UNKNOWN * 39e66d00bae3c21b41c12881cd2680f74ffbf2e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7154) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on pull request #13512: [FLINK-19457][core] Add a number sequence generating source for the New Source API.
StephanEwen commented on pull request #13512: URL: https://github.com/apache/flink/pull/13512#issuecomment-702303125 No worries, let's figure this out together. ### About the flink-core package structure. Agree, `flink-core`'s package structure is wild. Most of it is because of the pre-streaming origin, and there was not too much thought there. `api.common` was shared between Batch Java and Batch Scala APIs. `api.java` was originally in a different project (Batch Java API) and moved to core when streaming came into the mix to be shared with the Streaming Java API. `core.io` was IIRC IO-related stuff that was not part of the API, but indirectly relevant in core API classes, like the utils around the managed memory abstraction. `api.common.io` was for API-related I/O, like sources, sinks. A lot of stuff was also added over time in the wrong places, admittedly, because few people in the early days paid attention to package structure. The packages stuck because we didn't want to break user applications. For Flink 2.0, we should try and simplify this. tl;dr - it is the way it is for historic reasons and because the "not breaking application" consideration ended up trumping the "clean up packages" consideration. Seems the structure was not getting too much in the way of users after all. ### "flink-connector-base" versus "flink-core" What I would like to keep is that this source implementation stays in `flink-core` and does not move to another module. We have a lot of dependencies already and it really doesn't hurt in my opinion to have that implementation there in the core (given that it is lightweight and has no dependencies). The Split-Reader API could be moved to flink-core, theoretically. I didn't argue for that in the original PR, because it is not used for simple sources (like here), but typically for sources with external systems that need separate I/O threads. These sources would be in separate modules anyways. So having the Split Reader API in a separate module would not make any difference to users, so I just went with your suggestion there (if it makes no difference, I typically go with what the code author suggests). IT has the slight advantage even. of being more flexible with test dependencies, so Integration Tests can be in the module directly. ### Packages in this PR About the `lib` and `util` package names, I am not opinionated on those. In fact, I don't particularly like `lib` either. Anything that is reasonably "discoverable" for users would work here from my side. Let's try to up with different names or locations, I am definitely open to suggestions. Maybe we'll move this in a follow-up effort, to help unblock some features where I wanted to use this source in the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
root
{{ |-- C0: BIGINT}}
{{ |-- C1: STRING}}
{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
was:
I have a table T0 with the following schema -
{{root}}
|-- C0: BIGINT
|-- C1: STRING
|-- blaze_itime: TIMESTAMP(3)
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
{{ |-- C0: BIGINT}}
{{ |-- C1: STRING}}
{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
was:
I have a table T0 with the following schema -
{{root}}
|-- C0: BIGINT
|-- C1: STRING
|-- blaze_itime: TIMESTAMP(3)
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
|-- C0: BIGINT
|-- C1: STRING
|-- blaze_itime: TIMESTAMP(3)
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
was:
I have a table T0 with the following schema -
{{root}}
{{ |-- C0: BIGINT}}
{{ |-- C1: STRING}}
{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Environment:
Flink version: 1.11.1
was:
Flink version: 1.11.1
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> \{{ |-- C0: BIGINT}}
> \{{ |-- C1: STRING}}
> \{{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() - INTERVAL '10' MINUTE AND NOW());}}
>
> {{The generated code for the above query tries to assign a long value to
> timestamp type and fails to compile with the following exception -}}
>
> {{java.lang.RuntimeException: Could not instantiate generated class
> 'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated
> class 'BatchCalc$14' at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
> at
> org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
> at}}
> {{...}}
> {{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
> 21: Assignment conversion not possible from type "long" to type
> "org.apache.flink.table.data.TimestampData"Caused by:
> org.codehaus.commons.compiler.CompileException: Line 55, Column 21:
> Assignment conversion not possible from type "long" to type
> "org.apache.flink.table.data.TimestampData" at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
> org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
> org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
>
> The generated code is added as an attachment to the issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
was:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> \{{ |-- C0: BIGINT}}
> \{{ |-- C1: STRING}}
> \{{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() - INTERVAL '10' MINUTE AND
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
|-- C0: BIGINT
|-- C1: STRING
|-- blaze_itime: TIMESTAMP(3)
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
was:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
>
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at}}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
was:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at
...
{{
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
}}
The generated code is added as an attachment to the issue.
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
> The Environment has the following configuration parameters -
> {{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
> {{env.getConfig().enableObjectReuse();}}
> {{var settings = EnvironmentSettings.newInstance()}}
> {{ .useBlinkPlanner()}}
> {{ .inBatchMode()}}
> {{ .build();}}
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> \{{ |-- C0: BIGINT}}
> \{{ |-- C1: STRING}}
> \{{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() - INTERVAL '10'
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at
...
{{
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
}}
The generated code is added as an attachment to the issue.
was:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at ...Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)
The generated code is added as an attachment to the issue.
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
> The Environment has the following configuration parameters -
> {{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
> {{env.getConfig().enableObjectReuse();}}
> {{var settings = EnvironmentSettings.newInstance()}}
> {{ .useBlinkPlanner()}}
> {{ .inBatchMode()}}
> {{ .build();}}
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> \{{ |-- C0: BIGINT}}
> \{{ |-- C1: STRING}}
> \{{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() -
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at }}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
was:
I have a table T0 with the following schema -
{{root}}
{{ |-- C0: BIGINT}}
{{ |-- C1: STRING}}
{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at }}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
{{Here is the generated code -}}
/* 1 */
/* 2 */ public class BatchCalc$14 extends
org.apache.flink.table.runtime.operators.TableStreamOperator
/* 3 */ implements
org.apache.flink.streaming.api.operators.OneInputStreamOperator {
/* 4 */
/* 5 */ private final Object[] references;
/* 6 */ org.apache.flink.table.data.BoxedWrapperRowData out = new
org.apache.flink.table.data.BoxedWrapperRowData(2);
/* 7 */ private final
org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement = new
org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);
/* 8 */
/* 9 */ public BatchCalc$14(
/* 10 */ Object[] references,
/* 11 */ org.apache.flink.streaming.runtime.tasks.StreamTask task,
/* 12 */ org.apache.flink.streaming.api.graph.StreamConfig config,
/* 13 */ org.apache.flink.streaming.api.operators.Output output,
/* 14 */ org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
processingTimeService) throws Exception {
/* 15 */ this.references = references;
/* 16 */
/* 17 */ this.setup(task, config, output);
/* 18 */ if (this instanceof
org.apache.flink.streaming.api.operators.AbstractStreamOperator) {
/* 19 */
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Description:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at ...Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)
The generated code is added as an attachment to the issue.
was:
I have a table T0 with the following schema -
{{root}}
\{{ |-- C0: BIGINT}}
\{{ |-- C1: STRING}}
\{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at }}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
The generated code is added as an attachment to the issue.
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
> The Environment has the following configuration parameters -
> {{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
> {{env.getConfig().enableObjectReuse();}}
> {{var settings = EnvironmentSettings.newInstance()}}
> {{ .useBlinkPlanner()}}
> {{ .inBatchMode()}}
> {{ .build();}}
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> \{{ |-- C0: BIGINT}}
> \{{ |-- C1: STRING}}
> \{{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() - INTERVAL '10' MINUTE AND
[jira] [Updated] (FLINK-19488) Failed compilation of generated class
[
https://issues.apache.org/jira/browse/FLINK-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-19488:
---
Attachment: code.java
> Failed compilation of generated class
> -
>
> Key: FLINK-19488
> URL: https://issues.apache.org/jira/browse/FLINK-19488
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
> Environment: Flink version: 1.11.1
> The Environment has the following configuration parameters -
> {{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
> {{env.getConfig().enableObjectReuse();}}
> {{var settings = EnvironmentSettings.newInstance()}}
> {{ .useBlinkPlanner()}}
> {{ .inBatchMode()}}
> {{ .build();}}
>Reporter: Satyam Shekhar
>Priority: Major
> Attachments: code.java
>
>
> I have a table T0 with the following schema -
> {{root}}
> {{ |-- C0: BIGINT}}
> {{ |-- C1: STRING}}
> {{ |-- blaze_itime: TIMESTAMP(3)}}
>
> The following SQL query fails for the above table -
> {{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime
> BETWEEN NOW() - INTERVAL '10' MINUTE AND NOW());}}
>
> {{The generated code for the above query tries to assign a long value to
> timestamp type and fails to compile with the following exception -}}
> {{java.lang.RuntimeException: Could not instantiate generated class
> 'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated
> class 'BatchCalc$14' at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
> at
> org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
> at }}
> {{...}}
> {{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
> 21: Assignment conversion not possible from type "long" to type
> "org.apache.flink.table.data.TimestampData"Caused by:
> org.codehaus.commons.compiler.CompileException: Line 55, Column 21:
> Assignment conversion not possible from type "long" to type
> "org.apache.flink.table.data.TimestampData" at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
> org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
> org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
>
> {{Here is the generated code -}}
> /* 1 */
> /* 2 */ public class BatchCalc$14 extends
> org.apache.flink.table.runtime.operators.TableStreamOperator
> /* 3 */ implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator {
> /* 4 */
> /* 5 */ private final Object[] references;
> /* 6 */ org.apache.flink.table.data.BoxedWrapperRowData out = new
> org.apache.flink.table.data.BoxedWrapperRowData(2);
> /* 7 */ private final
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement = new
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);
> /* 8 */
> /* 9 */ public BatchCalc$14(
> /* 10 */ Object[] references,
> /* 11 */ org.apache.flink.streaming.runtime.tasks.StreamTask task,
> /* 12 */ org.apache.flink.streaming.api.graph.StreamConfig config,
> /* 13 */ org.apache.flink.streaming.api.operators.Output output,
> /* 14 */ org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
> processingTimeService) throws Exception {
> /* 15 */ this.references = references;
> /* 16 */
> /* 17 */ this.setup(task, config, output);
> /* 18 */ if (this instanceof
> org.apache.flink.streaming.api.operators.AbstractStreamOperator) {
> /* 19 */ ((org.apache.flink.streaming.api.operators.AbstractStreamOperator)
> this)
> /* 20 */ .setProcessingTimeService(processingTimeService);
> /* 21 */ }
> /* 22 */ }
> /* 23 */
> /* 24 */ @Override
> /* 25 */ public void open() throws Exception {
> /* 26 */ super.open();
> /* 27 */
> /* 28 */ }
> /* 29 */
> /* 30 */ @Override
> /* 31 */ public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception {
> /* 32 */ org.apache.flink.table.data.RowData in1 =
> (org.apache.flink.table.data.RowData) element.getValue();
> /* 33 */
> /* 34 */ org.apache.flink.table.data.TimestampData field$2;
> /* 35 */ boolean isNull$2;
> /* 36 */ boolean isNull$3;
> /* 37 */ org.apache.flink.table.data.TimestampData result$4;
> /* 38 */ boolean isNull$5;
> /* 39 */ org.apache.flink.table.data.TimestampData result$6;
> /* 40 */ boolean isNull$7;
> /* 41 */ boolean result$8;
> /* 42 */ boolean isNull$9;
> /* 43 */
[jira] [Created] (FLINK-19488) Failed compilation of generated class
Satyam Shekhar created FLINK-19488:
--
Summary: Failed compilation of generated class
Key: FLINK-19488
URL: https://issues.apache.org/jira/browse/FLINK-19488
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.11.1
Environment: Flink version: 1.11.1
The Environment has the following configuration parameters -
{{env.setParallelism(Integer.getInteger("flinkParallelism", 2));}}
{{env.getConfig().enableObjectReuse();}}
{{var settings = EnvironmentSettings.newInstance()}}
{{ .useBlinkPlanner()}}
{{ .inBatchMode()}}
{{ .build();}}
Reporter: Satyam Shekhar
I have a table T0 with the following schema -
{{root}}
{{ |-- C0: BIGINT}}
{{ |-- C1: STRING}}
{{ |-- blaze_itime: TIMESTAMP(3)}}
The following SQL query fails for the above table -
{{SELECT A.C0 AS output, A.C1 AS dim0 FROM T0 AS A WHERE (A.blaze_itime BETWEEN
NOW() - INTERVAL '10' MINUTE AND NOW());}}
{{The generated code for the above query tries to assign a long value to
timestamp type and fails to compile with the following exception -}}
{{java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14'java.lang.RuntimeException: Could not instantiate generated class
'BatchCalc$14' at
org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:67)
at
org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
at
org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:70)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:470)
at }}
{{...}}
{{Caused by: org.codehaus.commons.compiler.CompileException: Line 55, Column
21: Assignment conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData"Caused by:
org.codehaus.commons.compiler.CompileException: Line 55, Column 21: Assignment
conversion not possible from type "long" to type
"org.apache.flink.table.data.TimestampData" at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12124) at
org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10975)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3788) at
org.codehaus.janino.UnitCompiler.access$6100(UnitCompiler.java:215)}}
{{Here is the generated code -}}
/* 1 */
/* 2 */ public class BatchCalc$14 extends
org.apache.flink.table.runtime.operators.TableStreamOperator
/* 3 */ implements
org.apache.flink.streaming.api.operators.OneInputStreamOperator {
/* 4 */
/* 5 */ private final Object[] references;
/* 6 */ org.apache.flink.table.data.BoxedWrapperRowData out = new
org.apache.flink.table.data.BoxedWrapperRowData(2);
/* 7 */ private final
org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement = new
org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);
/* 8 */
/* 9 */ public BatchCalc$14(
/* 10 */ Object[] references,
/* 11 */ org.apache.flink.streaming.runtime.tasks.StreamTask task,
/* 12 */ org.apache.flink.streaming.api.graph.StreamConfig config,
/* 13 */ org.apache.flink.streaming.api.operators.Output output,
/* 14 */ org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
processingTimeService) throws Exception {
/* 15 */ this.references = references;
/* 16 */
/* 17 */ this.setup(task, config, output);
/* 18 */ if (this instanceof
org.apache.flink.streaming.api.operators.AbstractStreamOperator) {
/* 19 */ ((org.apache.flink.streaming.api.operators.AbstractStreamOperator)
this)
/* 20 */ .setProcessingTimeService(processingTimeService);
/* 21 */ }
/* 22 */ }
/* 23 */
/* 24 */ @Override
/* 25 */ public void open() throws Exception {
/* 26 */ super.open();
/* 27 */
/* 28 */ }
/* 29 */
/* 30 */ @Override
/* 31 */ public void
processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
element) throws Exception {
/* 32 */ org.apache.flink.table.data.RowData in1 =
(org.apache.flink.table.data.RowData) element.getValue();
/* 33 */
/* 34 */ org.apache.flink.table.data.TimestampData field$2;
/* 35 */ boolean isNull$2;
/* 36 */ boolean isNull$3;
/* 37 */ org.apache.flink.table.data.TimestampData result$4;
/* 38 */ boolean isNull$5;
/* 39 */ org.apache.flink.table.data.TimestampData result$6;
/* 40 */ boolean isNull$7;
/* 41 */ boolean result$8;
/* 42 */ boolean isNull$9;
/* 43 */ org.apache.flink.table.data.TimestampData result$10;
/* 44 */ boolean isNull$11;
/* 45 */ boolean result$12;
/* 46 */
/* 47 */
/* 48 */ isNull$2 = in1.isNullAt(2);
/* 49 */ field$2 = null;
/* 50 */ if (!isNull$2) {
/* 51 */ field$2 = in1.getTimestamp(2, 3);
/* 52 */ }
/* 53 */ isNull$3 = false;
/* 54 */
/* 55 */ result$4 =
/* 56 */ org.apache.flink.table.runtime.functions.SqlDateTimeUtils.now()
/* 57 */ ;
/* 58 */
/* 59 */
/* 60 */
[GitHub] [flink] azagrebin commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
azagrebin commented on a change in pull request #13458:
URL: https://github.com/apache/flink/pull/13458#discussion_r498390360
##
File path: flink-runtime-web/src/test/resources/rest_api_v1.snapshot
##
@@ -557,7 +557,7 @@
},
"status" : {
"type" : "string",
-"enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED",
"CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ]
+"enum" : [ "INITIALIZING", "CREATED", "RUNNING", "FAILING",
"FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED",
"RECONCILING" ]
Review comment:
is this related to this PR?
also the next similar change in this file?
it does not look related to checkpointing, then it should be addressed at
least in a separate hotfix commit or better in another PR/Jira issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
azagrebin commented on a change in pull request #13458:
URL: https://github.com/apache/flink/pull/13458#discussion_r498392633
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointStatistics.java
##
@@ -229,8 +242,8 @@ public static CheckpointStatistics
generateCheckpointStatistics(AbstractCheckpoi
checkpointStatisticsPerTask.put(
taskStateStat.getJobVertexId(),
new TaskCheckpointStatistics(
-
checkpointStats.getCheckpointId(),
-
checkpointStats.getStatus(),
+
checkpointStats.getCheckpointId(),
+ checkpointStats.getStatus(),
Review comment:
this change also looks unrelated.
it looks like a minor code style fix which we usually do not do to preserve
git history.
if we do the code style fixes then it is usually either
- massively in files which have a lot of code style violation in separate
commit
- or close to related changes
##
File path: flink-runtime-web/src/test/resources/rest_api_v1.snapshot
##
@@ -557,7 +557,7 @@
},
"status" : {
"type" : "string",
-"enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED",
"CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ]
+"enum" : [ "INITIALIZING", "CREATED", "RUNNING", "FAILING",
"FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED",
"RECONCILING" ]
Review comment:
is this related to this PR?
also the next similar change in this file?
it does not look related to checkpointing, then it should addressed at least
in a separate hotfix commit or better in another PR/Jira issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI
azagrebin commented on a change in pull request #13458:
URL: https://github.com/apache/flink/pull/13458#discussion_r498389552
##
File path: flink-runtime-web/src/test/resources/rest_api_v1.snapshot
##
@@ -1254,6 +1263,9 @@
"num_acknowledged_subtasks" : {
"type" : "integer"
},
+"check_point_type" : {
Review comment:
It should be probably documented in `flink-runtime-web/README.md` if not
yet anywhere else
what do you think @zentol ?
@gm7y8 if it is not documented anywhere else, you can create a JIRA issue
for that and then open PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #13512: [FLINK-19457][core] Add a number sequence generating source for the New Source API.
becketqin commented on pull request #13512: URL: https://github.com/apache/flink/pull/13512#issuecomment-702245190 @StephanEwen Thanks for the explanation. I checked the classes and structure in `flink-core` again, I agree that this PR is in general following the existing convention. And I am fine with merge the PR as is. That said, I had a hard time to fully understand the structure of `flink-core`. It would be great if you can help me understand this a bit more. I am wondering if the users would have the same questions like below. - I was initially expecting the function interfaces in `org.apache.api.java.functions` path, but there are only `KeySelector` and `NullByteKeySelector`. The actual function interfaces are in `org.apache.flink.common.functions` package. What is the difference between those two packages? - Also, I was not quite sure about the differences between `org.apache.flink.api.common.io` and `org.apache.flink.core.io`. - In the existing package paths in `flink-core`, the implementation and interface classes are put in the same package. This PR introduces a new package `lib`. Should the other implementations also be put in a `lib` path? - In the existing package paths, `util` packages usually contains internal helper classes for the (abstract) implementations of the interfaces. This PR seems putting the abstract implementations of the `SourceReader` / `Enumerator` interfaces in the `util` package, and expecting other Iterator based sources to leverage that? Or are those classes also internal? - The follow up question to the above one, it seems that the `IteratorSourceReader`/ `IteratorSourceEnumerator` classes are just another base implementation of the `Source` interface, why the base implementation of the source in `flink-connector-base` was not put into `org.apache.flink.api.connector.source.lib.util`? Sorry for sticking to discussion, my main concern is that If we ever want to revisit the API and package structure, right now might be the best timing, because we are essentially introducing new `Source` APIs and will probably deprecate old `SourceFunction` and `InputFormat` APIs in some coming release. This would be a natural migration towards new APIs if we want to do that. Keeping the API / package structure as is and changing them in the future could be even more disruptive. Thanks again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin edited a comment on pull request #13512: [FLINK-19457][core] Add a number sequence generating source for the New Source API.
becketqin edited a comment on pull request #13512: URL: https://github.com/apache/flink/pull/13512#issuecomment-702245190 @StephanEwen Thanks for the explanation. I checked the classes and structure in `flink-core` again, I agree that this PR is in general following the existing convention. And I am fine with merging the PR as is. That said, I had a hard time to fully understand the structure of `flink-core`. It would be great if you can help me understand this a bit more. I am wondering if the users would have the same questions like below. - I was initially expecting the function interfaces in `org.apache.api.java.functions` path, but there are only `KeySelector` and `NullByteKeySelector`. The actual function interfaces are in `org.apache.flink.common.functions` package. What is the difference between those two packages? - Also, I was not quite sure about the differences between `org.apache.flink.api.common.io` and `org.apache.flink.core.io`. - In the existing package paths in `flink-core`, the implementation and interface classes are put in the same package. This PR introduces a new package `lib`. Should the other implementations also be put in a `lib` path? - In the existing package paths, `util` packages usually contains internal helper classes for the (abstract) implementations of the interfaces. This PR seems putting the abstract implementations of the `SourceReader` / `Enumerator` interfaces in the `util` package, and expecting other Iterator based sources to leverage that? Or are those classes also internal? - The follow up question to the above one, it seems that the `IteratorSourceReader`/ `IteratorSourceEnumerator` classes are just another base implementation of the `Source` interface, why the base implementation of the source in `flink-connector-base` was not put into `org.apache.flink.api.connector.source.lib.util`? Sorry for sticking to discussion, my main concern is that If we ever want to revisit the API and package structure, right now might be the best timing, because we are essentially introducing new `Source` APIs and will probably deprecate old `SourceFunction` and `InputFormat` APIs in some coming release. This would be a natural migration towards new APIs if we want to do that. Keeping the API / package structure as is and changing them in the future could be even more disruptive. Thanks again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * b7162b6109b03f905f2c52cc22002b198bc9438e UNKNOWN * 5fd3f465ce953ac5944f9f8516259245f50d8f9f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7086) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7087) * d83bf413dd7661c4ee87a692b80147f2a6750f46 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7157) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19319) Deprecate StreamExecutionEnvironment.setStreamTimeCharacteristic() and TimeCharacteristic
[
https://issues.apache.org/jira/browse/FLINK-19319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-19319.
Fix Version/s: 1.12.0
Release Note:
This is formatted in Markdown for easy inclusion in the release post.
In Flink 1.12 the default stream time characteristic has been changed to
`EventTime`, thus you don't need to call this method for enabling event-time
support anymore. Explicitly using processing-time windows and timers works in
event-time mode. If you need to disable watermarks, please use
`ExecutionConfig.setAutoWatermarkInterval(long)`. If you are using
`IngestionTime`, please manually set an appropriate `WatermarkStrategy`. If you
are using generic "time window" operations (for example
`KeyedStream.timeWindow()` that change behaviour based on the time
characteristic, please use equivalent operations that explicitly specify
processing time or event time.
Resolution: Fixed
master: f8cc82b0c7d3ddd35b17c7f6475b8908363c930a
> Deprecate StreamExecutionEnvironment.setStreamTimeCharacteristic() and
> TimeCharacteristic
> -
>
> Key: FLINK-19319
> URL: https://issues.apache.org/jira/browse/FLINK-19319
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.12.0
>
>
> After FLINK-19317 and FLINK-19318 we don't need this setting anymore. Using
> (explicit) processing-time windows and processing-time timers work fine in a
> program that has {{EventTime}} set as a time characteristic and once we
> deprecate {{timeWindow()}} there are not other operations that change
> behaviour depending on the time characteristic so there's no need to ever
> change from the new default of event-time. Similarly, the {{IngestionTime}}
> setting can be achieved in the future by providing an ingestion-time
> {{WatermarkStrategy}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13529: [FLINK-19473] Implement multi inputs sorting DataInput
flinkbot edited a comment on pull request #13529: URL: https://github.com/apache/flink/pull/13529#issuecomment-702193043 ## CI report: * 6270bf09d731ac358ca621e4150c817153882eb9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7155) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19317) Make EventTime the default StreamTimeCharacteristic
[
https://issues.apache.org/jira/browse/FLINK-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-19317.
Fix Version/s: 1.12.0
Resolution: Fixed
master: 1175364cc9f5bc6018d4c3f231d24cb8ff5e6341
> Make EventTime the default StreamTimeCharacteristic
> ---
>
> Key: FLINK-19317
> URL: https://issues.apache.org/jira/browse/FLINK-19317
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Excerpt from FLIP-134:
> {quote}
> As described above, event time is the only sensible time characteristic for
> batch. We therefore propose to change the default value of the
> StreamTimeCharacteristic from ProcessingTime to EventTime. This means the
> DataStream API programs that were using event time before now just work
> without manually changing this setting. Processing-time programs will also
> still work, because using processing-time timers is not dependent on the
> StreamTimeCharacteristic. DataStream programs that don't set a
> TimestampAssigner or WatermarkStrategy will also still work if they don't use
> operations that don't rely on (event-time) timestamps. This is true for both
> BATCH and STREAMING execution mode.
> The only real user-visible change of this is that programs that used the
> KeyedStream.timeWindow()/DataStream.timeWindow() operation, which is
> dependent on the StreamTimeCharacteristic will now use event time by default.
> We don't think this operation is useful because the behaviour can be
> surprising. We recommend users always use an explicit processing-time window
> or event-time window.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] aljoscha commented on pull request #13509: [FLINK-19317] Make EventTime the default TimeCharacteristic
aljoscha commented on pull request #13509: URL: https://github.com/apache/flink/pull/13509#issuecomment-702205959 Thanks for the reviews! I merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha closed pull request #13509: [FLINK-19317] Make EventTime the default TimeCharacteristic
aljoscha closed pull request #13509: URL: https://github.com/apache/flink/pull/13509 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * b7162b6109b03f905f2c52cc22002b198bc9438e UNKNOWN * 5fd3f465ce953ac5944f9f8516259245f50d8f9f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7086) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7087) * d83bf413dd7661c4ee87a692b80147f2a6750f46 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-6215) Make the StatefulSequenceSource scalable.
[
https://issues.apache.org/jira/browse/FLINK-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205583#comment-17205583
]
Kostas Kloudas commented on FLINK-6215:
---
Yes [~sewen], I will close the issue. The PRs are already closed.
> Make the StatefulSequenceSource scalable.
> -
>
> Key: FLINK-6215
> URL: https://issues.apache.org/jira/browse/FLINK-6215
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.3.0
>Reporter: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
>
> Currently the {{StatefulSequenceSource}} instantiates all the elements to
> emit first and keeps them in memory. This is not scalable as for large
> sequences of elements this can lead to out of memory exceptions.
> To solve this, we can pre-partition the sequence of elements based on the
> {{maxParallelism}} parameter, and just keep state (to checkpoint) per such
> partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-6215) Make the StatefulSequenceSource scalable.
[
https://issues.apache.org/jira/browse/FLINK-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-6215.
-
Resolution: Won't Fix
> Make the StatefulSequenceSource scalable.
> -
>
> Key: FLINK-6215
> URL: https://issues.apache.org/jira/browse/FLINK-6215
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.3.0
>Reporter: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
>
> Currently the {{StatefulSequenceSource}} instantiates all the elements to
> emit first and keeps them in memory. This is not scalable as for large
> sequences of elements this can lead to out of memory exceptions.
> To solve this, we can pre-partition the sequence of elements based on the
> {{maxParallelism}} parameter, and just keep state (to checkpoint) per such
> partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] echauchot commented on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
echauchot commented on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-702194047 @tillrohrmann I rebased on master and squashed the PR to only 2 commits: - production code + UTest of the CheckpointsDecider - integration test Can we merge the PR ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13529: [FLINK-19473] Implement multi inputs sorting DataInput
flinkbot commented on pull request #13529: URL: https://github.com/apache/flink/pull/13529#issuecomment-702193043 ## CI report: * 6270bf09d731ac358ca621e4150c817153882eb9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * ad55b5a30711435f6caf0d52ba1cdd702af96fda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7151) * c5164bceb05c9e11c652fdb2a0cefe289af0ec1b UNKNOWN * 39e66d00bae3c21b41c12881cd2680f74ffbf2e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7154) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
flinkbot edited a comment on pull request #12056: URL: https://github.com/apache/flink/pull/12056#issuecomment-626167900 ## CI report: * c42ef1f62396948ece865a3b8628bea229d5ffb9 UNKNOWN * ad971fd47ab3ac9e813afae9713c75986646 UNKNOWN * 4d2d57b8c2f6810c03d284fdd5807b81a2b554d7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7120) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7150) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7142) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-6215) Make the StatefulSequenceSource scalable.
[
https://issues.apache.org/jira/browse/FLINK-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205572#comment-17205572
]
Stephan Ewen commented on FLINK-6215:
-
This should issue should be subsumed by this one: FLINK-19457
[~kkl0u] Should we close this and the related PRs?
> Make the StatefulSequenceSource scalable.
> -
>
> Key: FLINK-6215
> URL: https://issues.apache.org/jira/browse/FLINK-6215
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.3.0
>Reporter: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
>
> Currently the {{StatefulSequenceSource}} instantiates all the elements to
> emit first and keeps them in memory. This is not scalable as for large
> sequences of elements this can lead to out of memory exceptions.
> To solve this, we can pre-partition the sequence of elements based on the
> {{maxParallelism}} parameter, and just keep state (to checkpoint) per such
> partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * ad55b5a30711435f6caf0d52ba1cdd702af96fda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7151) * c5164bceb05c9e11c652fdb2a0cefe289af0ec1b UNKNOWN * 39e66d00bae3c21b41c12881cd2680f74ffbf2e1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13529: [FLINK-19473] Implement multi inputs sorting DataInput
flinkbot commented on pull request #13529: URL: https://github.com/apache/flink/pull/13529#issuecomment-702178922 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6270bf09d731ac358ca621e4150c817153882eb9 (Thu Oct 01 14:35:15 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19473) Implement multi inputs sorting DataInput
[ https://issues.apache.org/jira/browse/FLINK-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19473: --- Labels: pull-request-available (was: ) > Implement multi inputs sorting DataInput > > > Key: FLINK-19473 > URL: https://issues.apache.org/jira/browse/FLINK-19473 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19447) HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not initialized after 200000ms"
[
https://issues.apache.org/jira/browse/FLINK-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205566#comment-17205566
]
Miklos Gergely commented on FLINK-19447:
Thank you [~dian.fu]. Created https://issues.apache.org/jira/browse/HBASE-25140
to track the issue of the test mini cluster compatibility.
> HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not
> initialized after 20ms"
> ---
>
> Key: FLINK-19447
> URL: https://issues.apache.org/jira/browse/FLINK-19447
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Miklos Gergely
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=60581941-0138-53c0-39fe-86d62be5f407
> {code}
> 2020-09-28T21:52:21.2146147Z
> org.apache.flink.connector.hbase2.HBaseConnectorITCase Time elapsed: 208.382
> sec <<< ERROR!
> 2020-09-28T21:52:21.2146638Z java.io.IOException: Shutting down
> 2020-09-28T21:52:21.2147004Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
> 2020-09-28T21:52:21.2147637Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:116)
> 2020-09-28T21:52:21.2148120Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1142)
> 2020-09-28T21:52:21.2148831Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
> 2020-09-28T21:52:21.2149347Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1061)
> 2020-09-28T21:52:21.2149896Z at
> org.apache.flink.connector.hbase2.util.HBaseTestingClusterAutoStarter.setUp(HBaseTestingClusterAutoStarter.java:122)
> 2020-09-28T21:52:21.2150721Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-09-28T21:52:21.2151136Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-09-28T21:52:21.2151609Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-09-28T21:52:21.2152039Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-09-28T21:52:21.2152462Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-09-28T21:52:21.2152941Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-09-28T21:52:21.2153489Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-09-28T21:52:21.2153962Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-09-28T21:52:21.2154406Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-09-28T21:52:21.2154828Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-09-28T21:52:21.2155381Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
> 2020-09-28T21:52:21.2155864Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
> 2020-09-28T21:52:21.2156378Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-09-28T21:52:21.2156865Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> 2020-09-28T21:52:21.2157458Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> 2020-09-28T21:52:21.2157993Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> 2020-09-28T21:52:21.2158470Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> 2020-09-28T21:52:21.2158890Z Caused by: java.lang.RuntimeException: Master
> not initialized after 20ms
> 2020-09-28T21:52:21.2159350Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
> 2020-09-28T21:52:21.2159823Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
> 2020-09-28T21:52:21.2160270Z at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> 2020-09-28T21:52:21.2160800Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
> 2020-09-28T21:52:21.2161096Z ... 22 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dawidwys opened a new pull request #13529: [FLINK-19473] Implement multi inputs sorting DataInput
dawidwys opened a new pull request #13529: URL: https://github.com/apache/flink/pull/13529 ## What is the purpose of the change The PR is still rough on the edges. It requires updates to the commit messages. I am opening the PR to get early feedback. Moreover it is based on #13521 ## Brief change log See commits subjects for the list of changes. ## Verifying this change Added tests in: * MultiInputSortingDataInputsTest * SortedDataInputITCase * SortingBoundedInputITCase.java ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19487) Checkpoint start delay is always zero for single channel tasks
Piotr Nowojski created FLINK-19487:
--
Summary: Checkpoint start delay is always zero for single channel
tasks
Key: FLINK-19487
URL: https://issues.apache.org/jira/browse/FLINK-19487
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.11.2
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
Fix For: 1.12.0
In {{CheckpointBarrierAligner}} we are marking start of the checkpoint only if
the number of channels is more than 1. As a result of this,
{{checkpointStartDelay}} is never updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] StephanEwen commented on pull request #13501: Single task result partition type
StephanEwen commented on pull request #13501: URL: https://github.com/apache/flink/pull/13501#issuecomment-702157253 I took a look only at the partial record handling. I like the idea of not transporting the partial record offsets at all across the networks, which means we don't have to make the deserializers aware of that. Regarding the implementation, I have the same comment as @pnowojski . Given that the partial lengths fields are not communicated across the network, I feel that this logic does not belong into buffers or `BufferBuilder`/`BufferConsumer`. Those should handle purely bytes without any knowledge of records. I would suggest to change the `buffers` queue from holding `BufferConsumer` to holding `BufferConsumerWithOffsets` or so (maybe come up with a shorter name). That way we transport the same information, but not inject it into the buffers. Some more things to watch out for in further refactoring: - The *per record methods* of *ResultPartition* and its subclasses are highly performance sensitive. They should not become virtual methods. We currently can guarantee that because there is only one class implementing these per-record methods (BufferWritingResultPartition). When we start having different implementations, we ideally have only one of them loaded in the JVM at any point, so that the de-virtualization works properly. That means design the classes/subclasses such that a type of job (batch, streaming, streaming with partial failover) would only ever rely on one of these partition types in a job. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19332) Special characters issue using Kinesis Data Analytics for Apache Flink
[
https://issues.apache.org/jira/browse/FLINK-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205532#comment-17205532
]
Dian Fu edited comment on FLINK-19332 at 10/1/20, 1:50 PM:
---
Hi [~zekuny] Personally I guess this should not be a *Blocker* issue. Could you
explain why you think this is a *Blocker* issue?
was (Author: dian.fu):
Hi [~zekuny] Personally I guess this should not be a **Blocker** issue. Could
you explain why you think this is a **Blocker** issue?
> Special characters issue using Kinesis Data Analytics for Apache Flink
> --
>
> Key: FLINK-19332
> URL: https://issues.apache.org/jira/browse/FLINK-19332
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.8.2
>Reporter: Zekun Yu
>Priority: Blocker
> Fix For: 1.8.2
>
>
> Hi there,
>
> I am encountering one special character issue while using Kinesis Data
> Analytics for Apache Flink (KDA).
>
> Our KDA is built for processing data and outputting to a Kinesis stream. We
> have a lambda function that subscribes to the Kinesis stream and reads
> records from the Kinesis stream.
> The library in the KDA I am using is
> "org.apache.flink.streaming.connectors.kinesis".
>
> Our KDA is only outputting one single record to the Kinesis sink using
> "collector.collect()" for a single key (details will be found below)
> Most times, the record received by the Lambda looks perfectly good.
> However, occasionally, when two records are sent to the kinesis sink using
> "collector.collect()" at the same time, *we noticed that those two records
> are combined somehow and there are some special characters in the record
> received by the Lambda function*.
>
>
>
>
>
>
> Below are some technical details:
>
> The KDA is not using any "TimeWindow()" but uses "keyBy()" by some keys.
> ).returns(MatchedDataForAlarm.class)
> .keyBy(MatchedDataForAlarm::getStateKey)
> .connect(ruleBroadcastStream)
> .process(new MetricProcess())
> .addSink(kinesis);
>
>
> The "MetricProcess()" extends "KeyedBroadcastProcessFunction" which overrides
> the "processElement" function. It uses collector.collect() for outputs.
> @Override
> public void processElement(MatchedDataForAlarm input, ReadOnlyContext
> ctx,Collector collector) throws Exception {
>
>
> We have our own AEMMatchedDataForAlarmSchemaSerialization which implements
> KinesisSerializationSchema. The serialization simply
> converts a "MatchedDataForAlarm" object to String using Gson and then
> converts to ByteBuffer.
>
> @Override
> public ByteBuffer serialize(MatchedDataForAlarm matchedDataForAlarm) {
> Gson gson = new Gson();
> String result = gson.toJson(matchedDataForAlarm);
> _log_.info("Alarm record sent to Kinesis stream: {}", result);
> return ByteBuffer._wrap_(result.getBytes());
> }
>
>
>
> *Here's the record shown in the Lambda logs when two records are combined
> somewhere somehow (most cases those two are received as two separate
> records):*
>
>
> 0??
> { "inAlarmState": false }
> ??
> { "inAlarmState": false}
> eEo?N9x
>
>
>
>
> I am not sure if it's a serialization issue or some default behaviors in the
> Kinesis sink library? It might be just some common mistakes that I made which
> I am not aware of.
> Could anyone help with this problem? I really appreciate it.
>
>
>
> Thanks,
> Zekun
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19332) Special characters issue using Kinesis Data Analytics for Apache Flink
[
https://issues.apache.org/jira/browse/FLINK-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205532#comment-17205532
]
Dian Fu commented on FLINK-19332:
-
Hi [~zekuny] Personally I guess this should not be a **Blocker** issue. Could
you explain why you think this is a **Blocker** issue?
> Special characters issue using Kinesis Data Analytics for Apache Flink
> --
>
> Key: FLINK-19332
> URL: https://issues.apache.org/jira/browse/FLINK-19332
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.8.2
>Reporter: Zekun Yu
>Priority: Blocker
> Fix For: 1.8.2
>
>
> Hi there,
>
> I am encountering one special character issue while using Kinesis Data
> Analytics for Apache Flink (KDA).
>
> Our KDA is built for processing data and outputting to a Kinesis stream. We
> have a lambda function that subscribes to the Kinesis stream and reads
> records from the Kinesis stream.
> The library in the KDA I am using is
> "org.apache.flink.streaming.connectors.kinesis".
>
> Our KDA is only outputting one single record to the Kinesis sink using
> "collector.collect()" for a single key (details will be found below)
> Most times, the record received by the Lambda looks perfectly good.
> However, occasionally, when two records are sent to the kinesis sink using
> "collector.collect()" at the same time, *we noticed that those two records
> are combined somehow and there are some special characters in the record
> received by the Lambda function*.
>
>
>
>
>
>
> Below are some technical details:
>
> The KDA is not using any "TimeWindow()" but uses "keyBy()" by some keys.
> ).returns(MatchedDataForAlarm.class)
> .keyBy(MatchedDataForAlarm::getStateKey)
> .connect(ruleBroadcastStream)
> .process(new MetricProcess())
> .addSink(kinesis);
>
>
> The "MetricProcess()" extends "KeyedBroadcastProcessFunction" which overrides
> the "processElement" function. It uses collector.collect() for outputs.
> @Override
> public void processElement(MatchedDataForAlarm input, ReadOnlyContext
> ctx,Collector collector) throws Exception {
>
>
> We have our own AEMMatchedDataForAlarmSchemaSerialization which implements
> KinesisSerializationSchema. The serialization simply
> converts a "MatchedDataForAlarm" object to String using Gson and then
> converts to ByteBuffer.
>
> @Override
> public ByteBuffer serialize(MatchedDataForAlarm matchedDataForAlarm) {
> Gson gson = new Gson();
> String result = gson.toJson(matchedDataForAlarm);
> _log_.info("Alarm record sent to Kinesis stream: {}", result);
> return ByteBuffer._wrap_(result.getBytes());
> }
>
>
>
> *Here's the record shown in the Lambda logs when two records are combined
> somewhere somehow (most cases those two are received as two separate
> records):*
>
>
> 0??
> { "inAlarmState": false }
> ??
> { "inAlarmState": false}
> eEo?N9x
>
>
>
>
> I am not sure if it's a serialization issue or some default behaviors in the
> Kinesis sink library? It might be just some common mistakes that I made which
> I am not aware of.
> Could anyone help with this problem? I really appreciate it.
>
>
>
> Thanks,
> Zekun
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19486) expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle
[ https://issues.apache.org/jira/browse/FLINK-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] appleyuchi updated FLINK-19486: --- Description: *I want to do an experiment of"incremental checkpoint"* my code is: [https://paste.ubuntu.com/p/DpTyQKq6Vk/] pom.xml is: http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 [http://maven.apache.org/xsd/maven-4.0.0.xsd];> 4.0.0 example datastream_api 1.0-SNAPSHOT org.apache.maven.plugins maven-compiler-plugin 3.1 1.8 1.8 org.scala-tools maven-scala-plugin 2.15.2 compile testCompile org.apache.flink flink-streaming-scala_2.11 1.11.1 org.apache.flink flink-clients_2.11 1.11.1 org.apache.flink flink-statebackend-rocksdb_2.11 1.11.2 org.apache.hadoop hadoop-client 3.3.0 org.apache.flink flink-core 1.11.1 org.apache.flink flink-cep_2.11 1.11.1 org.apache.flink flink-cep-scala_2.11 1.11.1 org.apache.flink flink-scala_2.11 1.11.1 org.projectlombok lombok 1.18.4 the error I got is: [https://paste.ubuntu.com/p/49HRYXFzR2/] *some of the above error is:* *Caused by: java.lang.IllegalStateException: Unexpected state handle type, expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle* The steps are: 1.mvn clean scala:compile compile package 2.nc -lk 3.flink run -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar Job has been submitted with JobID df6d62a43620f258155b8538ef0ddf1b 4.input the following conents in nc -lk before error error error error 5. flink run -s hdfs://Desktop:9000/tmp/flinkck/df6d62a43620f258155b8538ef0ddf1b/chk-22 -c StateWordCount datastream_api-1.0-SNAPSHOT.jar Then the above error happens. Please help,Thanks~! was: I want to do an experiment of"incremental checkpoint" my code is: [https://paste.ubuntu.com/p/DpTyQKq6Vk/] pom.xml is: http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> 4.0.0 example datastream_api 1.0-SNAPSHOT org.apache.maven.plugins maven-compiler-plugin 3.1 1.8 1.8 org.scala-tools maven-scala-plugin 2.15.2 compile testCompile org.apache.flink flink-streaming-scala_2.11 1.11.1 org.apache.flink flink-clients_2.11 1.11.1 org.apache.flink flink-statebackend-rocksdb_2.11 1.11.2 org.apache.hadoop hadoop-client 3.3.0 org.apache.flink flink-core 1.11.1 org.apache.flink flink-cep_2.11 1.11.1 org.apache.flink flink-cep-scala_2.11 1.11.1 org.apache.flink flink-scala_2.11 1.11.1 org.projectlombok lombok 1.18.4 the error I got is: [https://paste.ubuntu.com/p/49HRYXFzR2/] *some of the above error is:* *Caused by: java.lang.IllegalStateException: Unexpected state handle type, expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle* The steps are: 1.mvn clean scala:compile compile package 2.nc -lk 3.flink run -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar Job has been submitted with JobID df6d62a43620f258155b8538ef0ddf1b 4.input the following conents in nc -lk before error error error error 5. flink run -s hdfs://Desktop:9000/tmp/flinkck/df6d62a43620f258155b8538ef0ddf1b/chk-22 -c StateWordCount datastream_api-1.0-SNAPSHOT.jar Then the above error happens. Please help,Thanks~! > expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but > found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle > -- > > Key: FLINK-19486 > URL: https://issues.apache.org/jira/browse/FLINK-19486 > Project: Flink > Issue Type: Bug >Affects Versions: 1.11.1 >Reporter: appleyuchi >Priority: Blocker > Fix For: 1.11.1 > > > *I want to do an experiment of"incremental checkpoint"* > my code is: > [https://paste.ubuntu.com/p/DpTyQKq6Vk/] > > pom.xml is: > > http://maven.apache.org/POM/4.0.0; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > [http://maven.apache.org/xsd/maven-4.0.0.xsd];> > 4.0.0 > example > datastream_api > 1.0-SNAPSHOT > > > > org.apache.maven.plugins >
[jira] [Updated] (FLINK-19486) expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle
[ https://issues.apache.org/jira/browse/FLINK-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] appleyuchi updated FLINK-19486: --- Issue Type: Bug (was: Test) > expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but > found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle > -- > > Key: FLINK-19486 > URL: https://issues.apache.org/jira/browse/FLINK-19486 > Project: Flink > Issue Type: Bug >Affects Versions: 1.11.1 >Reporter: appleyuchi >Priority: Blocker > Fix For: 1.11.1 > > > I want to do an experiment of"incremental checkpoint" > my code is: > [https://paste.ubuntu.com/p/DpTyQKq6Vk/] > > pom.xml is: > > http://maven.apache.org/POM/4.0.0; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd;> > 4.0.0 > example > datastream_api > 1.0-SNAPSHOT > > > > org.apache.maven.plugins > maven-compiler-plugin > 3.1 > > 1.8 > 1.8 > > > > org.scala-tools > maven-scala-plugin > 2.15.2 > > > > compile > testCompile > > > > > > > > > > > org.apache.flink > flink-streaming-scala_2.11 > 1.11.1 > > > > > > > > > > org.apache.flink > flink-clients_2.11 > 1.11.1 > > > > org.apache.flink > flink-statebackend-rocksdb_2.11 > 1.11.2 > > > > org.apache.hadoop > hadoop-client > 3.3.0 > > > org.apache.flink > flink-core > 1.11.1 > > > > > > > > > > org.apache.flink > flink-cep_2.11 > 1.11.1 > > > org.apache.flink > flink-cep-scala_2.11 > 1.11.1 > > > org.apache.flink > flink-scala_2.11 > 1.11.1 > > > > org.projectlombok > lombok > 1.18.4 > > > > > > the error I got is: > [https://paste.ubuntu.com/p/49HRYXFzR2/] > > *some of the above error is:* > *Caused by: java.lang.IllegalStateException: Unexpected state handle type, > expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but > found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle* > > > The steps are: > 1.mvn clean scala:compile compile package > 2.nc -lk > 3.flink run -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar > Job has been submitted with JobID df6d62a43620f258155b8538ef0ddf1b > 4.input the following conents in nc -lk > before > error > error > error > error > 5. > flink run -s > hdfs://Desktop:9000/tmp/flinkck/df6d62a43620f258155b8538ef0ddf1b/chk-22 -c > StateWordCount datastream_api-1.0-SNAPSHOT.jar > Then the above error happens. > > Please help,Thanks~! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19486) expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle
appleyuchi created FLINK-19486: -- Summary: expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle Key: FLINK-19486 URL: https://issues.apache.org/jira/browse/FLINK-19486 Project: Flink Issue Type: Test Affects Versions: 1.11.1 Reporter: appleyuchi Fix For: 1.11.1 I want to do an experiment of"incremental checkpoint" my code is: [https://paste.ubuntu.com/p/DpTyQKq6Vk/] pom.xml is: http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> 4.0.0 example datastream_api 1.0-SNAPSHOT org.apache.maven.plugins maven-compiler-plugin 3.1 1.8 1.8 org.scala-tools maven-scala-plugin 2.15.2 compile testCompile org.apache.flink flink-streaming-scala_2.11 1.11.1 org.apache.flink flink-clients_2.11 1.11.1 org.apache.flink flink-statebackend-rocksdb_2.11 1.11.2 org.apache.hadoop hadoop-client 3.3.0 org.apache.flink flink-core 1.11.1 org.apache.flink flink-cep_2.11 1.11.1 org.apache.flink flink-cep-scala_2.11 1.11.1 org.apache.flink flink-scala_2.11 1.11.1 org.projectlombok lombok 1.18.4 the error I got is: [https://paste.ubuntu.com/p/49HRYXFzR2/] *some of the above error is:* *Caused by: java.lang.IllegalStateException: Unexpected state handle type, expected: class org.apache.flink.runtime.state.KeyGroupsStateHandle, but found: class org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle* The steps are: 1.mvn clean scala:compile compile package 2.nc -lk 3.flink run -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar Job has been submitted with JobID df6d62a43620f258155b8538ef0ddf1b 4.input the following conents in nc -lk before error error error error 5. flink run -s hdfs://Desktop:9000/tmp/flinkck/df6d62a43620f258155b8538ef0ddf1b/chk-22 -c StateWordCount datastream_api-1.0-SNAPSHOT.jar Then the above error happens. Please help,Thanks~! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19484) Kubernetes pyflink application test failed with "error executing jsonpath "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template: not in rang"
[
https://issues.apache.org/jira/browse/FLINK-19484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205514#comment-17205514
]
Dian Fu commented on FLINK-19484:
-
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7148=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> Kubernetes pyflink application test failed with "error executing jsonpath
> "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template:
> not in rang"
> ---
>
> Key: FLINK-19484
> URL: https://issues.apache.org/jira/browse/FLINK-19484
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7139=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-09-30T21:14:41.0570715Z 2020-09-30 21:14:41,056 INFO
> org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create
> flink application cluster flink-native-k8s-pyflink-application-1
> successfully, JobManager Web Interface: http://10.1.0.4:30141
> 2020-09-30T21:15:11.2323195Z error: timed out waiting for the condition on
> deployments/flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.2384760Z Stopping job timeout watchdog (with pid=111683)
> 2020-09-30T21:15:11.2386729Z Debugging failed Kubernetes test:
> 2020-09-30T21:15:11.2387191Z Currently existing Kubernetes resources
> 2020-09-30T21:15:11.3502610Z NAME
> TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE
> 2020-09-30T21:15:11.3506194Z service/flink-native-k8s-pyflink-application-1
> ClusterIP None 6123/TCP,6124/TCP 31s
> 2020-09-30T21:15:11.3507403Z
> service/flink-native-k8s-pyflink-application-1-rest NodePort
> 10.104.215.1 8081:30141/TCP 31s
> 2020-09-30T21:15:11.3529743Z service/kubernetes
> ClusterIP 10.96.0.1 443/TCP 18m
> 2020-09-30T21:15:11.3530391Z
> 2020-09-30T21:15:11.3531349Z NAME
> READY UP-TO-DATE AVAILABLE AGE
> 2020-09-30T21:15:11.3532200Z
> deployment.apps/flink-native-k8s-pyflink-application-1 0/1 0
> 0 31s
> 2020-09-30T21:15:11.4887105Z Name:
> flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4887491Z Namespace: default
> 2020-09-30T21:15:11.4888028Z Labels:
> app=flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4888534Ztype=flink-native-kubernetes
> 2020-09-30T21:15:11.443Z Annotations:
> 2020-09-30T21:15:11.4890558Z Selector:
> app=flink-native-k8s-pyflink-application-1,component=jobmanager,type=flink-native-kubernetes
> 2020-09-30T21:15:11.4900945Z Type: ClusterIP
> 2020-09-30T21:15:11.4901267Z IP:None
> 2020-09-30T21:15:11.4903601Z Port: jobmanager-rpc 6123/TCP
> 2020-09-30T21:15:11.4903911Z TargetPort:6123/TCP
> 2020-09-30T21:15:11.4904155Z Endpoints:
> 2020-09-30T21:15:11.4907175Z Port: blobserver 6124/TCP
> 2020-09-30T21:15:11.4907586Z TargetPort:6124/TCP
> 2020-09-30T21:15:11.4907842Z Endpoints:
> 2020-09-30T21:15:11.4908063Z Session Affinity: None
> 2020-09-30T21:15:11.4908298Z Events:
> 2020-09-30T21:15:11.4970434Z
> 2020-09-30T21:15:11.4970653Z
> 2020-09-30T21:15:11.4971488Z Name:
> flink-native-k8s-pyflink-application-1-rest
> 2020-09-30T21:15:11.4971843Z Namespace:default
> 2020-09-30T21:15:11.4972563Z Labels:
> app=flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4973297Z
> type=flink-native-kubernetes
> 2020-09-30T21:15:11.4973749Z Annotations:
> 2020-09-30T21:15:11.4974568Z Selector:
> app=flink-native-k8s-pyflink-application-1,component=jobmanager,type=flink-native-kubernetes
> 2020-09-30T21:15:11.4974969Z Type: NodePort
> 2020-09-30T21:15:11.4976265Z IP: 10.104.215.1
> 2020-09-30T21:15:11.4976620Z Port: rest 8081/TCP
> 2020-09-30T21:15:11.4976991Z TargetPort: 8081/TCP
> 2020-09-30T21:15:11.4977280Z NodePort: rest 30141/TCP
> 2020-09-30T21:15:11.4977860Z Endpoints:
> 2020-09-30T21:15:11.4978117Z Session Affinity: None
>
[jira] [Updated] (FLINK-19484) Kubernetes pyflink application test failed with "error executing jsonpath "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template: not in rang"
[
https://issues.apache.org/jira/browse/FLINK-19484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19484:
Priority: Critical (was: Major)
> Kubernetes pyflink application test failed with "error executing jsonpath
> "{range .items[*]}{.metadata.name}{\"\\n\"}{end}": Error executing template:
> not in rang"
> ---
>
> Key: FLINK-19484
> URL: https://issues.apache.org/jira/browse/FLINK-19484
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7139=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-09-30T21:14:41.0570715Z 2020-09-30 21:14:41,056 INFO
> org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create
> flink application cluster flink-native-k8s-pyflink-application-1
> successfully, JobManager Web Interface: http://10.1.0.4:30141
> 2020-09-30T21:15:11.2323195Z error: timed out waiting for the condition on
> deployments/flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.2384760Z Stopping job timeout watchdog (with pid=111683)
> 2020-09-30T21:15:11.2386729Z Debugging failed Kubernetes test:
> 2020-09-30T21:15:11.2387191Z Currently existing Kubernetes resources
> 2020-09-30T21:15:11.3502610Z NAME
> TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE
> 2020-09-30T21:15:11.3506194Z service/flink-native-k8s-pyflink-application-1
> ClusterIP None 6123/TCP,6124/TCP 31s
> 2020-09-30T21:15:11.3507403Z
> service/flink-native-k8s-pyflink-application-1-rest NodePort
> 10.104.215.1 8081:30141/TCP 31s
> 2020-09-30T21:15:11.3529743Z service/kubernetes
> ClusterIP 10.96.0.1 443/TCP 18m
> 2020-09-30T21:15:11.3530391Z
> 2020-09-30T21:15:11.3531349Z NAME
> READY UP-TO-DATE AVAILABLE AGE
> 2020-09-30T21:15:11.3532200Z
> deployment.apps/flink-native-k8s-pyflink-application-1 0/1 0
> 0 31s
> 2020-09-30T21:15:11.4887105Z Name:
> flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4887491Z Namespace: default
> 2020-09-30T21:15:11.4888028Z Labels:
> app=flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4888534Ztype=flink-native-kubernetes
> 2020-09-30T21:15:11.443Z Annotations:
> 2020-09-30T21:15:11.4890558Z Selector:
> app=flink-native-k8s-pyflink-application-1,component=jobmanager,type=flink-native-kubernetes
> 2020-09-30T21:15:11.4900945Z Type: ClusterIP
> 2020-09-30T21:15:11.4901267Z IP:None
> 2020-09-30T21:15:11.4903601Z Port: jobmanager-rpc 6123/TCP
> 2020-09-30T21:15:11.4903911Z TargetPort:6123/TCP
> 2020-09-30T21:15:11.4904155Z Endpoints:
> 2020-09-30T21:15:11.4907175Z Port: blobserver 6124/TCP
> 2020-09-30T21:15:11.4907586Z TargetPort:6124/TCP
> 2020-09-30T21:15:11.4907842Z Endpoints:
> 2020-09-30T21:15:11.4908063Z Session Affinity: None
> 2020-09-30T21:15:11.4908298Z Events:
> 2020-09-30T21:15:11.4970434Z
> 2020-09-30T21:15:11.4970653Z
> 2020-09-30T21:15:11.4971488Z Name:
> flink-native-k8s-pyflink-application-1-rest
> 2020-09-30T21:15:11.4971843Z Namespace:default
> 2020-09-30T21:15:11.4972563Z Labels:
> app=flink-native-k8s-pyflink-application-1
> 2020-09-30T21:15:11.4973297Z
> type=flink-native-kubernetes
> 2020-09-30T21:15:11.4973749Z Annotations:
> 2020-09-30T21:15:11.4974568Z Selector:
> app=flink-native-k8s-pyflink-application-1,component=jobmanager,type=flink-native-kubernetes
> 2020-09-30T21:15:11.4974969Z Type: NodePort
> 2020-09-30T21:15:11.4976265Z IP: 10.104.215.1
> 2020-09-30T21:15:11.4976620Z Port: rest 8081/TCP
> 2020-09-30T21:15:11.4976991Z TargetPort: 8081/TCP
> 2020-09-30T21:15:11.4977280Z NodePort: rest 30141/TCP
> 2020-09-30T21:15:11.4977860Z Endpoints:
> 2020-09-30T21:15:11.4978117Z Session Affinity: None
> 2020-09-30T21:15:11.4978374Z External Traffic Policy: Cluster
> 2020-09-30T21:15:11.4978719Z Events:
> 2020-09-30T21:15:11.5048919Z
>
[jira] [Closed] (FLINK-19264) Jobs with identical graph shapes cannot be run concurrently
[
https://issues.apache.org/jira/browse/FLINK-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu closed FLINK-19264.
---
Resolution: Fixed
Fixed via 6e5240f8ac24974826a07b5370908a85bd874ce1
> Jobs with identical graph shapes cannot be run concurrently
> ---
>
> Key: FLINK-19264
> URL: https://issues.apache.org/jira/browse/FLINK-19264
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Aljoscha Krettek
>Assignee: Yangze Guo
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> While working on FLINK-19123 I noticed that jobs often fail on
> {{MiniCluster}} when multiple jobs are running at the same time.
> I created a reproducer here:
> https://github.com/aljoscha/flink/tree/flink-19123-fix-test-stream-env-alternative.
> You can run {{MiniClusterConcurrencyITCase}} to see the problem in action.
> Sometimes the test will succeed, sometimes it will fail with
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.test.example.MiniClusterConcurrencyITCase.submitConcurrently(MiniClusterConcurrencyITCase.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:107)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:926)
> at akka.dispatch.OnComplete.internal(Future.scala:264)
> at
[GitHub] [flink] zhuzhurk merged pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
zhuzhurk merged pull request #13473: URL: https://github.com/apache/flink/pull/13473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
zhuzhurk commented on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-702137444 Thanks for further addressing the comments. @KarmaGYZ Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19447) HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not initialized after 200000ms"
[
https://issues.apache.org/jira/browse/FLINK-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-19447.
---
Resolution: Fixed
Fixed in master via c05027370b33c178d8b5d597b80a98857c941ba0
> HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not
> initialized after 20ms"
> ---
>
> Key: FLINK-19447
> URL: https://issues.apache.org/jira/browse/FLINK-19447
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Miklos Gergely
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=60581941-0138-53c0-39fe-86d62be5f407
> {code}
> 2020-09-28T21:52:21.2146147Z
> org.apache.flink.connector.hbase2.HBaseConnectorITCase Time elapsed: 208.382
> sec <<< ERROR!
> 2020-09-28T21:52:21.2146638Z java.io.IOException: Shutting down
> 2020-09-28T21:52:21.2147004Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
> 2020-09-28T21:52:21.2147637Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:116)
> 2020-09-28T21:52:21.2148120Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1142)
> 2020-09-28T21:52:21.2148831Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
> 2020-09-28T21:52:21.2149347Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1061)
> 2020-09-28T21:52:21.2149896Z at
> org.apache.flink.connector.hbase2.util.HBaseTestingClusterAutoStarter.setUp(HBaseTestingClusterAutoStarter.java:122)
> 2020-09-28T21:52:21.2150721Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-09-28T21:52:21.2151136Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-09-28T21:52:21.2151609Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-09-28T21:52:21.2152039Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-09-28T21:52:21.2152462Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-09-28T21:52:21.2152941Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-09-28T21:52:21.2153489Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-09-28T21:52:21.2153962Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-09-28T21:52:21.2154406Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-09-28T21:52:21.2154828Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-09-28T21:52:21.2155381Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
> 2020-09-28T21:52:21.2155864Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
> 2020-09-28T21:52:21.2156378Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-09-28T21:52:21.2156865Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> 2020-09-28T21:52:21.2157458Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> 2020-09-28T21:52:21.2157993Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> 2020-09-28T21:52:21.2158470Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> 2020-09-28T21:52:21.2158890Z Caused by: java.lang.RuntimeException: Master
> not initialized after 20ms
> 2020-09-28T21:52:21.2159350Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
> 2020-09-28T21:52:21.2159823Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
> 2020-09-28T21:52:21.2160270Z at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> 2020-09-28T21:52:21.2160800Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
> 2020-09-28T21:52:21.2161096Z ... 22 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
dianfu closed pull request #13525: URL: https://github.com/apache/flink/pull/13525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19426) Streaming File Sink end-to-end test sometimes fails with "Could not assign resource ... to current execution ..."
[
https://issues.apache.org/jira/browse/FLINK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205509#comment-17205509
]
Dian Fu commented on FLINK-19426:
-
[~azagrebin] Thanks a lot for the investigation!
> Streaming File Sink end-to-end test sometimes fails with "Could not assign
> resource ... to current execution ..."
> -
>
> Key: FLINK-19426
> URL: https://issues.apache.org/jira/browse/FLINK-19426
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6983=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729
> {code}
> 2020-09-26T22:16:26.9856525Z
> org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException:
> Connection for partition
> 619775973ed0f282e20f9d55d13913ab#0@bc764cd8ddf7a0cff126f51c16239658_0_1 not
> reachable.
> 2020-09-26T22:16:26.9857848Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:159)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9859168Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:336)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9860449Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:308)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9861677Z at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:95)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9862861Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.requestPartitions(StreamTask.java:542)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9864018Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.readRecoveredChannelState(StreamTask.java:507)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9865284Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:498)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9866415Z at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9867500Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:492)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9868514Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:550)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9869450Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870339Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870869Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-09-26T22:16:26.9872060Z Caused by: java.io.IOException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has been lost.
> 2020-09-26T22:16:26.9873511Z at
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:85)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9874788Z at
> org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:67)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9876084Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:156)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9876567Z ... 12 more
> 2020-09-26T22:16:26.9877477Z Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has
[jira] [Updated] (FLINK-19485) Decouple runtime operator implementation from DataStream operations
[
https://issues.apache.org/jira/browse/FLINK-19485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-19485:
-
Summary: Decouple runtime operator implementation from DataStream
operations (was: Consider runtime-mode when creating the StreamGraph)
> Decouple runtime operator implementation from DataStream operations
>
>
> Key: FLINK-19485
> URL: https://issues.apache.org/jira/browse/FLINK-19485
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
>
> Although DataStream is going to be the unified API for Batch and Streaming
> applications, some operations, e.g. Sinks, may need to have different runtime
> implementations depending on if they are intended to run on bounded or
> unbounded data. This is not necessarily only for optimisations but also for
> the exposed semantics, i.e. correctness.
> So far, DataStream had a 1-to-1 mapping between an API call and an operator.
> In a sense, the DataStream API was an "explicit" API. With this addition, we
> will decouple the API calls from the actual runtime implementations and thus
> allow different operations to have more than one runtime implementations,
> depending (for now) on the {{execution.runtime-mode}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] alpinegizmo commented on a change in pull request #13510: [FLINK-13095][state-processor-api] Introduce window bootstrap writer for writing window operator state
alpinegizmo commented on a change in pull request #13510:
URL: https://github.com/apache/flink/pull/13510#discussion_r498222828
##
File path: docs/dev/libs/state_processor_api.md
##
@@ -742,6 +740,56 @@ If a processing time timer is set but the state is not
restored until after that
Attention If your bootstrap function
creates timers, the state can only be restored using one of the [process]({{
site.baseurl }}/dev/stream/operators/process_function.html) type functions.
+### Window State
+
+The state processor api supports writing state for the [window operator]({{
site.baseurl }}/dev/stream/operators/windows.html).
+When writing window state, users specify the operator id, window assigner,
evictor, optional trigger, and aggregation type.
+It is important the configurations on the bootstrap transformation match the
configurations on the DataStream window.
+
+
+
+{% highlight java %}
+public class Account {
+public int id;
+
+public double amount;
+
+public long timestamp;
+}
+
+ExecutionEnvironment bEnv = ExecutionEnvironment.getExecutionEnvironment();
+
+DataSet accountDataSet = bEnv.fromCollection(accounts);
+
+BootstrapTransformation transformation = OperatorTransformation
+.bootstrapWith(accountDataSet)
+// When using event time windows, its important
Review comment:
```suggestion
// When using event time windows, it is important
```
##
File path: docs/dev/libs/state_processor_api.md
##
@@ -742,6 +740,56 @@ If a processing time timer is set but the state is not
restored until after that
Attention If your bootstrap function
creates timers, the state can only be restored using one of the [process]({{
site.baseurl }}/dev/stream/operators/process_function.html) type functions.
+### Window State
+
+The state processor api supports writing state for the [window operator]({{
site.baseurl }}/dev/stream/operators/windows.html).
+When writing window state, users specify the operator id, window assigner,
evictor, optional trigger, and aggregation type.
+It is important the configurations on the bootstrap transformation match the
configurations on the DataStream window.
+
+
+
+{% highlight java %}
+public class Account {
+public int id;
+
+public double amount;
+
+public long timestamp;
+}
+
+ExecutionEnvironment bEnv = ExecutionEnvironment.getExecutionEnvironment();
+
+DataSet accountDataSet = bEnv.fromCollection(accounts);
+
+BootstrapTransformation transformation = OperatorTransformation
+.bootstrapWith(accountDataSet)
+// When using event time windows, its important
+// to assign timestamps to each record.
+.assignTimestamps(account -> account.timestamp)
+.keyBy(acc -> acc.id)
+.window(TumblingEventTimeWindows.of)
+.reduce((left, right) -> left + right);
+{% endhighlight %}
+
+
+{% highlight scala %}
+case class Account(id: Int, amount: Double, timestamp: Long)
+
+val bEnv = ExecutionEnvironment.getExecutionEnvironment();
+val accountDataSet = bEnv.fromCollection(accounts);
+
+val transformation = OperatorTransformation
+.bootstrapWith(accountDataSet)
+// When using event time windows, its important
+// to assign timestamps to each record.
+.assignTimestamps(account -> account.timestamp)
+.keyBy(acc -> acc.id)
+.window(TumblingEventTimeWindows.of)
+.reduce((left, right) -> left + right)
Review comment:
This doesn't look like code that would actually compile.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
dawidwys commented on a change in pull request #12056:
URL: https://github.com/apache/flink/pull/12056#discussion_r498214070
##
File path:
flink-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQDeserializationSchema.java
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.rabbitmq;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.util.Collector;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Envelope;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Interface for the set of methods required to parse an RMQ delivery.
+ * @param The output type of the {@link RMQSource}
+ */
+public interface RMQDeserializationSchema extends Serializable,
ResultTypeQueryable {
+
+ /**
+* Initialization method for the schema. It is called before the actual
working methods
+* {@link #deserialize} and thus suitable for one time setup work.
+*
+* The provided {@link DeserializationSchema.InitializationContext}
can be used to access additional features such as e.g.
+* registering user metrics.
+*
+* @param context Contextual information that can be used during
initialization.
+*/
+ public void open(DeserializationSchema.InitializationContext context)
throws Exception;
+
+
+ /**
+* This method takes all the RabbitMQ delivery information supplied by
the client extract the data and pass it to the
+* collector.
+* NOTICE: The implementation of this method MUST call {@link
RMQCollector#setMessageIdentifiers(String, long)} with
+* the correlation ID of the message if checkpointing and
UseCorrelationID (in the RMQSource constructor) were enabled
+* the {@link RMQSource}.
+* @param envelope
+* @param properties
+* @param body
+* @throws IOException
+*/
Review comment:
```suggestion
* This method takes all the RabbitMQ delivery information supplied by
the client extract the data and pass it to the
* collector.
*
* NOTICE: The implementation of this method MUST call {@link
RMQCollector#setMessageIdentifiers(String, long)} with
* the correlation ID of the message if checkpointing and
UseCorrelationID (in the RMQSource constructor) were enabled
* the {@link RMQSource}.
*/
```
##
File path:
flink-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQDeserializationSchema.java
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.rabbitmq;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.util.Collector;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Envelope;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Interface for the set of methods required to parse
[GitHub] [flink] juha-mynttinen-king commented on pull request #13393: [FLINK-19238] [RocksDB] Sanity check for arena block size
juha-mynttinen-king commented on pull request #13393: URL: https://github.com/apache/flink/pull/13393#issuecomment-702109330 Hey. I'm going to lose access to the GitHub account soon, and not be able to modify the branch anymore. If changes are needed before merge, I guess I could create a new PR using another GH account. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * ad55b5a30711435f6caf0d52ba1cdd702af96fda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7151) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
flinkbot edited a comment on pull request #13525: URL: https://github.com/apache/flink/pull/13525#issuecomment-701444731 ## CI report: * df45481e9c3a07bcbfac6e070bc16a38fe83f3ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * 4824a2e5258649246c5372200dd44fb61e20d012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7143) * ad55b5a30711435f6caf0d52ba1cdd702af96fda UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18044) Add the subtask index information to the SourceReaderContext.
[ https://issues.apache.org/jira/browse/FLINK-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205439#comment-17205439 ] Stephan Ewen commented on FLINK-18044: -- I am not strictly opposed to exposing the Subtask Id. It is more that this encourages users to write code that is not oblivious to parallelism, as code ideally should be. Code that makes explicit assumptions about the current parallelism and the current subtask index would break when we try to introduce mode dynamic models (where some subtasks can split/scale out without affecting other tasks). > Add the subtask index information to the SourceReaderContext. > - > > Key: FLINK-18044 > URL: https://issues.apache.org/jira/browse/FLINK-18044 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Jiangjie Qin >Priority: Major > Labels: pull-request-available > > It is useful for the `SourceReader` to retrieve its subtask id. For example, > Kafka readers can create a consumer with proper client id. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dawidwys commented on a change in pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
dawidwys commented on a change in pull request #13521:
URL: https://github.com/apache/flink/pull/13521#discussion_r498180993
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/CircularQueues.java
##
@@ -75,6 +75,9 @@ public void send(StageRunner.SortStage stage,
CircularElement element) {
@Override
public void sendResult(MutableObjectIterator result) {
+ if (iteratorFuture.isDone()) {
Review comment:
Actually these changes are unnecessary, I added them for helping me with
debugging and forgot to remove them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
dawidwys commented on a change in pull request #13521:
URL: https://github.com/apache/flink/pull/13521#discussion_r498177852
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java
##
@@ -55,4 +56,14 @@
"throughput")
)
.build());
+
+ @Documentation.ExcludeFromDocumentation("The option is considered
internal and should rather not be used" +
Review comment:
It won't work in `StreamNode`. Only `StreamConfig` is shipped to
`StreamTask`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-web] zentol commented on pull request #384: Improved Docs
zentol commented on pull request #384: URL: https://github.com/apache/flink-web/pull/384#issuecomment-702075483 I do not think this really enhances the documentation. If you wish to make a contribution to Flink, please read the [contribution guide](https://flink.apache.org/contributing/how-to-contribute.html). I'm sure you can find something else to work on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol closed pull request #384: Improved Docs
zentol closed pull request #384: URL: https://github.com/apache/flink-web/pull/384 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19441) Performance regression on 24.09.2020
[ https://issues.apache.org/jira/browse/FLINK-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205424#comment-17205424 ] Stephan Ewen commented on FLINK-19441: -- [~AHeise] That sounds like a plan. I'll start digging into the code, see if I can spot some virtual per-record calls. > Performance regression on 24.09.2020 > > > Key: FLINK-19441 > URL: https://issues.apache.org/jira/browse/FLINK-19441 > Project: Flink > Issue Type: Bug >Reporter: Arvid Heise >Priority: Blocker > > A couple of benchmarks are showing a small performance regression on > 24.09.2020: > http://codespeed.dak8s.net:8000/timeline/?ben=globalWindow=2 > http://codespeed.dak8s.net:8000/timeline/?ben=tupleKeyBy=2 (?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] StephanEwen commented on pull request #13512: [FLINK-19457][core] Add a number sequence generating source for the New Source API.
StephanEwen commented on pull request #13512: URL: https://github.com/apache/flink/pull/13512#issuecomment-702065085 For both topics, this PR here stays close to what the previous state of sources was, in an effort to port the previous state to the new API. We can open the discussions about having different package structures in the future, but do we need to block this PR on that? The package structure in flink-core has never separated API and implementation, and the `api` name was not chosen with very much consideration. IIRC mainly to separate SDK from runtime parts. (Arguable `.sdk.` would have been the correct name, but were were all younger and less experienced when we created these parts.) From the user's perspective I think it is better to stay consistent within one style then have different packages follow different styles. That adds more confusion and makes it harder to navigate. Similar for the "deep integration", I feel this should not block this PR (foremost because this PR does not touch anything there). That is a different discussion that we should either apply everywhere or not do it. Doing this now here differently than in all other places will add confusion for users. Another side of the problem is that many of these methods `env.generateSequence(...)` are public API that we should not break. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19479) Allow explicitly configuring time behaviour on KeyedStream.intervalJoin()
[
https://issues.apache.org/jira/browse/FLINK-19479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205423#comment-17205423
]
Nicholas Jiang commented on FLINK-19479:
[~aljoscha], IMO, I could also work for this issue.:D
> Allow explicitly configuring time behaviour on KeyedStream.intervalJoin()
> -
>
> Key: FLINK-19479
> URL: https://issues.apache.org/jira/browse/FLINK-19479
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
>
> With the deprecation of
> {{StreamExecutionEnvironment.setStreamTimeCharacteristic()}} in FLINK-19319
> we need a way of explicitly configuring the time behaviour of these join
> operations. Currently, all join operations use the characteristic to
> configure themselves.
> Alternatively, we might consider removing/deprecating these join operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19249) Detect broken connections in case TCP Timeout takes too long.
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-19249:
-
Summary: Detect broken connections in case TCP Timeout takes too long.
(was: Job would wait sometime(~10 min) before failover if some connection
broken)
> Detect broken connections in case TCP Timeout takes too long.
> -
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Major
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-19249:
-
Priority: Major (was: Critical)
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Major
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205414#comment-17205414
]
Stephan Ewen commented on FLINK-19249:
--
Digging into this a bit more, I think your are right. This is a corner case
caused by problematic network environments.
Let's label this as an improvement where Flink could try to do something on the
application layer to detect network issues when the kernel takes too long.
But we still miss a good proposal how to do that.
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Critical
> Fix For: 1.12.0
>
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-19249:
-
Fix Version/s: (was: 1.12.0)
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Critical
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
flinkbot edited a comment on pull request #12056: URL: https://github.com/apache/flink/pull/12056#issuecomment-626167900 ## CI report: * c42ef1f62396948ece865a3b8628bea229d5ffb9 UNKNOWN * ad971fd47ab3ac9e813afae9713c75986646 UNKNOWN * 4d2d57b8c2f6810c03d284fdd5807b81a2b554d7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7120) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7150) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7142) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-playgrounds] alpinegizmo commented on a change in pull request #16: [FLINK-19145][walkthroughs] Add PyFlink-walkthrough to Flink playground.
alpinegizmo commented on a change in pull request #16:
URL: https://github.com/apache/flink-playgrounds/pull/16#discussion_r498114079
##
File path: pyflink-walkthrough/README.md
##
@@ -0,0 +1,140 @@
+# pyflink-walkthrough
+
+## Background
+
+In this playground, you will learn how to build and run an end-to-end PyFlink
pipeline for data analytics, covering the following steps:
+
+* Reading data from a Kafka source;
+* Creating data using a
[UDF](https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/python/table-api-users-guide/udfs/python_udfs.html);
+* Performing a simple aggregation over the source data;
+* Writing the results to Elasticsearch and visualizing them in Kibana.
+
+The environment is based on Docker Compose, so the only requirement is that
you have [Docker](https://docs.docker.com/get-docker/)
+installed in your machine.
+
+### Kafka
+You will be using Kafka to store sample input data about payment transactions.
A simple data generator
[generate_source_data.py](generator/generate_source_data.py) is provided to
+continuously write new records to the `payment_msg` Kafka topic. Each record
is structured as follows:
+
+`{"createTime": "2020-08-12 06:29:02", "orderId": 1597213797, "payAmount":
28306.44976403719, "payPlatform": 0, "provinceId": 4}`
+
+* `createTime`: The creation time of the transaction.
+* `orderId`: The id of the current transaction.
+* `payAmount`: The amount being paid with this transaction.
+* `payPlatform`: The platform used to create this payment: pc or mobile.
+* `provinceId`: The id of the province for the user.
+
+You can use the following command to read data from the Kafka topic and check
whether it's generated correctly:
+```shell script
+$ docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server
kafka:9092 --topic payment_msg
+{"createTime":"2020-07-27
09:25:32.77","orderId":1595841867217,"payAmount":7732.44,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.231","orderId":1595841867218,"payAmount":75774.05,"payPlatform":0,"provinceId":3}
+{"createTime":"2020-07-27
09:25:33.72","orderId":1595841867219,"payAmount":65908.55,"payPlatform":0,"provinceId":0}
+{"createTime":"2020-07-27
09:25:34.216","orderId":1595841867220,"payAmount":15341.11,"payPlatform":0,"provinceId":1}
+{"createTime":"2020-07-27
09:25:34.698","orderId":1595841867221,"payAmount":37504.42,"payPlatform":0,"provinceId":0}
+```
+You can also create a new topic by executing the following command:
+```shell script
+$ docker-compose exec kafka kafka-topics.sh --bootstrap-server kafka:9092
--create --topic --partitions 8 --replication-factor 1
+```
+
+### PyFlink
+
+The transaction data will be processed with PyFlink using the Python script
[payment_msg_processing.py](payment_msg_proccessing.py).
+This script will first map the `provinceId` in the input records to its
corresponding province name using a Python UDF,
+and then compute the sum of the transaction amounts for each province.
+
+### ElasticSearch
+
+ElasticSearch is used to store the results and to provide an efficient query
service.
+
+### Kibana
+
+Kibana is an open source data visualization dashboard for ElasticSearch. You
will use it to visualize
+the results of your PyFlink pipeline.
+
+## Setup
+
+As mentioned, the environment for this walkthrough is based on Docker Compose;
It uses a custom image
+to spin up Flink (JobManager+TaskManager), Kafka+Zookeeper, the data
generator, and Elasticsearch+Kibana containers.
+
+You can find the [docker-compose.yaml](docker-compose.yml) file of the
pyflink-walkthrough in the `pyflink-walkthrough` root directory.
+
+### Building the Docker image
+
+First, build the Docker image by running:
+
+```bash
+docker-compose build
+```
+
+### Starting the Playground
+
+Once the Docker image build is complete, run the following command to start
the playground:
+
+```bash
+docker-compose up -d
+```
+
+One way of checking if the playground was successfully started is to access
some of the services that are exposed:
+
+1. visiting Flink Web UI [http://localhost:8081](http://localhost:8081).
+2. visiting Elasticsearch [http://localhost:9200](http://localhost:9200).
+3. visiting Kibana [http://localhost:5601](http://localhost:5601).
+
+**Note:** you may need to wait around 1 minute before all the services come up.
+
+### Stopping the Playground
+
+To stop the playground, run the following command:
+
+```bash
+docker-compose down
+```
+
+
+## Running the PyFlink job
+
+1. Submit the PyFlink job.
+```shell script
+$ docker-compose exec jobmanager ./bin/flink run -py
/opt/pyflink-walkthrough/payment_msg_proccessing.py -d
+```
+Navigate to the [Flink Web UI](http://localhost:8081) after the job is
submitted successfully. There should be a job in the running job list.
+Click the job to get more details. You should see that the `StreamGraph` of
the `payment_msg_proccessing` consists of two nodes, each with a parallelism of
1.
[GitHub] [flink] aljoscha commented on a change in pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
aljoscha commented on a change in pull request #13521:
URL: https://github.com/apache/flink/pull/13521#discussion_r498141996
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/KeyAndValueSerializer.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+
+/**
+ * A serializer used in {@link SortingDataInput} for serializing elements
alongside their key and
+ * timestamp. It serializes the record in a format known by the {@link
FixedLengthByteKeyComparator}
+ * and {@link VariableLengthByteKeyComparator}.
+ *
+ * If the key is of known constant length, the length is not serialized
with the data.
+ * Therefore the serialized data is as follows:
+ *
+ *
+ * [key-length] | key | timestamp | record
+ *
+ */
+final class KeyAndValueSerializer extends TypeSerializer>> {
+ private final TypeSerializer valueSerializer;
+ private final int serializedKeyLength;
Review comment:
Maybe add a comment explaining that this is either a fixed key length or
negative if it is variable.
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/SortingDataInput.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.AlgorithmOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter;
+import org.apache.flink.runtime.io.AvailabilityProvider;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.sort.ExternalSorter;
+import org.apache.flink.runtime.operators.sort.PushSorter;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.io.StreamMultipleInputProcessor;
+import org.apache.flink.streaming.runtime.io.StreamTaskInput;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.streamstatus.StreamStatus;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.MutableObjectIterator;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A
[GitHub] [flink] flinkbot edited a comment on pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
flinkbot edited a comment on pull request #12056: URL: https://github.com/apache/flink/pull/12056#issuecomment-626167900 ## CI report: * c42ef1f62396948ece865a3b8628bea229d5ffb9 UNKNOWN * ad971fd47ab3ac9e813afae9713c75986646 UNKNOWN * 4d2d57b8c2f6810c03d284fdd5807b81a2b554d7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7142) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7120) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7150) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] senegalo commented on pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
senegalo commented on pull request #12056: URL: https://github.com/apache/flink/pull/12056#issuecomment-702037986 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
