[GitHub] [flink] zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce JobManagerRunner interface
zhuzhurk commented on a change in pull request #9788: [FLINK-14259] Introduce
JobManagerRunner interface
URL: https://github.com/apache/flink/pull/9788#discussion_r329311200
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##
@@ -629,7 +630,7 @@ public void testJobSubmissionErrorAfterJobRecovery()
throws Exception {
}
/**
-* Tests that a blocking {@link JobManagerRunner} creation, e.g. due to
blocking FileSystem access,
+* Tests that a blocking {@link JobManagerRunnerImpl} creation, e.g.
due to blocking FileSystem access,
Review comment:
`JobManagerRunner` maybe?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-13056) Optimize region failover performance on calculating vertices to restart
[ https://issues.apache.org/jira/browse/FLINK-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-13056: Fix Version/s: 1.10.0 > Optimize region failover performance on calculating vertices to restart > --- > > Key: FLINK-13056 > URL: https://issues.apache.org/jira/browse/FLINK-13056 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently some region boundary structures are calculated each time of a > region failover. This calculation can be heavy as its complexity goes up with > execution edge count. > We tested it in a sample case with 8000 vertices and 16,000,000 edges. It > takes ~2.0s to calculate vertices to restart. > (more details in > [https://docs.google.com/document/d/197Ou-01h2obvxq8viKqg4FnOnsykOEKxk3r5WrVBPuA/edit?usp=sharing)] > That's why we'd propose to cache the region boundary structures to improve > the region failover performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14163:
Affects Version/s: 1.9.0
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> But the task deployment process (in {{Execution#deploy())}} will create
> {{ResultPartitionDeploymentDescriptor}} directly from
> {{Execution#producedPartitions}} without checking whether it's assigned.
> This may lead to a task deployed without its result partitions. And
> eventually cause the job to hang.
> It is not problematic at the moment when using Flink default shuffle master
> {{NettyShuffleMaster}} since it returns a completed future on registration.
> However, if the behavior is changed or if users are using a customized
> {{ShuffleMaster}}, it may cause problems.
> Besides that, {{Execution#producedPartitions}} is also used for
> * generating downstream task input descriptor
> * retrieve {{ResultPartitionID}} for partition releasing
> To avoid issues to happen, we may need to change all the usages of
> {{Execution#producedPartitions}} to a callback way, e.g. change
> {{Execution#producedPartitions}} from {{Map ResultPartitionDeploymentDescriptor>}} to
> {{CompletableFuture ResultPartitionDeploymentDescriptor>>}} and adjust all its usages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13056) Optimize region failover performance on calculating vertices to restart
[ https://issues.apache.org/jira/browse/FLINK-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-13056: Affects Version/s: 1.10.0 > Optimize region failover performance on calculating vertices to restart > --- > > Key: FLINK-13056 > URL: https://issues.apache.org/jira/browse/FLINK-13056 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently some region boundary structures are calculated each time of a > region failover. This calculation can be heavy as its complexity goes up with > execution edge count. > We tested it in a sample case with 8000 vertices and 16,000,000 edges. It > takes ~2.0s to calculate vertices to restart. > (more details in > [https://docs.google.com/document/d/197Ou-01h2obvxq8viKqg4FnOnsykOEKxk3r5WrVBPuA/edit?usp=sharing)] > That's why we'd propose to cache the region boundary structures to improve > the region failover performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-13656) Upgrade Calcite dependency to 1.21
[ https://issues.apache.org/jira/browse/FLINK-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-13656: -- Assignee: Danny Chen > Upgrade Calcite dependency to 1.21 > -- > > Key: FLINK-13656 > URL: https://issues.apache.org/jira/browse/FLINK-13656 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: Jark Wu >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Umbrella issue for all tasks related to the next Calcite upgrade to 1.21.x > release > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > - Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > - We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > - CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-13656) Upgrade Calcite dependency to 1.21
[ https://issues.apache.org/jira/browse/FLINK-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-13656. -- Fix Version/s: 1.10.0 Resolution: Fixed > Upgrade Calcite dependency to 1.21 > -- > > Key: FLINK-13656 > URL: https://issues.apache.org/jira/browse/FLINK-13656 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: Jark Wu >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Umbrella issue for all tasks related to the next Calcite upgrade to 1.21.x > release > Calcite 1.21 has been released recently, we need to upgrade to version 1.21 > for these reasons: > - Previously we have made some temp code to support full data types in sql > parser, since CALCITE-3213 has been resolved, we can do some refactoring for > these codes; > - We also fixed some important bug for Join which bring in from Calcite 1.20 > join like expression promotion, such as CALCITE-3170, CALCITE-3171. > - CALCITE-2302 has been resolved, there is possibility we support implicit > type coercion for Flink now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KurtYoung closed pull request #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0
KurtYoung closed pull request #9712: [FLINK-13656] [sql-parser][table-planner][table-planner-blink] Bump Calcite dependency to 1.21.0 URL: https://github.com/apache/flink/pull/9712 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129545862) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129545862) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout
flinkbot edited a comment on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout URL: https://github.com/apache/flink/pull/9794#issuecomment-536174033 ## CI report: * f9fbec1b59a8519c83bfb3dff4f6f1d9a785350b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129544195) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment.
flinkbot edited a comment on issue #9781: [FLINK-14098] Support multiple statements for TableEnvironment. URL: https://github.com/apache/flink/pull/9781#issuecomment-535813967 ## CI report: * ed0e04d1cd481b096d68d0c9bf36a997dcca2550 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129400381) * ec8fccdeb53c5e2e0f089708753be33839706e2a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129441309) * ad27f8e71d91534867d093f00cde3d5b81534282 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129526289) * 5247e6b03266d0d25d42a185e0a447643116d96f : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129544930) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout
flinkbot edited a comment on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout URL: https://github.com/apache/flink/pull/9794#issuecomment-536174033 ## CI report: * f9fbec1b59a8519c83bfb3dff4f6f1d9a785350b : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129544195) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
flinkbot edited a comment on issue #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build URL: https://github.com/apache/flink/pull/9783#issuecomment-535848177 ## CI report: * 5d9e3113e10bbf0220b4f8e080ecd1cddd9c9059 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129411044) * 55c766190d205798534f6d09170e66463c832082 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9783: [FLINK-14040][travis] Enable MiniCluster tests based on schedulerNG in Flink cron build
zhuzhurk commented on a change in pull request #9783: [FLINK-14040][travis]
Enable MiniCluster tests based on schedulerNG in Flink cron build
URL: https://github.com/apache/flink/pull/9783#discussion_r329306187
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobExecutionITCase.java
##
@@ -37,6 +39,7 @@
/**
* Integration tests for job scheduling.
*/
+@Category(AlsoRunWithSchedulerNG.class)
public class JobExecutionITCase extends TestLogger {
Review comment:
This hotfix enabled NG scheduler testing for JobExecutionITCase which can
pass at the moment.
It helps to verify the changes of FLINK-14040.
In this way we can have FLINK-14040 merged first.
FLINK-14040 would be helpful when we are to annotate (and maybe need to fix)
more tests. We can tell if the change works by checking both the PR test
result(so that it works for legacy) and the cron job result(so that it works
for NG).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout
flinkbot commented on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout URL: https://github.com/apache/flink/pull/9794#issuecomment-536174033 ## CI report: * f9fbec1b59a8519c83bfb3dff4f6f1d9a785350b : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129539518) * af2f1aa3e11f3c27876dd5e765cca90ae2431a1a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129542555) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout
flinkbot commented on issue #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout URL: https://github.com/apache/flink/pull/9794#issuecomment-536172589 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f9fbec1b59a8519c83bfb3dff4f6f1d9a785350b (Sat Sep 28 10:10:30 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk opened a new pull request #9794: Flink 14247 no resource available exception for allocation timeout
zhuzhurk opened a new pull request #9794: Flink 14247 no resource available exception for allocation timeout URL: https://github.com/apache/flink/pull/9794 ## What is the purpose of the change This PR is to use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout. This makes the error easier to understand for users. ## Brief change log - *Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout before triggering error handling* - *other hotfix changes see each commit* ## Verifying this change - *Added unit tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129539518) * af2f1aa3e11f3c27876dd5e765cca90ae2431a1a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129542555) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129539518) * af2f1aa3e11f3c27876dd5e765cca90ae2431a1a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-14267) Introduce RowCsvOutputFormat to new CSV module
Jingsong Lee created FLINK-14267: Summary: Introduce RowCsvOutputFormat to new CSV module Key: FLINK-14267 URL: https://issues.apache.org/jira/browse/FLINK-14267 Project: Flink Issue Type: Sub-task Components: Connectors / FileSystem Reporter: Jingsong Lee Fix For: 1.10.0 Now, we have an old CSV, but that is not standard CSV support. we should support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14266) Introduce RowCsvInputFormat to new CSV module
Jingsong Lee created FLINK-14266: Summary: Introduce RowCsvInputFormat to new CSV module Key: FLINK-14266 URL: https://issues.apache.org/jira/browse/FLINK-14266 Project: Flink Issue Type: Sub-task Components: Connectors / FileSystem Reporter: Jingsong Lee Fix For: 1.10.0 Now, we have an old CSV, but that is not standard CSV support. we should support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14265) Don't use ContinuousFileReaderOperator to support multiple paths
Jingsong Lee created FLINK-14265: Summary: Don't use ContinuousFileReaderOperator to support multiple paths Key: FLINK-14265 URL: https://issues.apache.org/jira/browse/FLINK-14265 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Jingsong Lee Fix For: 1.10.0 Now, blink planner use ContinuousFileReaderOperator to support InputFormat, but ContinuousFileReaderOperator not support multiple paths. If read partitioned source, after partition pruning, we need let InputFormat to read multiple partitions which are multiple paths. We can use InputFormatSourceFunction directly to support InputFormat. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14108) Support for Confluent Kafka schema registry for Avro serialisation
[ https://issues.apache.org/jira/browse/FLINK-14108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939901#comment-16939901 ] Lasse Nedergaard commented on FLINK-14108: --- Sure. I'm on the road next week and the week after to Flink Forward. I will try to find the time for it. > Support for Confluent Kafka schema registry for Avro serialisation > --- > > Key: FLINK-14108 > URL: https://issues.apache.org/jira/browse/FLINK-14108 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.10.0 >Reporter: Lasse Nedergaard >Assignee: Lasse Nedergaard >Priority: Minor > > The current implementation in flink-avro-confluent-registry support > deserialization with schema lookup in Confluent Kafka schema registry. > I would like support for serialization as well, following the same structure > as deserialization. With the feature it would be possible to use Confluent > schema registry in Sink writing Avro to Kafka and at the same time register > the schema used. > The test in TestAvroConsumerConfluent need to be updated together with the > comment as it indicate it use Confluent schema registry for write, but the > example code use SimpleStringSchema. > We have a running version, that we would like to give back to the community. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129539518) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129539518) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14250) Introduce separate "bin/flink runpy" and remove python support from "bin/flink run"
[
https://issues.apache.org/jira/browse/FLINK-14250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939887#comment-16939887
]
Kostas Kloudas commented on FLINK-14250:
I am working on refactoring a bit the {{CustomCommandLine}}. I will open a
separate Jira issue later for that. This Jira focuses on making Python CLI
option parsing and validation self-contained.
> Introduce separate "bin/flink runpy" and remove python support from
> "bin/flink run"
> ---
>
> Key: FLINK-14250
> URL: https://issues.apache.org/jira/browse/FLINK-14250
> Project: Flink
> Issue Type: Sub-task
> Components: Client / Job Submission
>Reporter: Aljoscha Krettek
>Priority: Major
>
> Currently, "bin/flink run" supports both Java and Python programs and there
> is quite some complexity in command line parsing and validation and the code
> is spread across different classes.
> I think if we had a separate "bin/flink runpy" then we could simplify the
> parsing quite a bit and the usage help of each command would only show those
> options that are relevant for the given use case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) * cb4a93fcc87066aa52af6118f5c1eff94c3059cc : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/129537802) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods
flinkbot edited a comment on issue #9793: [FLINK-14130][client] Remove ClusterClient.run() methods URL: https://github.com/apache/flink/pull/9793#issuecomment-536153382 ## CI report: * ea5b4330b5c9fb6eff889eb1c55c3f9a7bcfb6a9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129535988) * ce83637c6e309e99f09e08f8e013dbd77407124a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
