[jira] [Commented] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254980#comment-17254980 ] hehuiyuan commented on FLINK-20771: --- Hi [~ZhuShang] , fix , it is good. > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > the current bucket is active , so the notifyBucketInactive is not executed. > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > > !image-2020-12-25-18-19-53-746.png|width=697,height=56! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20427) Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss
[
https://issues.apache.org/jira/browse/FLINK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254988#comment-17254988
]
Liwen Liu commented on FLINK-20427:
---
Hi，[~trohrmann]
Could you assign this ticket to me ?
Thanks~
> Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to
> data loss
> ---
>
> Key: FLINK-20427
> URL: https://issues.apache.org/jira/browse/FLINK-20427
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Checkpointing
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.13.0
>
>
> The {{CheckpointConfig.setPreferCheckpointForRecovery}} allows to configure
> whether Flink prefers checkpoints for recovery if the
> {{CompletedCheckpointStore}} contains savepoints and checkpoints. This is
> problematic because due to this feature, Flink might prefer older checkpoints
> over newer savepoints for recovery. Since some components expect that the
> always the latest checkpoint/savepoint is used (e.g. the
> {{SourceCoordinator}}), it breaks assumptions and can lead to
> {{SourceSplits}} which are not read. This effectively means that the system
> loses data. Similarly, this behaviour can cause that exactly once sinks might
> output results multiple times which violates the processing guarantees.
> Hence, I believe that we should remove this setting because it changes
> Flink's behaviour in some very significant way potentially w/o the user
> noticing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-training] sv3ndk opened a new pull request #15: add support for Flink 1.12
sv3ndk opened a new pull request #15: URL: https://github.com/apache/flink-training/pull/15 hi, While reading Flink documentation, I noticed that [learn-flink/etl.html](https://ci.apache.org/projects/flink/flink-docs-release-1.12/learn-flink/etl.html) has a link at the bottom of the page to https://github.com/apache/flink-training/tree/release-1.12/rides-and-fares, which does not exists yet. I tried to bump up the Flink version here to 1.12 and everything seems to work (I didn't do all the exercises though). Based on the gradle warning I saw during `./gradlew test shadowJar`, I also removed the following statements from the scala exercises: ``` env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) ``` since it's my understanding that this method is now deprecated and event-time characteristics is the default anyway (cf https://github.com/apache/flink/blob/release-1.12/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/StreamExecutionEnvironment.scala#L411 ). Hope this helps, thanks for making this training freely available :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14465: [FLINK-20594][runtime] Remove DefaultExecutionSlotAllocator and its Factory class.
flinkbot edited a comment on pull request #14465: URL: https://github.com/apache/flink/pull/14465#issuecomment-749579707 ## CI report: * 2147c4b63bfdf589e2c5ab167b0c41cacc7a514c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11260) * 23d374243c9726e72ec4b1e1e87739739cded4b9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14465: [FLINK-20594][runtime] Remove DefaultExecutionSlotAllocator and its Factory class.
flinkbot edited a comment on pull request #14465: URL: https://github.com/apache/flink/pull/14465#issuecomment-749579707 ## CI report: * 2147c4b63bfdf589e2c5ab167b0c41cacc7a514c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11260) * 23d374243c9726e72ec4b1e1e87739739cded4b9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11342) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sv3ndk opened a new pull request #14497: Typo doc operation playground
sv3ndk opened a new pull request #14497: URL: https://github.com/apache/flink/pull/14497 ## What is the purpose of the change fix typo in "Flink Operation Playground": "are generate" => "are generated" ## Brief change log fix typo ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14497: Typo doc operation playground
flinkbot commented on pull request #14497: URL: https://github.com/apache/flink/pull/14497#issuecomment-751373419 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1d7173eb96b45442008011d6008f7e7a3831b116 (Sat Dec 26 16:46:00 UTC 2020) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14497: Typo doc operation playground
flinkbot commented on pull request #14497: URL: https://github.com/apache/flink/pull/14497#issuecomment-751375809 ## CI report: * 1d7173eb96b45442008011d6008f7e7a3831b116 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14497: Typo doc operation playground
flinkbot edited a comment on pull request #14497: URL: https://github.com/apache/flink/pull/14497#issuecomment-751375809 ## CI report: * 1d7173eb96b45442008011d6008f7e7a3831b116 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11343) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14497: Typo doc operation playground
flinkbot edited a comment on pull request #14497: URL: https://github.com/apache/flink/pull/14497#issuecomment-751375809 ## CI report: * 1d7173eb96b45442008011d6008f7e7a3831b116 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11343) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14465: [FLINK-20594][runtime] Remove DefaultExecutionSlotAllocator and its Factory class.
flinkbot edited a comment on pull request #14465: URL: https://github.com/apache/flink/pull/14465#issuecomment-749579707 ## CI report: * 23d374243c9726e72ec4b1e1e87739739cded4b9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11342) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
[
https://issues.apache.org/jira/browse/FLINK-20461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255156#comment-17255156
]
Zhenqiu Huang commented on FLINK-20461:
---
[~xintongsong] [~hxbks2ks]
Tried run the test class 100 times in IntelliJ, none of them failed. I will try
to change the azure-pipelines to run/repeat this single test.
> YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
> --
>
> Key: FLINK-20461
> URL: https://issues.apache.org/jira/browse/FLINK-20461
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Zhenqiu Huang
>Priority: Major
> Labels: testability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> [ERROR]
> testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
> Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does
> not exist:
> hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
wuchong merged pull request #13800: URL: https://github.com/apache/flink/pull/13800 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19650) Support the limit push down for the Jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-19650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-19650.
---
Fix Version/s: 1.13.0
Resolution: Fixed
Fixed in master: a859e2cca7961cc966cf0a3990e821a62fb52654
> Support the limit push down for the Jdbc connector
> --
>
> Key: FLINK-19650
> URL: https://issues.apache.org/jira/browse/FLINK-19650
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC, Table SQL / API
>Reporter: Shengkai Fang
>Assignee: Shengkai Fang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently the blink planner has already supported rule
> {{PushLimitIntoLegacyTableSourceScanRule}}. It's ready to add this feature
> for the jdbc connector.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishment
wuchong merged pull request #14466: URL: https://github.com/apache/flink/pull/14466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20657) Migrate jdbc connection code of JdbcInputFormat, JdbcLookupFunction, JdbcRowDataLookupFunction to SimpleJdbcConnectionProvider
[
https://issues.apache.org/jira/browse/FLINK-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-20657.
---
Resolution: Fixed
Fixed in master: 0c36b666094c01a96d341d2b0ee0b6863a6484e8
> Migrate jdbc connection code of JdbcInputFormat, JdbcLookupFunction,
> JdbcRowDataLookupFunction to SimpleJdbcConnectionProvider
> --
>
> Key: FLINK-20657
> URL: https://issues.apache.org/jira/browse/FLINK-20657
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC, Table SQL / Ecosystem
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, there are multiple places to establish jdbc connection, several of
> them were written before introducing of {{JdbcConnectionProvider}} in
> FLINK-15782. We need to migrate these classes for reasons:
> * Code reusability.
> * Circumvent pitfalls of {{DriverManager}} such as FLINK-19435 and possible
> [others|https://ci.apache.org/projects/tomcat/tomcat85/docs/jndi-datasource-examples-howto.html#DriverManager,_the_service_provider_mechanism_and_memory_leaks]
> in one place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[
https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255172#comment-17255172
]
Jark Wu commented on FLINK-20773:
-
Sounds good to me, as Jackson has also this feature
{{JsonReadFeature#ALLOW_UNESCAPED_CONTROL_CHARS}}.
I assigned this issue to you [~xiaozilong].
> Support allow-unescaped-control-chars option for JSON format
>
>
> Key: FLINK-20773
> URL: https://issues.apache.org/jira/browse/FLINK-20773
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: xiaozilong
>Priority: Major
> Attachments: image-2020-12-25-20-21-50-637.png
>
>
> Can we add an option `allow-unescaped-control-chars` for json format because
> of it will throw exception when exist illegal unquoted characters in the data.
> !image-2020-12-25-20-21-50-637.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20773: Fix Version/s: 1.13.0 > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Assignee: xiaozilong >Priority: Major > Fix For: 1.13.0 > > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add an option `allow-unescaped-control-chars` for json format because > of it will throw exception when exist illegal unquoted characters in the data. > !image-2020-12-25-20-21-50-637.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20773: --- Assignee: xiaozilong > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Assignee: xiaozilong >Priority: Major > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add an option `allow-unescaped-control-chars` for json format because > of it will throw exception when exist illegal unquoted characters in the data. > !image-2020-12-25-20-21-50-637.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16175) Add config option to switch case sensitive for column names in SQL
[ https://issues.apache.org/jira/browse/FLINK-16175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-16175: Fix Version/s: (was: 1.13.0) > Add config option to switch case sensitive for column names in SQL > -- > > Key: FLINK-16175 > URL: https://issues.apache.org/jira/browse/FLINK-16175 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available, usability > Time Spent: 10m > Remaining Estimate: 0h > > Flink SQL is default CaseSensitive and have no option to config. This issue > aims to support > a configOption so that user can set CaseSensitive for their SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17099) Refactoring State TTL solution in Group Agg、Deduplication、TopN operators replace Timer with StateTtlConfig
[
https://issues.apache.org/jira/browse/FLINK-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-17099.
---
Fix Version/s: (was: 1.13.0)
Resolution: Duplicate
This has been fixed by FLINK-17096. But note that TopN is not migrated to the
StateTtlConfig becuase it holds multiple states and needs to sync with each
other.
> Refactoring State TTL solution in Group Agg、Deduplication、TopN operators
> replace Timer with StateTtlConfig
> --
>
> Key: FLINK-17099
> URL: https://issues.apache.org/jira/browse/FLINK-17099
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.9.0, 1.10.0
>Reporter: dalongliu
>Assignee: dalongliu
>Priority: Major
>
> At the moment, there are 2 ways to cleanup states.
> 1) registering a processing-time timer, and cleanup entries when the timer is
> callback.
> - pros: can cleanup multiple states at the same time (state consistent)
> - cons: timer space depends on the key size, which may lead to OOM (heap
> timer).
> - used in Group Aggregation, Over Aggregation, TopN
> 2) using the {{StateTtlConfig}} provided by DataStream [1].
> - pros: decouple the logic of state ttl with the record processing, easy to
> program (take a look at old planner NonWindowJoin which bundles ttl timestamp
> with records in MapState).
> - cons: can't cleanup multiple states at the same time.
> - useed in Sream-Stream Joins.
> For timer solution, although it can cleanup multiple states at the same time,
> but it also will lead to OOM when there have a great many state keys,
> besides, StateTtlConfig is used in stream-stream join case, and will be used
> in more operator. Therefore，in order to unify the state ttl solution,
> simplify the code implemention, and improve the readability of codes, so we
> should refactor state cleanup way which use StateTtlConfig to replace
> processing-time timer in Group Aggregation、Deduplication、TopN operators.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15064) Remove XmlOutput util class in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15064:
Fix Version/s: (was: 1.13.0)
> Remove XmlOutput util class in blink planner
>
>
> Key: FLINK-15064
> URL: https://issues.apache.org/jira/browse/FLINK-15064
> Project: Flink
> Issue Type: Task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
>
> {{org.apache.flink.table.planner.utils.XmlOutput}} is introduced in blink
> planner to fix a redundant line break for each CDATA section. This has been
> fixed in Calcite via 1745f752561be04ae34d1fa08593c2d3ba4470e8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9477) FLIP-90: Support SQL 2016 JSON functions in Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-9477: --- Flagged: Impediment > FLIP-90: Support SQL 2016 JSON functions in Flink SQL > - > > Key: FLINK-9477 > URL: https://issues.apache.org/jira/browse/FLINK-9477 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Shuyi Chen >Assignee: Forward Xu >Priority: Major > Fix For: 1.13.0 > > > FLIP Link > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9477) FLIP-90: Support SQL 2016 JSON functions in Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-9477: --- Flagged: (was: Impediment) > FLIP-90: Support SQL 2016 JSON functions in Flink SQL > - > > Key: FLINK-9477 > URL: https://issues.apache.org/jira/browse/FLINK-9477 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Shuyi Chen >Assignee: Forward Xu >Priority: Major > Fix For: 1.13.0 > > > FLIP Link > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724550 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18778) Support the SupportsProjectionPushDown interface for LookupTableSource
[ https://issues.apache.org/jira/browse/FLINK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18778: Fix Version/s: 1.13.0 > Support the SupportsProjectionPushDown interface for LookupTableSource > -- > > Key: FLINK-18778 > URL: https://issues.apache.org/jira/browse/FLINK-18778 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: hailong wang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20774) Test
Jark Wu created FLINK-20774: --- Summary: Test Key: FLINK-20774 URL: https://issues.apache.org/jira/browse/FLINK-20774 Project: Flink Issue Type: New Feature Reporter: Jark Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20774) Test
[ https://issues.apache.org/jira/browse/FLINK-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20774. --- Resolution: Invalid > Test > > > Key: FLINK-20774 > URL: https://issues.apache.org/jira/browse/FLINK-20774 > Project: Flink > Issue Type: New Feature >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16201) Support JSON_VALUE for blink planner
[ https://issues.apache.org/jira/browse/FLINK-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-16201: Labels: pull-request-available sprint (was: pull-request-available) > Support JSON_VALUE for blink planner > > > Key: FLINK-16201 > URL: https://issues.apache.org/jira/browse/FLINK-16201 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Zili Chen >Assignee: Forward Xu >Priority: Major > Labels: pull-request-available, sprint > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20562) Support ExplainDetails for EXPLAIN sytnax
[
https://issues.apache.org/jira/browse/FLINK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20562:
Labels: sprint (was: )
> Support ExplainDetails for EXPLAIN sytnax
> -
>
> Key: FLINK-20562
> URL: https://issues.apache.org/jira/browse/FLINK-20562
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
> Labels: sprint
> Fix For: 1.13.0
>
>
> Currently, {{EXPLAIN}} syntax only supports to print the default AST, logical
> plan, and physical plan. However, it doesn't support to print detailed
> information such as CHANGELOG_MODE, ESTIMATED_COST, JSON_EXECUTION_PLAN which
> are defined in {{ExplainDetail}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20348) Make "schema-registry.subject" optional for Kafka sink with avro-confluent format
[
https://issues.apache.org/jira/browse/FLINK-20348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20348:
Labels: sprint (was: )
> Make "schema-registry.subject" optional for Kafka sink with avro-confluent
> format
> -
>
> Key: FLINK-20348
> URL: https://issues.apache.org/jira/browse/FLINK-20348
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Priority: Major
> Labels: sprint
> Fix For: 1.13.0
>
>
> Currently, configuration "schema-registry.subject" in avro-confluent format
> is required by sink. However, this is quite verbose set it manually. By
> default, it can be to set to {{-key}} and {{-value}}
> if it works with kafka or upsert-kafka connector. This can also makes
> 'avro-confluent' format to be more handy and works better with
> Kafka/Confluent ecosystem.
> {code:sql}
> CREATE TABLE kafka_gmv (
> day_str STRING,
> gmv BIGINT,
> PRIMARY KEY (day_str) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'topic' = 'kafka_gmv',
> 'properties.bootstrap.servers' = 'localhost:9092',
> -- 'key.format' = 'raw',
> 'key.format' = 'avro-confluent',
> 'key.avro-confluent.schema-registry.url' = 'http://localhost:8181',
> 'key.avro-confluent.schema-registry.subject' = 'kafka_gmv-key',
> 'value.format' = 'avro-confluent',
> 'value.avro-confluent.schema-registry.url' = 'http://localhost:8181',
> 'value.avro-confluent.schema-registry.subject' = 'kafka_gmv-value'
> );
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17855) UDF with parameter Array(Row) can not work
[
https://issues.apache.org/jira/browse/FLINK-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-17855:
Labels: sprint (was: )
> UDF with parameter Array(Row) can not work
> --
>
> Key: FLINK-17855
> URL: https://issues.apache.org/jira/browse/FLINK-17855
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jingsong Lee
>Priority: Major
> Labels: sprint
> Fix For: 1.13.0
>
>
> {code:java}
> public String eval(Row[] rows) {
> ...
> }
> {code}
> Can not work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14364) Allow comments fail when not ignore parse errors in CsvRowDeserializationSchema
[ https://issues.apache.org/jira/browse/FLINK-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-14364: Fix Version/s: (was: 1.13.0) > Allow comments fail when not ignore parse errors in > CsvRowDeserializationSchema > --- > > Key: FLINK-14364 > URL: https://issues.apache.org/jira/browse/FLINK-14364 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Jingsong Lee >Assignee: Jiayi Liao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Use CsvRowDeserializationSchema, when setIgnoreParseErrors(false) and > setAllowComments(true). > If there are some comments in msg, will throw MismatchedInputException. > If this a bug? and we should catch MismatchedInputException and return null? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14364) Allow comments fail when not ignore parse errors in CsvRowDeserializationSchema
[
https://issues.apache.org/jira/browse/FLINK-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255178#comment-17255178
]
Jark Wu commented on FLINK-14364:
-
According to {{CsvRowDataSerDeSchemaTest#testDeserializeIgnoreComment}} and
{{CsvRowDataSerDeSchemaTest#testDeserializeAllowComment}}, I think this has
been fixed in the new CSV format? Could we close this?
> Allow comments fail when not ignore parse errors in
> CsvRowDeserializationSchema
> ---
>
> Key: FLINK-14364
> URL: https://issues.apache.org/jira/browse/FLINK-14364
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jingsong Lee
>Assignee: Jiayi Liao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Use CsvRowDeserializationSchema, when setIgnoreParseErrors(false) and
> setAllowComments(true).
> If there are some comments in msg, will throw MismatchedInputException.
> If this a bug? and we should catch MismatchedInputException and return null?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20321) Get NPE when using AvroDeserializationSchema to deserialize null input
[
https://issues.apache.org/jira/browse/FLINK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255179#comment-17255179
]
Jark Wu commented on FLINK-20321:
-
Hi [~sampadsaha5], are you still working on this? What's the status of this
issue?
> Get NPE when using AvroDeserializationSchema to deserialize null input
> --
>
> Key: FLINK-20321
> URL: https://issues.apache.org/jira/browse/FLINK-20321
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: Shengkai Fang
>Assignee: Sampad Kumar Saha
>Priority: Major
> Labels: starter
> Fix For: 1.13.0
>
>
> You can reproduce the bug by adding the code into the
> {{AvroDeserializationSchemaTest}}.
> The code follows
> {code:java}
> @Test
> public void testSpecificRecord2() throws Exception {
> DeserializationSchema deserializer =
> AvroDeserializationSchema.forSpecific(Address.class);
> Address deserializedAddress = deserializer.deserialize(null);
> assertEquals(null, deserializedAddress);
> }
> {code}
> Exception stack:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.formats.avro.utils.MutableByteArrayInputStream.setBuffer(MutableByteArrayInputStream.java:43)
> at
> org.apache.flink.formats.avro.AvroDeserializationSchema.deserialize(AvroDeserializationSchema.java:131)
> at
> org.apache.flink.formats.avro.AvroDeserializationSchemaTest.testSpecificRecord2(AvroDeserializationSchemaTest.java:69)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20321) Get NPE when using AvroDeserializationSchema to deserialize null input
[
https://issues.apache.org/jira/browse/FLINK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20321:
Labels: sprint starter (was: starter)
> Get NPE when using AvroDeserializationSchema to deserialize null input
> --
>
> Key: FLINK-20321
> URL: https://issues.apache.org/jira/browse/FLINK-20321
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: Shengkai Fang
>Assignee: Sampad Kumar Saha
>Priority: Major
> Labels: sprint, starter
> Fix For: 1.13.0
>
>
> You can reproduce the bug by adding the code into the
> {{AvroDeserializationSchemaTest}}.
> The code follows
> {code:java}
> @Test
> public void testSpecificRecord2() throws Exception {
> DeserializationSchema deserializer =
> AvroDeserializationSchema.forSpecific(Address.class);
> Address deserializedAddress = deserializer.deserialize(null);
> assertEquals(null, deserializedAddress);
> }
> {code}
> Exception stack:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.formats.avro.utils.MutableByteArrayInputStream.setBuffer(MutableByteArrayInputStream.java:43)
> at
> org.apache.flink.formats.avro.AvroDeserializationSchema.deserialize(AvroDeserializationSchema.java:131)
> at
> org.apache.flink.formats.avro.AvroDeserializationSchemaTest.testSpecificRecord2(AvroDeserializationSchemaTest.java:69)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17868) Proctime in DDL can not work in batch mode
[ https://issues.apache.org/jira/browse/FLINK-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-17868: Labels: sprint (was: ) > Proctime in DDL can not work in batch mode > -- > > Key: FLINK-17868 > URL: https://issues.apache.org/jira/browse/FLINK-17868 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jingsong Lee >Priority: Major > Labels: sprint > Fix For: 1.13.0 > > > The data of this proctime column will be all null. Should same to current > timestamp. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19062) Improve alter table DDL
[ https://issues.apache.org/jira/browse/FLINK-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-19062: Labels: starter (was: ) > Improve alter table DDL > --- > > Key: FLINK-19062 > URL: https://issues.apache.org/jira/browse/FLINK-19062 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Jingsong Lee >Priority: Major > Labels: starter > Fix For: 1.13.0 > > > Support: > * ALTER TABLE table_name UNSET PROPERTIES (key) > * ALTER TABLE table_name CHANGE [COLUMN] col_old_name col_new_name > column_type [COMMENT col_comment] > * ALTER TABLE table_name add `column` ... > * ALTER TABLE table_name replace `column` ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19062) Improve alter table DDL
[ https://issues.apache.org/jira/browse/FLINK-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-19062: Labels: (was: starter) > Improve alter table DDL > --- > > Key: FLINK-19062 > URL: https://issues.apache.org/jira/browse/FLINK-19062 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.13.0 > > > Support: > * ALTER TABLE table_name UNSET PROPERTIES (key) > * ALTER TABLE table_name CHANGE [COLUMN] col_old_name col_new_name > column_type [COMMENT col_comment] > * ALTER TABLE table_name add `column` ... > * ALTER TABLE table_name replace `column` ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16627) Support only generate non-null values when serializing into JSON
[
https://issues.apache.org/jira/browse/FLINK-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255180#comment-17255180
]
Jark Wu commented on FLINK-16627:
-
Hi [~jackray], are you still working on this issue?
I agree with [~nilerzhou] that a new config option key style has been
introduced in FLIP-122, we may need to come up with a new key for this option.
{{'json.encode.ignore-null-fields'=false/true}} sounds good to me. As an
alternative, {{'json.encode.include-null-fields'=true/false}} is also fine to
me.
> Support only generate non-null values when serializing into JSON
>
>
> Key: FLINK-16627
> URL: https://issues.apache.org/jira/browse/FLINK-16627
> Project: Flink
> Issue Type: New Feature
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Planner
>Affects Versions: 1.10.0
>Reporter: jackray wang
>Assignee: jackray wang
>Priority: Major
> Fix For: 1.13.0
>
>
> {code:java}
> //sql
> CREATE TABLE sink_kafka ( subtype STRING , svt STRING ) WITH (……）
> {code}
>
> {code:java}
> //sql
> CREATE TABLE source_kafka ( subtype STRING , svt STRING ) WITH (……）
> {code}
>
> {code:java}
> //scala udf
> class ScalaUpper extends ScalarFunction {
> def eval(str: String) : String= {
>if(str == null){
>return ""
>}else{
>return str
>}
> }
>
> }
> btenv.registerFunction("scala_upper", new ScalaUpper())
> {code}
>
> {code:java}
> //sql
> insert into sink_kafka select subtype, scala_upper(svt) from source_kafka
> {code}
>
>
>
> Sometimes the svt's value is null, inert into kafkas json like
> \{"subtype":"qin","svt":null}
> If the amount of data is small, it is acceptable，but we process 10TB of data
> every day, and there may be many nulls in the json, which affects the
> efficiency. If you can add a parameter to remove the null key when defining a
> sinktable, the performance will be greatly improved
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16627) Support only generate non-null values when serializing into JSON
[
https://issues.apache.org/jira/browse/FLINK-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-16627:
Labels: sprint (was: )
> Support only generate non-null values when serializing into JSON
>
>
> Key: FLINK-16627
> URL: https://issues.apache.org/jira/browse/FLINK-16627
> Project: Flink
> Issue Type: New Feature
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Planner
>Affects Versions: 1.10.0
>Reporter: jackray wang
>Assignee: jackray wang
>Priority: Major
> Labels: sprint
> Fix For: 1.13.0
>
>
> {code:java}
> //sql
> CREATE TABLE sink_kafka ( subtype STRING , svt STRING ) WITH (……）
> {code}
>
> {code:java}
> //sql
> CREATE TABLE source_kafka ( subtype STRING , svt STRING ) WITH (……）
> {code}
>
> {code:java}
> //scala udf
> class ScalaUpper extends ScalarFunction {
> def eval(str: String) : String= {
>if(str == null){
>return ""
>}else{
>return str
>}
> }
>
> }
> btenv.registerFunction("scala_upper", new ScalaUpper())
> {code}
>
> {code:java}
> //sql
> insert into sink_kafka select subtype, scala_upper(svt) from source_kafka
> {code}
>
>
>
> Sometimes the svt's value is null, inert into kafkas json like
> \{"subtype":"qin","svt":null}
> If the amount of data is small, it is acceptable，but we process 10TB of data
> every day, and there may be many nulls in the json, which affects the
> efficiency. If you can add a parameter to remove the null key when defining a
> sinktable, the performance will be greatly improved
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20644) Check return type of ScalarFunction eval method shouldn't be void
[
https://issues.apache.org/jira/browse/FLINK-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20644:
Labels: sprint starter (was: starter)
> Check return type of ScalarFunction eval method shouldn't be void
> -
>
> Key: FLINK-20644
> URL: https://issues.apache.org/jira/browse/FLINK-20644
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.11.1
> Environment: groupId:org.apache.flink
> artifactId:flink-table-api-scala-bridge_2.11
> version:1.11.1
>Reporter: shiyu
>Priority: Major
> Labels: sprint, starter
> Fix For: 1.13.0
>
> Attachments: image-2020-12-17-16-04-15-131.png,
> image-2020-12-17-16-07-39-827.png
>
>
> flink-table-api-scala-bridge_2.11
> !image-2020-12-17-16-07-39-827.png!
> !image-2020-12-17-16-04-15-131.png!
> console:
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Failed
> to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to
> no-operation (NOP) logger implementationSLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.ERROR
> StatusLogger Log4j2 could not find a logging implementation. Please add
> log4j-core to the classpath. Using SimpleLogger to log to the console.../* 1
> *//* 2 */ public class StreamExecCalc$13 extends
> org.apache.flink.table.runtime.operators.AbstractProcessStreamOperator/* 3 */
> implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator \{/* 4 *//* 5
> */ private final Object[] references;/* 6 */ private transient
> org.apache.flink.table.runtime.typeutils.StringDataSerializer
> typeSerializer$6;/* 7 */ private transient
> org.apache.flink.table.data.util.DataFormatConverters.StringConverter
> converter$9;/* 8 */ private transient
> cn.bicon.tableapitest.udf.ScalarFunctionTest$HashCode
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58;/*
> 9 */ private transient
> org.apache.flink.table.data.util.DataFormatConverters.GenericConverter
> converter$12;/* 10 */ org.apache.flink.table.data.BoxedWrapperRowData
> out = new org.apache.flink.table.data.BoxedWrapperRowData(3);/* 11 */
> private final org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> outElement = new
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);/* 12 *//*
> 13 */ public StreamExecCalc$13(/* 14 */ Object[]
> references,/* 15 */
> org.apache.flink.streaming.runtime.tasks.StreamTask task,/* 16 */
> org.apache.flink.streaming.api.graph.StreamConfig config,/* 17 */
> org.apache.flink.streaming.api.operators.Output output,/* 18 */
> org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
> processingTimeService) throws Exception {/* 19 */ this.references =
> references;/* 20 */ typeSerializer$6 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0]));/* 21 */ converter$9 =
> (((org.apache.flink.table.data.util.DataFormatConverters.StringConverter)
> references[1]));/* 22 */
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58
> = (((cn.bicon.tableapitest.udf.ScalarFunctionTest$HashCode)
> references[2]));/* 23 */ converter$12 =
> (((org.apache.flink.table.data.util.DataFormatConverters.GenericConverter)
> references[3]));/* 24 */ this.setup(task, config, output);/* 25 */
> if (this instanceof
> org.apache.flink.streaming.api.operators.AbstractStreamOperator) {/* 26 */
> ((org.apache.flink.streaming.api.operators.AbstractStreamOperator)
> this)/* 27 */
> .setProcessingTimeService(processingTimeService);/* 28 */ }/* 29 */
> }/* 30 *//* 31 */ @Override/* 32 */ public void open()
> throws Exception \{/* 33 */ super.open();/* 34 */ /* 35 */
>
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58.open(new
> org.apache.flink.table.functions.FunctionContext(getRuntimeContext()));/* 36
> */ /* 37 */ }/* 38 *//* 39 */ @Override/* 40 */
> public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception \{/* 41 */
> org.apache.flink.table.data.RowData in1 =
> (org.apache.flink.table.data.RowData) element.getValue();/* 42 */ /*
> 43 */ org.apache.flink.table.data.binary.BinaryStringData field$5;/*
> 44 */ boolean isNull$5;/* 45 */
> org.apache.flink.table.d
[jira] [Closed] (FLINK-18345) Support filter push down for lookup table source
[ https://issues.apache.org/jira/browse/FLINK-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-18345. --- Fix Version/s: (was: 1.13.0) Resolution: Duplicate > Support filter push down for lookup table source > > > Key: FLINK-18345 > URL: https://issues.apache.org/jira/browse/FLINK-18345 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jingsong Lee >Priority: Major > > Especially for Filesystem lookup table source, it stores all records in > memory, if there is filter push down, can reduce memory effectively for > lookup table source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20644) Check return type of ScalarFunction eval method shouldn't be void
[
https://issues.apache.org/jira/browse/FLINK-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20644:
Labels: starter (was: sprint starter)
> Check return type of ScalarFunction eval method shouldn't be void
> -
>
> Key: FLINK-20644
> URL: https://issues.apache.org/jira/browse/FLINK-20644
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.11.1
> Environment: groupId:org.apache.flink
> artifactId:flink-table-api-scala-bridge_2.11
> version:1.11.1
>Reporter: shiyu
>Priority: Major
> Labels: starter
> Fix For: 1.13.0
>
> Attachments: image-2020-12-17-16-04-15-131.png,
> image-2020-12-17-16-07-39-827.png
>
>
> flink-table-api-scala-bridge_2.11
> !image-2020-12-17-16-07-39-827.png!
> !image-2020-12-17-16-04-15-131.png!
> console:
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Failed
> to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to
> no-operation (NOP) logger implementationSLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.ERROR
> StatusLogger Log4j2 could not find a logging implementation. Please add
> log4j-core to the classpath. Using SimpleLogger to log to the console.../* 1
> *//* 2 */ public class StreamExecCalc$13 extends
> org.apache.flink.table.runtime.operators.AbstractProcessStreamOperator/* 3 */
> implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator \{/* 4 *//* 5
> */ private final Object[] references;/* 6 */ private transient
> org.apache.flink.table.runtime.typeutils.StringDataSerializer
> typeSerializer$6;/* 7 */ private transient
> org.apache.flink.table.data.util.DataFormatConverters.StringConverter
> converter$9;/* 8 */ private transient
> cn.bicon.tableapitest.udf.ScalarFunctionTest$HashCode
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58;/*
> 9 */ private transient
> org.apache.flink.table.data.util.DataFormatConverters.GenericConverter
> converter$12;/* 10 */ org.apache.flink.table.data.BoxedWrapperRowData
> out = new org.apache.flink.table.data.BoxedWrapperRowData(3);/* 11 */
> private final org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> outElement = new
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);/* 12 *//*
> 13 */ public StreamExecCalc$13(/* 14 */ Object[]
> references,/* 15 */
> org.apache.flink.streaming.runtime.tasks.StreamTask task,/* 16 */
> org.apache.flink.streaming.api.graph.StreamConfig config,/* 17 */
> org.apache.flink.streaming.api.operators.Output output,/* 18 */
> org.apache.flink.streaming.runtime.tasks.ProcessingTimeService
> processingTimeService) throws Exception {/* 19 */ this.references =
> references;/* 20 */ typeSerializer$6 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0]));/* 21 */ converter$9 =
> (((org.apache.flink.table.data.util.DataFormatConverters.StringConverter)
> references[1]));/* 22 */
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58
> = (((cn.bicon.tableapitest.udf.ScalarFunctionTest$HashCode)
> references[2]));/* 23 */ converter$12 =
> (((org.apache.flink.table.data.util.DataFormatConverters.GenericConverter)
> references[3]));/* 24 */ this.setup(task, config, output);/* 25 */
> if (this instanceof
> org.apache.flink.streaming.api.operators.AbstractStreamOperator) {/* 26 */
> ((org.apache.flink.streaming.api.operators.AbstractStreamOperator)
> this)/* 27 */
> .setProcessingTimeService(processingTimeService);/* 28 */ }/* 29 */
> }/* 30 *//* 31 */ @Override/* 32 */ public void open()
> throws Exception \{/* 33 */ super.open();/* 34 */ /* 35 */
>
> function_cn$bicon$tableapitest$udf$ScalarFunctionTest$HashCode$8999e79cc91b971a8777461fb7698c58.open(new
> org.apache.flink.table.functions.FunctionContext(getRuntimeContext()));/* 36
> */ /* 37 */ }/* 38 *//* 39 */ @Override/* 40 */
> public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception \{/* 41 */
> org.apache.flink.table.data.RowData in1 =
> (org.apache.flink.table.data.RowData) element.getValue();/* 42 */ /*
> 43 */ org.apache.flink.table.data.binary.BinaryStringData field$5;/*
> 44 */ boolean isNull$5;/* 45 */
> org.apache.flink.table.data.bina
[jira] [Updated] (FLINK-18779) Support the SupportsFilterPushDown interface for LookupTableSource
[ https://issues.apache.org/jira/browse/FLINK-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18779: Labels: pull-request-available (was: pull-request-available sprint) > Support the SupportsFilterPushDown interface for LookupTableSource > -- > > Key: FLINK-18779 > URL: https://issues.apache.org/jira/browse/FLINK-18779 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18779) Support the SupportsFilterPushDown interface for LookupTableSource
[ https://issues.apache.org/jira/browse/FLINK-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18779: Labels: pull-request-available sprint (was: pull-request-available) > Support the SupportsFilterPushDown interface for LookupTableSource > -- > > Key: FLINK-18779 > URL: https://issues.apache.org/jira/browse/FLINK-18779 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available, sprint > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18778) Support the SupportsProjectionPushDown interface for LookupTableSource
[ https://issues.apache.org/jira/browse/FLINK-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18778: Labels: pull-request-available sprint (was: pull-request-available) > Support the SupportsProjectionPushDown interface for LookupTableSource > -- > > Key: FLINK-18778 > URL: https://issues.apache.org/jira/browse/FLINK-18778 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: hailong wang >Priority: Major > Labels: pull-request-available, sprint > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18202: Labels: pull-request-available sprint (was: pull-request-available) > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Priority: Major > Labels: pull-request-available, sprint > Fix For: 1.13.0 > > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18202: Fix Version/s: 1.13.0 > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-18202) Introduce Protobuf format
[ https://issues.apache.org/jira/browse/FLINK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-18202: --- Assignee: Suhan Mao > Introduce Protobuf format > - > > Key: FLINK-18202 > URL: https://issues.apache.org/jira/browse/FLINK-18202 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Benchao Li >Assignee: Suhan Mao >Priority: Major > Labels: pull-request-available, sprint > Fix For: 1.13.0 > > Attachments: image-2020-06-15-17-18-03-182.png > > > PB[1] is a very famous and wildly used (de)serialization framework. The ML[2] > also has some discussions about this. It's a useful feature. > This issue maybe needs some designs, or a FLIP. > [1] [https://developers.google.com/protocol-buffers] > [2] [http://apache-flink.147419.n8.nabble.com/Flink-SQL-UDF-td3725.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16201) Support JSON_VALUE for blink planner
[ https://issues.apache.org/jira/browse/FLINK-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-16201: --- Assignee: (was: Forward Xu) > Support JSON_VALUE for blink planner > > > Key: FLINK-16201 > URL: https://issues.apache.org/jira/browse/FLINK-16201 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Zili Chen >Priority: Major > Labels: pull-request-available, sprint > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
flinkbot edited a comment on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324 ## CI report: * d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN * c6cd36a8dfd854ecf63b2d248a4a28f889c873ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11341) * d58927642909f50571ed6242605aac564e074f89 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #14437: [FLINK-20458][docs] translate gettingStarted.zh.md and correct spelling errors in gettingStarted.md
fsk119 commented on a change in pull request #14437:
URL: https://github.com/apache/flink/pull/14437#discussion_r549073093
##
File path: docs/dev/table/sql/gettingStarted.zh.md
##
@@ -22,54 +22,54 @@ specific language governing permissions and limitations
under the License.
-->
-Flink SQL makes it simple to develop streaming applications using standard
SQL. It is easy to learn Flink if you have ever worked with a database or SQL
like system by remaining ANSI-SQL 2011 compliant. This tutorial will help you
get started quickly with a Flink SQL development environment.
+Flink SQL 使得使用标准 SQL 开发流应用程序变的简单。如果你曾经在工作中使用过兼容 ANSI-SQL 2011 的数据库或类似的 SQL
系统，那么就很容易学习 Flink。本教程将帮助你快速入门 Flink SQL 开发环境。
* This will be replaced by the TOC
{:toc}
-### Prerequisetes
+### 先决条件
-You only need to have basic knowledge of SQL to follow along. No other
programming experience is assumed.
+你只需要具备 SQL 的基础知识即可，不需要其他编程经验。
-### Installation
+### 安装
-There are multiple ways to install Flink. For experimentation, the most common
option is to download the binaries and run them locally. You can follow the
steps in [local installation]({%link try-flink/local_installation.zh.md %}) to
set up an environment for the rest of the tutorial.
+安装 Flink 有多种方式。为了实验，最常见的选择是下载二进制包并在本地运行。你可以按照[本地模式安装]({% link
try-flink/local_installation.zh.md %})中的步骤为本教程的剩余部分设置环境。
-Once you're all set, use the following command to start a local cluster from
the installation folder:
+完成所有设置后，在安装文件夹中使用以下命令启动本地集群：
{% highlight bash %}
./bin/start-cluster.sh
{% endhighlight %}
-Once started, the Flink WebUI on [localhost:8081](localhost:8081) is available
locally, from which you can monitor the different jobs.
+启动完成后，就可以在本地访问 Flink WebUI [localhost:8081](localhost:8081)，你可以通过它来监视不同的作业。
Review comment:
读起来似乎不太通顺？ 或者 另起一句：通过它，你可以监控不同的作业。
##
File path: docs/dev/table/sql/gettingStarted.zh.md
##
@@ -129,11 +128,11 @@ FROM employee_information
GROUP BY dep_id;
{% endhighlight %}
-Such queries are considered _stateful_. Flink's advanced fault-tolerance
mechanism will maintain internal state and consistency, so queries always
return the correct result, even in the face of hardware failure.
+这样的查询被认为是 _有状态的_。Flink 的高级容错机制将维持内部状态和一致性，因此即使遇到硬件故障，查询也始终返回正确结果。
-## Sink Tables
+## Sink 表
-When running this query, the SQL client provides output in real-time but in a
read-only fashion. Storing results - to power a report or dashboard - requires
writing out to another table. This can be achieved using an `INSERT INTO`
statement. The table referenced in this clause is known as a sink table. An
`INSERT INTO` statement will be submitted as a detached query to the Flink
cluster.
+当运行此查询时，SQL 客户端实时但是以只读方式提供输出。存储结果（为报表或仪表板提供数据来源）需要写到另一个表。这可以使用 `INSERT INTO`
语句来实现。本节中引用的表称为 sink 表。`INSERT INTO` 语句将作为一个独立查询被提交到 Flink 集群中。
Review comment:
"存储结果（为报表或仪表板提供数据来源）需要写到另一个表" -> "存储结果，作为报表或仪表板提供数据来源，需要写到另一个表"
##
File path: docs/dev/table/sql/gettingStarted.zh.md
##
@@ -22,54 +22,54 @@ specific language governing permissions and limitations
under the License.
-->
-Flink SQL makes it simple to develop streaming applications using standard
SQL. It is easy to learn Flink if you have ever worked with a database or SQL
like system by remaining ANSI-SQL 2011 compliant. This tutorial will help you
get started quickly with a Flink SQL development environment.
+Flink SQL 使得使用标准 SQL 开发流应用程序变的简单。如果你曾经在工作中使用过兼容 ANSI-SQL 2011 的数据库或类似的 SQL
系统，那么就很容易学习 Flink。本教程将帮助你快速入门 Flink SQL 开发环境。
Review comment:
get started quickly with a Flink SQL development environment -> 在 Flink
SQL 开发环境下快速入门
##
File path: docs/dev/table/sql/gettingStarted.zh.md
##
@@ -79,16 +79,15 @@ SELECT CURRENT_TIMESTAMP;
{% top %}
-## Source Tables
+## Source 表
-As with all SQL engines, Flink queries operate on top of tables.
-It differs from a traditional database because Flink does not manage data at
rest locally; instead, its queries operate continuously over external tables.
+与所有 SQL 引擎一样，Flink 查询在表上进行操作。与传统数据库不同，因为 Flink 不在本地管理静态数据；相反，它的查询在外部表上连续运行。
-Flink data processing pipelines begin with source tables. Source tables
produce rows operated over during the query's execution; they are the tables
referenced in the `FROM` clause of a query. These could be Kafka topics,
databases, filesystems, or any other system that Flink knows how to consume.
+Flink 数据处理管道开始于 source 表。在查询执行期间，source 表产生操作的行；它们是查询时 `FROM` 子句中引用的表。这些表可能是
Kafka 的 topics，数据库，文件系统，或 Flink 知道如何消费的任何其他系统。
Review comment:
pipeline 在这里应该是流水线的意思。
"Source tables produce rows operated over during the query's execution" ->
Source 表产生在查询执行期间可以被操作的行 。
“or any other system that Flink knows how to consume” -> 或者 任何其它 Flink
知道如何消费的系统
##
File path: docs/dev/table/sql/gettingStarted.zh.md
##
@@ -113,13 +112,13 @@ SELECT * from employee_information WHERE DeptId = 1;
{% top %}
