[jira] [Assigned] (FLINK-34147) TimestampData to/from LocalDateTime is ambiguous
[
https://issues.apache.org/jira/browse/FLINK-34147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-34147:
--
Assignee: Yunhong Zheng
> TimestampData to/from LocalDateTime is ambiguous
>
>
> Key: FLINK-34147
> URL: https://issues.apache.org/jira/browse/FLINK-34147
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Rui Li
>Assignee: Yunhong Zheng
>Priority: Major
>
> It seems TimestampData is essentially an {{Instant}}. Therefore an implicit
> time zone is used in the {{fromLocalDateTime}} and {{toLocalDateTime}}
> methods. However neither the method name nor the API doc indicates which time
> zone is used. So from caller's perspective, the results of these two methods
> are ambiguous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34147) TimestampData to/from LocalDateTime is ambiguous
[
https://issues.apache.org/jira/browse/FLINK-34147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814581#comment-17814581
]
Rui Li commented on FLINK-34147:
Thanks [~337361...@qq.com], just assigned to you!
> TimestampData to/from LocalDateTime is ambiguous
>
>
> Key: FLINK-34147
> URL: https://issues.apache.org/jira/browse/FLINK-34147
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Rui Li
>Assignee: Yunhong Zheng
>Priority: Major
>
> It seems TimestampData is essentially an {{Instant}}. Therefore an implicit
> time zone is used in the {{fromLocalDateTime}} and {{toLocalDateTime}}
> methods. However neither the method name nor the API doc indicates which time
> zone is used. So from caller's perspective, the results of these two methods
> are ambiguous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34147) TimestampData to/from LocalDateTime is ambiguous
[
https://issues.apache.org/jira/browse/FLINK-34147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810347#comment-17810347
]
Rui Li commented on FLINK-34147:
I think having two separate classes is fine, but it might be an overkill.
Perhaps we can just add methods that take a time zone parameter? Or we can at
least improve the doc to clarify all conversions are performed in UTC.
> TimestampData to/from LocalDateTime is ambiguous
>
>
> Key: FLINK-34147
> URL: https://issues.apache.org/jira/browse/FLINK-34147
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Rui Li
>Priority: Major
>
> It seems TimestampData is essentially an {{Instant}}. Therefore an implicit
> time zone is used in the {{fromLocalDateTime}} and {{toLocalDateTime}}
> methods. However neither the method name nor the API doc indicates which time
> zone is used. So from caller's perspective, the results of these two methods
> are ambiguous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34147) TimestampData to/from LocalDateTime is ambiguous
Rui Li created FLINK-34147:
--
Summary: TimestampData to/from LocalDateTime is ambiguous
Key: FLINK-34147
URL: https://issues.apache.org/jira/browse/FLINK-34147
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Rui Li
It seems TimestampData is essentially an {{Instant}}. Therefore an implicit
time zone is used in the {{fromLocalDateTime}} and {{toLocalDateTime}} methods.
However neither the method name nor the API doc indicates which time zone is
used. So from caller's perspective, the results of these two methods are
ambiguous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29527) Make unknownFieldsIndices work for single ParquetReader
[ https://issues.apache.org/jira/browse/FLINK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617444#comment-17617444 ] Rui Li commented on FLINK-29527: [~suns] Assigned. Thanks for taking the issue > Make unknownFieldsIndices work for single ParquetReader > --- > > Key: FLINK-29527 > URL: https://issues.apache.org/jira/browse/FLINK-29527 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Sun Shun >Assignee: Sun Shun >Priority: Major > Labels: pull-request-available > > Currently, from the improvement FLINK-23715, Flink use a collection named > `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside > the `ParquetVectorizedInputFormat`, and applied to all parquet files under > given path. > However, some fields may only be nonexistent in some of the historical > parquet files, while exist in latest ones. And based on > `unknownFieldsIndices`, flink will always skip these fields, even thought > they are existing in the later parquets. > As a result, the value of these fields will become empty when they are > nonexistent in some historical parquet files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29527) Make unknownFieldsIndices work for single ParquetReader
[ https://issues.apache.org/jira/browse/FLINK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-29527: -- Assignee: Sun Shun > Make unknownFieldsIndices work for single ParquetReader > --- > > Key: FLINK-29527 > URL: https://issues.apache.org/jira/browse/FLINK-29527 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Sun Shun >Assignee: Sun Shun >Priority: Major > Labels: pull-request-available > > Currently, from the improvement FLINK-23715, Flink use a collection named > `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside > the `ParquetVectorizedInputFormat`, and applied to all parquet files under > given path. > However, some fields may only be nonexistent in some of the historical > parquet files, while exist in latest ones. And based on > `unknownFieldsIndices`, flink will always skip these fields, even thought > they are existing in the later parquets. > As a result, the value of these fields will become empty when they are > nonexistent in some historical parquet files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26043) Add periodic kerberos relogin to DelegationTokenManager
[ https://issues.apache.org/jira/browse/FLINK-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605597#comment-17605597 ] Rui Li commented on FLINK-26043: Got it, thanks [~gaborgsomogyi]! > Add periodic kerberos relogin to DelegationTokenManager > --- > > Key: FLINK-26043 > URL: https://issues.apache.org/jira/browse/FLINK-26043 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26043) Add periodic kerberos relogin to DelegationTokenManager
[
https://issues.apache.org/jira/browse/FLINK-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605341#comment-17605341
]
Rui Li commented on FLINK-26043:
I mean what if delegation token is disabled, i.e.
{{security.kerberos.fetch.delegation-token}} is set to false and user provides
keytab via {{security.kerberos.login.keytab}}? It seems there won't be re-login
in that case right?
> Add periodic kerberos relogin to DelegationTokenManager
> ---
>
> Key: FLINK-26043
> URL: https://issues.apache.org/jira/browse/FLINK-26043
> Project: Flink
> Issue Type: Sub-task
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26043) Add periodic kerberos relogin to DelegationTokenManager
[ https://issues.apache.org/jira/browse/FLINK-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605315#comment-17605315 ] Rui Li commented on FLINK-26043: Hey [~gaborgsomogyi], just curious, does the periodical re-login only work with delegation tokens? For keytab mode, I guess we also need to avoid TGT expiration, right? > Add periodic kerberos relogin to DelegationTokenManager > --- > > Key: FLINK-26043 > URL: https://issues.apache.org/jira/browse/FLINK-26043 > Project: Flink > Issue Type: Sub-task >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-28247) Exception will be thrown when over window contains grouping in Hive Dialect
[
https://issues.apache.org/jira/browse/FLINK-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-28247.
Fix Version/s: 1.16.0
Resolution: Fixed
> Exception will be thrown when over window contains grouping in Hive Dialect
> ---
>
> Key: FLINK-28247
> URL: https://issues.apache.org/jira/browse/FLINK-28247
> Project: Flink
> Issue Type: Sub-task
>Reporter: luoyuxia
>Assignee: luoyuxia
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> The exception will be reprodued by the following sql when using Hive Dialect:
> {code:java}
> create table t(category int, live int, comments int)
> SELECT grouping(category), lag(live) over(partition by grouping(category))
> FROM t GROUP BY category, live; {code}
> The reson is it will first call
> `HiveParserCalcitePlanner#genSelectForWindowing` to generate the window,
> which will then call `HiveParserUtils#rewriteGroupingFunctionAST` to rewrite
> the group function in the over window :
>
> {code:java}
> // rewrite grouping function
> if (current.getType() == HiveASTParser.TOK_FUNCTION
> && current.getChildCount() >= 2) {
> HiveParserASTNode func = (HiveParserASTNode) current.getChild(0);
> if (func.getText().equals("grouping")) {
> visited.setValue(true);
> convertGrouping(
> current, grpByAstExprs, noneSet, legacyGrouping, found);
> }
> }
> {code}
>
> So `grouping(category)` will be converted to `grouping(0, 1)`.
> After `HiveParserCalcitePlanner#genSelectForWindowing`, it will try to
> rewrite it again:
>
> {code:java}
> if (!qbp.getDestToGroupBy().isEmpty()) {
> // Special handling of grouping function
> expr =
> rewriteGroupingFunctionAST(
> getGroupByForClause(qbp, selClauseName),
> expr,
> !cubeRollupGrpSetPresent);
> } {code}
> And it will also fall back to `convertGrouping` again as
> `current.getChildCount() >= 2` will be true. But then, it can't find any
> field
> presented in group by for it's `grouping(0, 1)` now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28247) Exception will be thrown when over window contains grouping in Hive Dialect
[
https://issues.apache.org/jira/browse/FLINK-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564078#comment-17564078
]
Rui Li commented on FLINK-28247:
Fixed in master: a069a306a0c95ba62bbc63f6afc995fd3216a326
> Exception will be thrown when over window contains grouping in Hive Dialect
> ---
>
> Key: FLINK-28247
> URL: https://issues.apache.org/jira/browse/FLINK-28247
> Project: Flink
> Issue Type: Sub-task
>Reporter: luoyuxia
>Assignee: luoyuxia
>Priority: Major
> Labels: pull-request-available
>
> The exception will be reprodued by the following sql when using Hive Dialect:
> {code:java}
> create table t(category int, live int, comments int)
> SELECT grouping(category), lag(live) over(partition by grouping(category))
> FROM t GROUP BY category, live; {code}
> The reson is it will first call
> `HiveParserCalcitePlanner#genSelectForWindowing` to generate the window,
> which will then call `HiveParserUtils#rewriteGroupingFunctionAST` to rewrite
> the group function in the over window :
>
> {code:java}
> // rewrite grouping function
> if (current.getType() == HiveASTParser.TOK_FUNCTION
> && current.getChildCount() >= 2) {
> HiveParserASTNode func = (HiveParserASTNode) current.getChild(0);
> if (func.getText().equals("grouping")) {
> visited.setValue(true);
> convertGrouping(
> current, grpByAstExprs, noneSet, legacyGrouping, found);
> }
> }
> {code}
>
> So `grouping(category)` will be converted to `grouping(0, 1)`.
> After `HiveParserCalcitePlanner#genSelectForWindowing`, it will try to
> rewrite it again:
>
> {code:java}
> if (!qbp.getDestToGroupBy().isEmpty()) {
> // Special handling of grouping function
> expr =
> rewriteGroupingFunctionAST(
> getGroupByForClause(qbp, selClauseName),
> expr,
> !cubeRollupGrpSetPresent);
> } {code}
> And it will also fall back to `convertGrouping` again as
> `current.getChildCount() >= 2` will be true. But then, it can't find any
> field
> presented in group by for it's `grouping(0, 1)` now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-28247) Exception will be thrown when over window contains grouping in Hive Dialect
[
https://issues.apache.org/jira/browse/FLINK-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-28247:
--
Assignee: luoyuxia
> Exception will be thrown when over window contains grouping in Hive Dialect
> ---
>
> Key: FLINK-28247
> URL: https://issues.apache.org/jira/browse/FLINK-28247
> Project: Flink
> Issue Type: Sub-task
>Reporter: luoyuxia
>Assignee: luoyuxia
>Priority: Major
> Labels: pull-request-available
>
> The exception will be reprodued by the following sql when using Hive Dialect:
> {code:java}
> create table t(category int, live int, comments int)
> SELECT grouping(category), lag(live) over(partition by grouping(category))
> FROM t GROUP BY category, live; {code}
> The reson is it will first call
> `HiveParserCalcitePlanner#genSelectForWindowing` to generate the window,
> which will then call `HiveParserUtils#rewriteGroupingFunctionAST` to rewrite
> the group function in the over window :
>
> {code:java}
> // rewrite grouping function
> if (current.getType() == HiveASTParser.TOK_FUNCTION
> && current.getChildCount() >= 2) {
> HiveParserASTNode func = (HiveParserASTNode) current.getChild(0);
> if (func.getText().equals("grouping")) {
> visited.setValue(true);
> convertGrouping(
> current, grpByAstExprs, noneSet, legacyGrouping, found);
> }
> }
> {code}
>
> So `grouping(category)` will be converted to `grouping(0, 1)`.
> After `HiveParserCalcitePlanner#genSelectForWindowing`, it will try to
> rewrite it again:
>
> {code:java}
> if (!qbp.getDestToGroupBy().isEmpty()) {
> // Special handling of grouping function
> expr =
> rewriteGroupingFunctionAST(
> getGroupByForClause(qbp, selClauseName),
> expr,
> !cubeRollupGrpSetPresent);
> } {code}
> And it will also fall back to `convertGrouping` again as
> `current.getChildCount() >= 2` will be true. But then, it can't find any
> field
> presented in group by for it's `grouping(0, 1)` now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-20951) IllegalArgumentException when reading Hive parquet table if condition not contain all partitioned fields
[
https://issues.apache.org/jira/browse/FLINK-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-20951.
Resolution: Duplicate
> IllegalArgumentException when reading Hive parquet table if condition not
> contain all partitioned fields
>
>
> Key: FLINK-20951
> URL: https://issues.apache.org/jira/browse/FLINK-20951
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.0
> Environment: flink 1.12.0release-12
> sql-cli
>Reporter: YUJIANBO
>Priority: Not a Priority
> Labels: auto-deprioritized-critical, auto-deprioritized-major,
> auto-deprioritized-minor, auto-unassigned
>
> The production hive table is partitioned by two fields：datekey and event
> I have do this test by Flink-sql-cli:(Spark Sql All is OK)
> （1）First:
> SELECT vid From table_A WHERE datekey = '20210112' AND event = 'XXX' AND vid
> = 'aa';(OK)
> SELECT vid From table_A WHERE datekey = '20210112' AND vid = 'aa';
> (Error)
> （2）Second:
> SELECT vid From table_B WHERE datekey = '20210112' AND event = 'YYY' AND vid
> = 'bb';(OK)
> SELECT vid From table_B WHERE datekey = '20210112' AND vid = 'bb';
> (Error)
> The exception is:
> {code}
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:273)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:395)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:609)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: SplitFetcher thread 19 received
> unexpected exception while polling the records
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.flink.hive.shaded.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes(Binary.java:424)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:79)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:33)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:199)
> at
> org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:359)
> at
> org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:328)
> at
> org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
> at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138)
> ... 6 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20951) IllegalArgumentException when reading Hive parquet table if condition not contain all partitioned fields
[
https://issues.apache.org/jira/browse/FLINK-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518606#comment-17518606
]
Rui Li commented on FLINK-20951:
Thanks [~hackergin] for verifying the fix. I'll close this as a duplicate.
> IllegalArgumentException when reading Hive parquet table if condition not
> contain all partitioned fields
>
>
> Key: FLINK-20951
> URL: https://issues.apache.org/jira/browse/FLINK-20951
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.0
> Environment: flink 1.12.0release-12
> sql-cli
>Reporter: YUJIANBO
>Priority: Not a Priority
> Labels: auto-deprioritized-critical, auto-deprioritized-major,
> auto-deprioritized-minor, auto-unassigned
>
> The production hive table is partitioned by two fields：datekey and event
> I have do this test by Flink-sql-cli:(Spark Sql All is OK)
> （1）First:
> SELECT vid From table_A WHERE datekey = '20210112' AND event = 'XXX' AND vid
> = 'aa';(OK)
> SELECT vid From table_A WHERE datekey = '20210112' AND vid = 'aa';
> (Error)
> （2）Second:
> SELECT vid From table_B WHERE datekey = '20210112' AND event = 'YYY' AND vid
> = 'bb';(OK)
> SELECT vid From table_B WHERE datekey = '20210112' AND vid = 'bb';
> (Error)
> The exception is:
> {code}
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:273)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:395)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:609)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: SplitFetcher thread 19 received
> unexpected exception while polling the records
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.flink.hive.shaded.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes(Binary.java:424)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:79)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:33)
> at
> org.apache.flink.hive.shaded.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:199)
> at
> org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:359)
> at
> org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:328)
> at
> org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
> at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138)
>
[jira] [Assigned] (FLINK-19004) Fail to call Hive percentile function together with distinct aggregate call
[
https://issues.apache.org/jira/browse/FLINK-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-19004:
--
Assignee: luoyuxia
> Fail to call Hive percentile function together with distinct aggregate call
> ---
>
> Key: FLINK-19004
> URL: https://issues.apache.org/jira/browse/FLINK-19004
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Planner
>Reporter: Rui Li
>Assignee: luoyuxia
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> The following test case would fail:
> {code}
> @Test
> public void test() throws Exception {
> TableEnvironment tableEnv = getTableEnvWithHiveCatalog();
> tableEnv.unloadModule("core");
> tableEnv.loadModule("hive", new HiveModule());
> tableEnv.loadModule("core", CoreModule.INSTANCE);
> tableEnv.executeSql("create table src(x int,y int)");
> tableEnv.executeSql("select count(distinct
> y),`percentile`(y,`array`(0.5,0.99)) from src group by x").collect();
> }
> {code}
> The error is:
> {noformat}
> org.apache.flink.table.api.TableException: Cannot generate a valid execution
> plan for the given query:
> FlinkLogicalLegacySink(name=[collect], fields=[EXPR$0, EXPR$1])
> +- FlinkLogicalCalc(select=[EXPR$0, EXPR$1])
>+- FlinkLogicalAggregate(group=[{0}], EXPR$0=[COUNT($1) FILTER $3],
> EXPR$1=[MIN($2) FILTER $4])
> +- FlinkLogicalCalc(select=[x, y, EXPR$1, =(CASE(=($e, 0:BIGINT),
> 0:BIGINT, 1:BIGINT), 0) AS $g_0, =(CASE(=($e, 0:BIGINT), 0:BIGINT, 1:BIGINT),
> 1) AS $g_1])
> +- FlinkLogicalAggregate(group=[{0, 1, 3}], EXPR$1=[percentile($4,
> $2)])
> +- FlinkLogicalExpand(projects=[x, y, $f2, $e, y_0])
>+- FlinkLogicalCalc(select=[x, y, array(0.5:DECIMAL(2, 1),
> 0.99:DECIMAL(3, 2)) AS $f2])
> +- FlinkLogicalLegacyTableSourceScan(table=[[test-catalog,
> default, src, source: [HiveTableSource(x, y) TablePath: default.src,
> PartitionPruned: false, PartitionNums: null]]], fields=[x, y])
> Min aggregate function does not support type: ''ARRAY''.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-15854) Use the new type inference for hive udtf
[ https://issues.apache.org/jira/browse/FLINK-15854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-15854: -- Assignee: luoyuxia > Use the new type inference for hive udtf > > > Key: FLINK-15854 > URL: https://issues.apache.org/jira/browse/FLINK-15854 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-unassigned, pull-request-available > > Remove HiveTableSqlFunction and use the new type inference. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467078#comment-17467078
]
Rui Li commented on FLINK-25023:
Hey [~keremulutas], when I investigated FLINK-15239, moving hadoop deps to
parent loader didn't really prevent the leak, but it mitigated the issue by
leaking only one thread for all the jobs, instead of leaking one thread for
each job submitted. Sorry my comment was misleading.
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-14588) Support Hive version 1.0.0 and 1.0.1
[ https://issues.apache.org/jira/browse/FLINK-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428794#comment-17428794 ] Rui Li commented on FLINK-14588: [~camilesing] Yes that's because we want a single hive connector binary to support multiple hive versions. And therefore we can't directly call APIs that are different among these hive versions. > Support Hive version 1.0.0 and 1.0.1 > > > Key: FLINK-14588 > URL: https://issues.apache.org/jira/browse/FLINK-14588 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23316) There's no test for custom PartitionCommitPolicy
[ https://issues.apache.org/jira/browse/FLINK-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23316. Fix Version/s: 1.15.0 Resolution: Fixed Fixed in master: 1b1d6fc2518b78e34d69c5f17fcf6b21ae0f2182 > There's no test for custom PartitionCommitPolicy > > > Key: FLINK-23316 > URL: https://issues.apache.org/jira/browse/FLINK-23316 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Rui Li >Assignee: jackwangcs >Priority: Critical > Labels: pull-request-available > Fix For: 1.15.0 > > > We support custom PartitionCommitPolicy but currently there's no test > coverage for that use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23316) There's no test for custom PartitionCommitPolicy
[ https://issues.apache.org/jira/browse/FLINK-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415498#comment-17415498 ] Rui Li commented on FLINK-23316: Hey [~jackwangcs], I think what you proposed can be some useful improvements. However, introducing/changing public APIs requires a more broad discussion, which you can drive in the dev mailing list. > There's no test for custom PartitionCommitPolicy > > > Key: FLINK-23316 > URL: https://issues.apache.org/jira/browse/FLINK-23316 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Rui Li >Assignee: jackwangcs >Priority: Critical > Labels: pull-request-available > > We support custom PartitionCommitPolicy but currently there's no test > coverage for that use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23316) There's no test for custom PartitionCommitPolicy
[ https://issues.apache.org/jira/browse/FLINK-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23316: -- Assignee: jackwangcs > There's no test for custom PartitionCommitPolicy > > > Key: FLINK-23316 > URL: https://issues.apache.org/jira/browse/FLINK-23316 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Rui Li >Assignee: jackwangcs >Priority: Critical > > We support custom PartitionCommitPolicy but currently there's no test > coverage for that use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23316) There's no test for custom PartitionCommitPolicy
[ https://issues.apache.org/jira/browse/FLINK-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412942#comment-17412942 ] Rui Li commented on FLINK-23316: [~jackwangcs] Thanks for volunteering to help. Assigning to you. > There's no test for custom PartitionCommitPolicy > > > Key: FLINK-23316 > URL: https://issues.apache.org/jira/browse/FLINK-23316 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Rui Li >Priority: Critical > > We support custom PartitionCommitPolicy but currently there's no test > coverage for that use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[
https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410952#comment-17410952
]
Rui Li commented on FLINK-23857:
OK, I tried with an empty source table w/o the {{1 != 1}} filter, and hive
doesn't clear the partition data in that case. Maybe there's some inconsistency
in hive. I guess it's more intuitive to clear the data since user intends to do
{{OVERWRITE}}. And therefore I prefer not to make a difference on this between
hive and default dialect.
> insert overwirite table select * from t where 1 != 1, Unable to clear table
> data
>
>
> Key: FLINK-23857
> URL: https://issues.apache.org/jira/browse/FLINK-23857
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: lixu
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
> Attachments: 7F086A05-FE73-47F2-BA6C-95371E89116C.png
>
>
> insert overwirite table select * from t where 1 != 1，Unable to clear table
> data，Unlike hive。
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[ https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410356#comment-17410356 ] Rui Li commented on FLINK-23857: [~luoyuxia] Please also verify with different hive versions. IIRC hive didn't clear partition data when I tried with hive-2.3.4. > insert overwirite table select * from t where 1 != 1, Unable to clear table > data > > > Key: FLINK-23857 > URL: https://issues.apache.org/jira/browse/FLINK-23857 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Connectors / Hive >Affects Versions: 1.13.1 >Reporter: lixu >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > > insert overwirite table select * from t where 1 != 1，Unable to clear table > data，Unlike hive。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[ https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407788#comment-17407788 ] Rui Li commented on FLINK-23857: bq. insert overwrite table t1 partition(f2=1) select * from t2 where 1 != 1 [~leexu] Are you able to clear partition data with this in Hive? And could you share the hive version you're using? > insert overwirite table select * from t where 1 != 1, Unable to clear table > data > > > Key: FLINK-23857 > URL: https://issues.apache.org/jira/browse/FLINK-23857 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Connectors / Hive >Affects Versions: 1.13.1 >Reporter: lixu >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > > insert overwirite table select * from t where 1 != 1，Unable to clear table > data，Unlike hive。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23715) Support for reading fields that do not exist in Parquet files
[ https://issues.apache.org/jira/browse/FLINK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23715: -- Assignee: tartarus > Support for reading fields that do not exist in Parquet files > - > > Key: FLINK-23715 > URL: https://issues.apache.org/jira/browse/FLINK-23715 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > > In a production environment, it is often encountered that users add fields to > the hive table, but do not refresh the data of the historical partition. > Therefore, if the new field is not in the historical partition file, an error > will be reported when reading the historical partition. > General users would expect that if there is no such field, then fill in null > and return. > The current flink Parquet format does not support this function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23715) Support for reading fields that do not exist in Parquet files
[ https://issues.apache.org/jira/browse/FLINK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23715. Fix Version/s: 1.15.0 Resolution: Fixed Pushed to master: 8abf8c627ca5ceab7a7e4f5b40f89b7ccceb9ceb > Support for reading fields that do not exist in Parquet files > - > > Key: FLINK-23715 > URL: https://issues.apache.org/jira/browse/FLINK-23715 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > In a production environment, it is often encountered that users add fields to > the hive table, but do not refresh the data of the historical partition. > Therefore, if the new field is not in the historical partition file, an error > will be reported when reading the historical partition. > General users would expect that if there is no such field, then fill in null > and return. > The current flink Parquet format does not support this function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[ https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23857. Fix Version/s: (was: 1.13.3) 1.15.0 Resolution: Fixed Fixed in master: 6f5995d783bc07b7bb26653555da366659255f45 Fixed in release-1.14: 2650fd63172b4d7b218187c7f65178d14fdaeb1b > insert overwirite table select * from t where 1 != 1, Unable to clear table > data > > > Key: FLINK-23857 > URL: https://issues.apache.org/jira/browse/FLINK-23857 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Connectors / Hive >Affects Versions: 1.13.1 >Reporter: lixu >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > > insert overwirite table select * from t where 1 != 1，Unable to clear table > data，Unlike hive。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23897) Fix obsolete doc about creating hive table with flink dialect
[ https://issues.apache.org/jira/browse/FLINK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23897. Resolution: Fixed Fixed in master: 5d5565523283332414fb25f4f8765a7268c11b83 Fixed in release-1.14: d3e55535417215fa47f8c1e283d88e87bc1fa508 > Fix obsolete doc about creating hive table with flink dialect > - > > Key: FLINK-23897 > URL: https://issues.apache.org/jira/browse/FLINK-23897 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23897) Fix obsolete doc about creating hive table with flink dialect
[ https://issues.apache.org/jira/browse/FLINK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23897: --- Fix Version/s: 1.15.0 1.14.0 > Fix obsolete doc about creating hive table with flink dialect > - > > Key: FLINK-23897 > URL: https://issues.apache.org/jira/browse/FLINK-23897 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23897) Fix obsolete doc about creating hive table with flink dialect
[ https://issues.apache.org/jira/browse/FLINK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23897: -- Assignee: Rui Li > Fix obsolete doc about creating hive table with flink dialect > - > > Key: FLINK-23897 > URL: https://issues.apache.org/jira/browse/FLINK-23897 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23897) Fix obsolete doc about creating hive table with flink dialect
[ https://issues.apache.org/jira/browse/FLINK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23897: --- Component/s: Documentation Connectors / Hive > Fix obsolete doc about creating hive table with flink dialect > - > > Key: FLINK-23897 > URL: https://issues.apache.org/jira/browse/FLINK-23897 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23897) Fix obsolete doc about creating hive table with flink
Rui Li created FLINK-23897: -- Summary: Fix obsolete doc about creating hive table with flink Key: FLINK-23897 URL: https://issues.apache.org/jira/browse/FLINK-23897 Project: Flink Issue Type: Bug Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23897) Fix obsolete doc about creating hive table with flink dialect
[ https://issues.apache.org/jira/browse/FLINK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23897: --- Summary: Fix obsolete doc about creating hive table with flink dialect (was: Fix obsolete doc about creating hive table with flink) > Fix obsolete doc about creating hive table with flink dialect > - > > Key: FLINK-23897 > URL: https://issues.apache.org/jira/browse/FLINK-23897 > Project: Flink > Issue Type: Bug >Reporter: Rui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[ https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23857: -- Assignee: Rui Li > insert overwirite table select * from t where 1 != 1, Unable to clear table > data > > > Key: FLINK-23857 > URL: https://issues.apache.org/jira/browse/FLINK-23857 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Connectors / Hive >Affects Versions: 1.13.1 >Reporter: lixu >Assignee: Rui Li >Priority: Major > Fix For: 1.14.0, 1.13.3 > > > insert overwirite table select * from t where 1 != 1，Unable to clear table > data，Unlike hive。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23857) insert overwirite table select * from t where 1 != 1, Unable to clear table data
[ https://issues.apache.org/jira/browse/FLINK-23857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23857: --- Component/s: (was: Table SQL / Planner) Connectors / Hive Connectors / FileSystem > insert overwirite table select * from t where 1 != 1, Unable to clear table > data > > > Key: FLINK-23857 > URL: https://issues.apache.org/jira/browse/FLINK-23857 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Connectors / Hive >Affects Versions: 1.13.1 >Reporter: lixu >Priority: Major > Fix For: 1.14.0, 1.13.3 > > > insert overwirite table select * from t where 1 != 1，Unable to clear table > data，Unlike hive。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-22914) Use Kafka new source in Table/SQL connector
[ https://issues.apache.org/jira/browse/FLINK-22914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-22914. Resolution: Fixed Pushed to master: 4c8f3a7036f5aecfba88381c50b285566bcbcbd5 > Use Kafka new source in Table/SQL connector > --- > > Key: FLINK-22914 > URL: https://issues.apache.org/jira/browse/FLINK-22914 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka, Table SQL / Ecosystem >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.14.0 > > > Currently the Kafka Table/SQL connector is still using the legacy Kafka > SourceFunction. In order to align DataStream and Table/SQL API, the new Kafka > source should also be used in Table/SQL connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22334) Fail to translate the hive-sql in STREAMING mode
[
https://issues.apache.org/jira/browse/FLINK-22334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17398466#comment-17398466
]
Rui Li commented on FLINK-22334:
bq. Is expected that cluster by isn't supported in streaming mode now?
Yes this is a known issue. Actually there's no test to run HiveQL in streaming
mode.
> Fail to translate the hive-sql in STREAMING mode
>
>
> Key: FLINK-22334
> URL: https://issues.apache.org/jira/browse/FLINK-22334
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Shengkai Fang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.14.0
>
>
> Please run in the streaming mode.
> The failed statement
> {code:java}
> // Some comments here
> insert into dest(y,x) select x,y from foo cluster by x
> {code}
> Exception stack:
> {code:java}
> org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not
> enough rules to produce a node with desired properties: convention=LOGICAL,
> FlinkRelDistributionTraitDef=any, MiniBatchIntervalTraitDef=None: 0,
> ModifyKindSetTraitDef=[NONE], UpdateKindTraitDef=[NONE].
> Missing conversion is LogicalDistribution[convention: NONE -> LOGICAL]
> There is 1 empty subset: rel#5176:RelSubset#43.LOGICAL.any.None:
> 0.[NONE].[NONE], the relevant part of the original plan is as follows
> 5174:LogicalDistribution(collation=[[0 ASC-nulls-first]], dist=[[]])
> 5172:LogicalProject(subset=[rel#5173:RelSubset#42.NONE.any.None:
> 0.[NONE].[NONE]], x=[$0])
> 5106:LogicalTableScan(subset=[rel#5171:RelSubset#41.NONE.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, foo]])
> Root: rel#5176:RelSubset#43.LOGICAL.any.None: 0.[NONE].[NONE]
> Original rel:
> FlinkLogicalLegacySink(subset=[rel#4254:RelSubset#8.LOGICAL.any.None:
> 0.[NONE].[NONE]], name=[collect], fields=[_o__c0]): rowcount = 1.0E8,
> cumulative cost = {1.0E8 rows, 1.0E8 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 4276
> FlinkLogicalCalc(subset=[rel#4275:RelSubset#7.LOGICAL.any.None:
> 0.[NONE].[NONE]], select=[CASE(IS NULL($f1), 0:BIGINT, $f1) AS _o__c0]):
> rowcount = 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 4288
> FlinkLogicalJoin(subset=[rel#4272:RelSubset#6.LOGICAL.any.None:
> 0.[NONE].[NONE]], condition=[=($0, $1)], joinType=[left]): rowcount = 1.0E8,
> cumulative cost = {1.0E8 rows, 1.0856463237676364E8 cpu, 4.0856463237676364E8
> io, 0.0 network, 0.0 memory}, id = 4271
>
> FlinkLogicalTableSourceScan(subset=[rel#4270:RelSubset#1.LOGICAL.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, bar, project=[i]]], fields=[i]):
> rowcount = 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 4.0E8 io, 0.0
> network, 0.0 memory}, id = 4279
> FlinkLogicalAggregate(subset=[rel#4268:RelSubset#5.LOGICAL.any.None:
> 0.[NONE].[NONE]], group=[{1}], agg#0=[COUNT($0)]): rowcount =
> 8564632.376763644, cumulative cost = {9.0E7 rows, 1.89E8 cpu, 7.2E8 io, 0.0
> network, 0.0 memory}, id = 4286
> FlinkLogicalCalc(subset=[rel#4283:RelSubset#3.LOGICAL.any.None:
> 0.[NONE].[NONE]], select=[x, y], where=[IS NOT NULL(y)]): rowcount = 9.0E7,
> cumulative cost = {9.0E7 rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
> = 4282
>
> FlinkLogicalTableSourceScan(subset=[rel#4262:RelSubset#2.LOGICAL.any.None:
> 0.[NONE].[NONE]], table=[[myhive, default, foo]], fields=[x, y]): rowcount =
> 1.0E8, cumulative cost = {1.0E8 rows, 1.0E8 cpu, 8.0E8 io, 0.0 network, 0.0
> memory}, id = 4261
> Sets:
> Set#41, type: RecordType(INTEGER x, INTEGER y)
> rel#5171:RelSubset#41.NONE.any.None: 0.[NONE].[NONE], best=null
> rel#5106:LogicalTableScan.NONE.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo]), rowcount=1.0E8, cumulative
> cost={inf}
> rel#5179:RelSubset#41.LOGICAL.any.None: 0.[NONE].[NONE], best=rel#5178
> rel#5178:FlinkLogicalTableSourceScan.LOGICAL.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo],fields=x, y), rowcount=1.0E8,
> cumulative cost={1.0E8 rows, 1.0E8 cpu, 8.0E8 io, 0.0 network, 0.0 memory}
> Set#42, type: RecordType(INTEGER x)
> rel#5173:RelSubset#42.NONE.any.None: 0.[NONE].[NONE], best=null
> rel#5172:LogicalProject.NONE.any.None:
> 0.[NONE].[NONE](input=RelSubset#5171,inputs=0), rowcount=1.0E8, cumulative
> cost={inf}
> rel#5180:LogicalTableScan.NONE.any.None:
> 0.[NONE].[NONE](table=[myhive, default, foo, project=[x]]), rowcount=1.0E8,
> cumulative cost={inf}
> rel#5182:LogicalCalc.NONE.any.None:
> 0.[NONE].[NONE](input=RelSubset#5171,expr#0..1={inputs},0=$t0),
> rowcount=1.0E8, cumulative cost={inf}
> rel#5184:RelSubset#42.LOGICAL.any.None: 0.[NONE].[NONE],
[jira] [Created] (FLINK-23752) Add document for using HiveSource API
Rui Li created FLINK-23752: -- Summary: Add document for using HiveSource API Key: FLINK-23752 URL: https://issues.apache.org/jira/browse/FLINK-23752 Project: Flink Issue Type: Improvement Components: Connectors / Hive, Documentation Reporter: Rui Li Fix For: 1.14.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23032) Refactor HiveSource to make it usable in data stream job
[ https://issues.apache.org/jira/browse/FLINK-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23032. Resolution: Fixed > Refactor HiveSource to make it usable in data stream job > > > Key: FLINK-23032 > URL: https://issues.apache.org/jira/browse/FLINK-23032 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23032) Refactor HiveSource to make it usable in data stream job
[ https://issues.apache.org/jira/browse/FLINK-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17398415#comment-17398415 ] Rui Li commented on FLINK-23032: Pushed to master: c4bdcac8ea255fd5a3eb6d49e349c6c5b9270939 > Refactor HiveSource to make it usable in data stream job > > > Key: FLINK-23032 > URL: https://issues.apache.org/jira/browse/FLINK-23032 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23650) A sql contains 'Case when' could not run successfully when choose Hive Dialect
[
https://issues.apache.org/jira/browse/FLINK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397194#comment-17397194
]
Rui Li commented on FLINK-23650:
This is actually a known issue of how hive dialect handles function calls, e.g.
FLINK-22383.
Ultimately, we should improve this such that w/o HiveModule, hive dialect can
fall back to flink/calcite functions, instead of generating a plan that can't
be executed.
For now, users have to load HiveModule to work around it.
> A sql contains 'Case when' could not run successfully when choose Hive Dialect
> --
>
> Key: FLINK-23650
> URL: https://issues.apache.org/jira/browse/FLINK-23650
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: JING ZHANG
>Priority: Major
>
> {code:java}
> tableEnv.sqlQuery(
> "select x,CASE WHEN x > 1 THEN 'aaa' WHEN x >1 AND x < 3 THEN 'bbb'
> ELSE 'ccc' END as y from src"){code}
> If use Default dialect, the above code could run successfully. However if use
> Hive dialect, following exception would be thrown out.
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call:
> when(BOOLEAN, STRING NOT NULL, BOOLEAN, STRING NOT NULL, STRING NOT NULL)
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call:
> when(BOOLEAN, STRING NOT NULL, BOOLEAN, STRING NOT NULL, STRING NOT NULL) If
> you think this function should be supported, you can create an issue and
> start a discussion for it.
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7$$anonfun$apply$2.apply(ExprCodeGenerator.scala:837)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7$$anonfun$apply$2.apply(ExprCodeGenerator.scala:837)
> at scala.Option.getOrElse(Option.scala:121) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7.apply(ExprCodeGenerator.scala:836)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7.apply(ExprCodeGenerator.scala:841)
> at scala.Option.getOrElse(Option.scala:121) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:829)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:503)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:58)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateExpression(ExprCodeGenerator.scala:157)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:137)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:137)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.produceProjectionCode$1(CalcCodeGenerator.scala:137)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateProcessCode(CalcCodeGenerator.scala:162)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateCalcOperator(CalcCodeGenerator.scala:48)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator.generateCalcOperator(CalcCodeGenerator.scala)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:94)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:250)
> at
> org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.java:58)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
> at
> org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$1.apply(BatchPlanner.scala:81)
> at
> org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$1.apply(BatchPlanner.scala:80)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableL
[jira] [Commented] (FLINK-23650) A sql contains 'Case when' could not run successfully when choose Hive Dialect
[
https://issues.apache.org/jira/browse/FLINK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397131#comment-17397131
]
Rui Li commented on FLINK-23650:
Thanks [~qingru zhang] for reporting the issue. I tried locally but the query
run successfully. Could you double check whether you have loaded HiveModule?
I'm able to reproduce w/o HiveModule.
> A sql contains 'Case when' could not run successfully when choose Hive Dialect
> --
>
> Key: FLINK-23650
> URL: https://issues.apache.org/jira/browse/FLINK-23650
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: JING ZHANG
>Priority: Major
>
> {code:java}
> tableEnv.sqlQuery(
> "select x,CASE WHEN x > 1 THEN 'aaa' WHEN x >1 AND x < 3 THEN 'bbb'
> ELSE 'ccc' END as y from src"){code}
> If use Default dialect, the above code could run successfully. However if use
> Hive dialect, following exception would be thrown out.
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call:
> when(BOOLEAN, STRING NOT NULL, BOOLEAN, STRING NOT NULL, STRING NOT NULL)
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call:
> when(BOOLEAN, STRING NOT NULL, BOOLEAN, STRING NOT NULL, STRING NOT NULL) If
> you think this function should be supported, you can create an issue and
> start a discussion for it.
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7$$anonfun$apply$2.apply(ExprCodeGenerator.scala:837)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7$$anonfun$apply$2.apply(ExprCodeGenerator.scala:837)
> at scala.Option.getOrElse(Option.scala:121) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7.apply(ExprCodeGenerator.scala:836)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$7.apply(ExprCodeGenerator.scala:841)
> at scala.Option.getOrElse(Option.scala:121) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:829)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:503)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:58)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateExpression(ExprCodeGenerator.scala:157)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:137)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:137)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.produceProjectionCode$1(CalcCodeGenerator.scala:137)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateProcessCode(CalcCodeGenerator.scala:162)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateCalcOperator(CalcCodeGenerator.scala:48)
> at
> org.apache.flink.table.planner.codegen.CalcCodeGenerator.generateCalcOperator(CalcCodeGenerator.scala)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:94)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:250)
> at
> org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.java:58)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134)
> at
> org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$1.apply(BatchPlanner.scala:81)
> at
> org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$1.apply(BatchPlanner.scala:80)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
> scala.collection.AbstractIterator.foreach(I
[jira] [Resolved] (FLINK-22606) Simplify the usage of SessionState
[ https://issues.apache.org/jira/browse/FLINK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-22606. Fix Version/s: 1.13.3 Resolution: Fixed Pushed to master: a5847e3871ffb9515af9c754bd10c42611976c82 Pushed to release-1.13: 00176be097fa111b24438539b29f965541a3f78b > Simplify the usage of SessionState > -- > > Key: FLINK-22606 > URL: https://issues.apache.org/jira/browse/FLINK-22606 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > > Hive SessionState is used by hive dialect. Starting a SessionState involves > some heavy operations like creating scratch folders and instantiating an HMS > client. We should investigate how to reduce these operations. > It's of course better to completely get rid of SessionState. But that's > difficult to achieve because some hive functions rely on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22246) when use HiveCatalog create table , can't set Table owner property correctly
[
https://issues.apache.org/jira/browse/FLINK-22246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394611#comment-17394611
]
Rui Li commented on FLINK-22246:
[~luoyuxia] Creating Hive {{SessionState}} involves heavy and unexpected
operations such as creating temp folders or changing thread context class
loader. It already causes us some problems in {{HiveParser}}. So we should
avoid it whenever possible.
[~Bo Cui] I think we can just set the owner to UGI current user. No need to get
the real user in case of a proxy, because that's why we create the proxy user
in the first place, right?
> when use HiveCatalog create table , can't set Table owner property correctly
> -
>
> Key: FLINK-22246
> URL: https://issues.apache.org/jira/browse/FLINK-22246
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.1, 1.12.0
>Reporter: xiangtao
>Priority: Minor
> Labels: auto-deprioritized-major
>
> when i use HiveCatalog create table in sql-client , i found it can't set
> Hive Table `owner` property correctly. i debug code , i found in
> `HiveCatalog.createTable` method
> {code:java}
> Table hiveTable =
> org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
> tablePath.getDatabaseName(), tablePath.getObjectName());
> {code}
>
> this get hiveTable obj , owner field is null . beacuse it set owner through
> {code:java}
> t.setOwner(SessionState.getUserFromAuthenticator());
> {code}
>
> but SessionState is null .
> Fix this bug , we can add one code in HiveCatalog.open method .
> {code:java}
> SessionState.setCurrentSessionState(new SessionState(hiveConf));
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23567) Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was not found
[ https://issues.apache.org/jira/browse/FLINK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391910#comment-17391910 ] Rui Li commented on FLINK-23567: [~wuyang09] I have pushed a fix for this issue. Please try it out and feel free to reopen this ticket if the issue persists. > Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was > not found > -- > > Key: FLINK-23567 > URL: https://issues.apache.org/jira/browse/FLINK-23567 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1 >Reporter: wuyang >Assignee: Rui Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > Attachments: image-2021-07-31-10-39-52-126.png, > image-2021-07-31-10-40-07-070.png, image-2021-08-02-13-55-26-467.png, > image-2021-08-02-14-00-26-096.png, image-2021-08-02-14-24-36-210.png > > > *First：I added the flink-sql-connector-hive-1.2.2_2.11-1.13.1.jar under the > Lib directory, the following error is prompted when publishing the task of > Flink SQL：* > java.lang.NoClassDefFoundError: org/json/JSONException > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeCreateTable(HiveParserDDLSemanticAnalyzer.java:646) > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeInternal(HiveParserDDLSemanticAnalyzer.java:373) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:235) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callSql(SqlSubmitHelper.java:201) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callCommand(SqlSubmitHelper.java:182) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.run(SqlSubmitHelper.java:124) > at me.ddmc.bigdata.sqlsubmit.SqlSubmit.main(SqlSubmit.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.lang.ClassNotFoundException: org.json.JSONException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 25 > > *Second: After investigation, it is found that the exclude is added to the > POM in the flink-sql-connector-hive-1.2.2 module, but other hive connectors > are not.* > !image-2021-07-31-10-40-07-070.png! > *But I didn't understand this remark. Is this a problem？* > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23567) Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was not found
[ https://issues.apache.org/jira/browse/FLINK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23567. Fix Version/s: 1.13.3 1.14.0 Resolution: Fixed Fixed in master: 48cd889b1de39080ad883da982287faf904bc520 Fixed in release-1.13: ef752202a32ee4c308d37d28e0163c9c7bec4fc6 > Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was > not found > -- > > Key: FLINK-23567 > URL: https://issues.apache.org/jira/browse/FLINK-23567 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1 >Reporter: wuyang >Assignee: Rui Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > Attachments: image-2021-07-31-10-39-52-126.png, > image-2021-07-31-10-40-07-070.png, image-2021-08-02-13-55-26-467.png, > image-2021-08-02-14-00-26-096.png, image-2021-08-02-14-24-36-210.png > > > *First：I added the flink-sql-connector-hive-1.2.2_2.11-1.13.1.jar under the > Lib directory, the following error is prompted when publishing the task of > Flink SQL：* > java.lang.NoClassDefFoundError: org/json/JSONException > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeCreateTable(HiveParserDDLSemanticAnalyzer.java:646) > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeInternal(HiveParserDDLSemanticAnalyzer.java:373) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:235) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callSql(SqlSubmitHelper.java:201) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callCommand(SqlSubmitHelper.java:182) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.run(SqlSubmitHelper.java:124) > at me.ddmc.bigdata.sqlsubmit.SqlSubmit.main(SqlSubmit.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.lang.ClassNotFoundException: org.json.JSONException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 25 > > *Second: After investigation, it is found that the exclude is added to the > POM in the flink-sql-connector-hive-1.2.2 module, but other hive connectors > are not.* > !image-2021-07-31-10-40-07-070.png! > *But I didn't understand this remark. Is this a problem？* > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23567) Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was not found
[ https://issues.apache.org/jira/browse/FLINK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23567: -- Assignee: Rui Li > Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was > not found > -- > > Key: FLINK-23567 > URL: https://issues.apache.org/jira/browse/FLINK-23567 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1 >Reporter: wuyang >Assignee: Rui Li >Priority: Blocker > Attachments: image-2021-07-31-10-39-52-126.png, > image-2021-07-31-10-40-07-070.png, image-2021-08-02-13-55-26-467.png, > image-2021-08-02-14-00-26-096.png, image-2021-08-02-14-24-36-210.png > > > *First：I added the flink-sql-connector-hive-1.2.2_2.11-1.13.1.jar under the > Lib directory, the following error is prompted when publishing the task of > Flink SQL：* > java.lang.NoClassDefFoundError: org/json/JSONException > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeCreateTable(HiveParserDDLSemanticAnalyzer.java:646) > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeInternal(HiveParserDDLSemanticAnalyzer.java:373) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:235) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callSql(SqlSubmitHelper.java:201) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callCommand(SqlSubmitHelper.java:182) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.run(SqlSubmitHelper.java:124) > at me.ddmc.bigdata.sqlsubmit.SqlSubmit.main(SqlSubmit.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.lang.ClassNotFoundException: org.json.JSONException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 25 > > *Second: After investigation, it is found that the exclude is added to the > POM in the flink-sql-connector-hive-1.2.2 module, but other hive connectors > are not.* > !image-2021-07-31-10-40-07-070.png! > *But I didn't understand this remark. Is this a problem？* > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23567) Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was not found
[ https://issues.apache.org/jira/browse/FLINK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23567: --- Fix Version/s: (was: 1.13.1) > Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was > not found > -- > > Key: FLINK-23567 > URL: https://issues.apache.org/jira/browse/FLINK-23567 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1 >Reporter: wuyang >Priority: Blocker > Attachments: image-2021-07-31-10-39-52-126.png, > image-2021-07-31-10-40-07-070.png, image-2021-08-02-13-55-26-467.png, > image-2021-08-02-14-00-26-096.png > > > *First：I added the flink-sql-connector-hive-3.1.2_2.11-1.13.1.jar under the > Lib directory, the following error is prompted when publishing the task of > Flink SQL：* > java.lang.NoClassDefFoundError: org/json/JSONException > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeCreateTable(HiveParserDDLSemanticAnalyzer.java:646) > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeInternal(HiveParserDDLSemanticAnalyzer.java:373) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:235) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callSql(SqlSubmitHelper.java:201) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callCommand(SqlSubmitHelper.java:182) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.run(SqlSubmitHelper.java:124) > at me.ddmc.bigdata.sqlsubmit.SqlSubmit.main(SqlSubmit.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.lang.ClassNotFoundException: org.json.JSONException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 25 > > *Second: After investigation, it is found that the exclude is added to the > POM in the flink-sql-connector-hive-1.2.2 module, but other hive connectors > are not.* > !image-2021-07-31-10-40-07-070.png! > *But I didn't understand this remark. Is this a problem？* > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23567) Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was not found
[ https://issues.apache.org/jira/browse/FLINK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391274#comment-17391274 ] Rui Li commented on FLINK-23567: [~wuyang09] The filter is there because the JSON dependency's license doesn't allow us to include it in our distribution. Could you share the DDL that triggers this issue? > Hive 1.1.0 failed to write using flink sql 1.13.1 because the JSON class was > not found > -- > > Key: FLINK-23567 > URL: https://issues.apache.org/jira/browse/FLINK-23567 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1 >Reporter: wuyang >Priority: Blocker > Fix For: 1.13.1 > > Attachments: image-2021-07-31-10-39-52-126.png, > image-2021-07-31-10-40-07-070.png > > > *First：I added the flink-sql-connector-hive-3.1.2_2.11-1.13.1.jar under the > Lib directory, the following error is prompted when publishing the task of > Flink SQL：* > java.lang.NoClassDefFoundError: org/json/JSONException > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeCreateTable(HiveParserDDLSemanticAnalyzer.java:646) > at > org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.analyzeInternal(HiveParserDDLSemanticAnalyzer.java:373) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:235) > at > org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:217) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callSql(SqlSubmitHelper.java:201) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.callCommand(SqlSubmitHelper.java:182) > at > me.ddmc.bigdata.sqlsubmit.helper.SqlSubmitHelper.run(SqlSubmitHelper.java:124) > at me.ddmc.bigdata.sqlsubmit.SqlSubmit.main(SqlSubmit.java:34) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.lang.ClassNotFoundException: org.json.JSONException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 25 > > *Second: After investigation, it is found that the exclude is added to the > POM in the flink-sql-connector-hive-1.2.2 module, but other hive connectors > are not.* > !image-2021-07-31-10-40-07-070.png! > *But I didn't understand this remark. Is this a problem？* > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-23239.
Resolution: Fixed
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390336#comment-17390336
]
Rui Li commented on FLINK-23239:
[~liyu] Thanks for the reminder. I'm closing this one.
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated FLINK-23239:
---
Fix Version/s: 1.13.3
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389912#comment-17389912
]
Rui Li commented on FLINK-23239:
Fixed in master: 65f3197cf055630ec35dc660e7ca99364eb50dde
Fixed in release-1.13: e4d62ba2b8a716afdd32819b9fbe650ce1faa4dd
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-22657) HiveParserDDLSemanticAnalyzer can directly return operations
[ https://issues.apache.org/jira/browse/FLINK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-22657. Fix Version/s: 1.13.3 Resolution: Fixed Pushed to master: c3088af32543c807a69eced7129f89284204c2af Pushed to release-1.13: b3f3b6b5780d3e9d668d3cd92e0dfd5b1950d1c8 > HiveParserDDLSemanticAnalyzer can directly return operations > > > Key: FLINK-22657 > URL: https://issues.apache.org/jira/browse/FLINK-22657 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: luoyuxia >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > > There's no need to first generate some "desc" and later convert to operation > with DDLOperationConverter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-23239:
--
Assignee: Rui Li
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Rui Li
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23239) HiveTableSinkITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17387344#comment-17387344
]
Rui Li commented on FLINK-23239:
I examined the test log and guess I've found the cause. We write two records
into each partition of {{source_table}}, and each of these two records can
trigger partition commit in {{sink_table}}. So if a CP is done in between of
these two records, the partition will be committed with only one record. And if
the reading job checks {{sink_table}} for new partitions before the 2nd record
is written, it only finds the 1st record but believes the partition has all the
data. Therefore we'll hang at the verification code waiting for the 2nd record,
which is now a late event.
> HiveTableSinkITCase hangs on azure
> --
>
> Key: FLINK-23239
> URL: https://issues.apache.org/jira/browse/FLINK-23239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19872&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=420bd9ec-164e-562e-8947-0dacde3cec91&l=23845
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fac9000b800 nid=0x619b waiting on
> condition [0x7fac98621000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.fetchRows(HiveTableSinkITCase.java:384)
> at
> org.apache.flink.connectors.hive.HiveTableSinkITCase.testStreamingSinkWithTimestampLtzWatermark(HiveTableSinkITCase.java:360)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInPro
[jira] [Commented] (FLINK-20975) HiveTableSourceITCase.testPartitionFilter fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383347#comment-17383347
]
Rui Li commented on FLINK-20975:
Pushed a fix to:
* master: a846a4856549a0992f2172fec9d1c3abe2d46215
* release-1.13: a6b823498aa77e3de5ff34e191e3a2e3d42f3620
* release-1.12: da62d0dcd12e915507b783a94a177907d441c761
Let's keep this open for a while.
> HiveTableSourceITCase.testPartitionFilter fails on AZP
> --
>
> Key: FLINK-20975
> URL: https://issues.apache.org/jira/browse/FLINK-20975
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Rui Li
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> pull-request-available, test-stability
> Fix For: 1.13.2
>
>
> The test {{HiveTableSourceITCase.testPartitionFilter}} fails on AZP with the
> following exception:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testPartitionFilter(HiveTableSourceITCase.java:278)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-22898. Resolution: Fixed > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0, 1.12.6, 1.13.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381294#comment-17381294 ] Rui Li commented on FLINK-22898: Fixed in release-1.13: 004d3b58e4f4637efb09e3c7c84f01c3bd64ad9f Fixed in release-1.12: 344f6e42cdcdcc4d9d98fe414016138cdd748bf3 > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-22898: --- Fix Version/s: 1.13.3 1.12.6 > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0, 1.12.6, 1.13.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20975) HiveTableSourceITCase.testPartitionFilter fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-20975:
--
Assignee: Rui Li
> HiveTableSourceITCase.testPartitionFilter fails on AZP
> --
>
> Key: FLINK-20975
> URL: https://issues.apache.org/jira/browse/FLINK-20975
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Rui Li
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.13.2
>
>
> The test {{HiveTableSourceITCase.testPartitionFilter}} fails on AZP with the
> following exception:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testPartitionFilter(HiveTableSourceITCase.java:278)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20975) HiveTableSourceITCase.testPartitionFilter fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379174#comment-17379174
]
Rui Li commented on FLINK-20975:
I tried the test locally dozens of times but didn't reproduce it. According to
test logs, the test failed because of:
1. Direct SQL failed due to:
{noformat}
Caused by: org.apache.derby.iapi.error.StandardException: Invalid character
string format for type DECIMAL.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at org.apache.derby.iapi.types.DataType.invalidFormat(Unknown Source)
~[derby-10.10.2.0.jar:?]
at org.apache.derby.iapi.types.DataType.setValue(Unknown Source)
~[derby-10.10.2.0.jar:?]
at
org.apache.derby.exe.ac1d3a42f2x017ax8d9bxd637x338c7ac889.e4(Unknown
Source) ~[?:?]
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at
org.apache.derby.impl.sql.execute.NestedLoopJoinResultSet.getNextRowCore(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at
org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
Source) ~[derby-10.10.2.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
~[derby-10.10.2.0.jar:?]
at
org.datanucleus.store.rdbms.query.ForwardQueryResult.initialise(ForwardQueryResult.java:99)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:676)
~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.query.Query.executeQuery(Query.java:1855)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.store.rdbms.query.SQLQuery.executeWithArray(SQLQuery.java:807)
~[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
~[datanucleus-api-jdo-4.2.4.jar:?]
... 121 more
{noformat}
Since the partition column type is INT and should be safe to cast to decimal,
this might be a derby bug.
2. Then hive tried to fallback to JDO method to pushdown the filter. But failed
again because by default "Filtering is supported only on partition keys of type
string".
3. Finally the pushdown is aborted and we fallback to listing all partitions in
metastore, and thus the test failure.
I guess we can enable filter pushdown on integral partition column types so
that we'll have a 2nd chance in step #2
> HiveTableSourceITCase.testPartitionFilter fails on AZP
> --
>
> Key: FLINK-20975
> URL: https://issues.apache.org/jira/browse/FLINK-20975
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.13.2
>
>
> The test {{HiveTableSourceITCase.testPartitionFilter}} fails on AZP with the
> following exception:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testPartitionFilter(HiveTableSourceITCase.java:278)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChi
[jira] [Commented] (FLINK-20975) HiveTableSourceITCase.testPartitionFilter fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378892#comment-17378892
]
Rui Li commented on FLINK-20975:
Sure I'll look into this.
> HiveTableSourceITCase.testPartitionFilter fails on AZP
> --
>
> Key: FLINK-20975
> URL: https://issues.apache.org/jira/browse/FLINK-20975
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
> Fix For: 1.13.2
>
>
> The test {{HiveTableSourceITCase.testPartitionFilter}} fails on AZP with the
> following exception:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testPartitionFilter(HiveTableSourceITCase.java:278)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-13400) Remove Hive and Hadoop dependencies from SQL Client
[
https://issues.apache.org/jira/browse/FLINK-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-13400:
--
Assignee: frank wang
> Remove Hive and Hadoop dependencies from SQL Client
> ---
>
> Key: FLINK-13400
> URL: https://issues.apache.org/jira/browse/FLINK-13400
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Timo Walther
>Assignee: frank wang
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned, stale-major
> Fix For: 1.14.0
>
>
> 340/550 lines in the SQL Client {{pom.xml}} are just around Hive and Hadoop
> dependencies. Hive has nothing to do with the SQL Client and it will be hard
> to maintain the long list of exclusion there. Some dependencies are even in
> a {{provided}} scope and not {{test}} scope.
> We should remove all dependencies on Hive/Hadoop and replace catalog-related
> tests by a testing catalog. Similar to how we tests source/sinks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13400) Remove Hive and Hadoop dependencies from SQL Client
[
https://issues.apache.org/jira/browse/FLINK-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377965#comment-17377965
]
Rui Li commented on FLINK-13400:
[~frank wang] Thanks for offering to help with this. Assigning to you.
> Remove Hive and Hadoop dependencies from SQL Client
> ---
>
> Key: FLINK-13400
> URL: https://issues.apache.org/jira/browse/FLINK-13400
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Timo Walther
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned, stale-major
> Fix For: 1.14.0
>
>
> 340/550 lines in the SQL Client {{pom.xml}} are just around Hive and Hadoop
> dependencies. Hive has nothing to do with the SQL Client and it will be hard
> to maintain the long list of exclusion there. Some dependencies are even in
> a {{provided}} scope and not {{test}} scope.
> We should remove all dependencies on Hive/Hadoop and replace catalog-related
> tests by a testing catalog. Similar to how we tests source/sinks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[ https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23074: --- Fix Version/s: 1.13.2 1.12.5 > There is a class conflict between flink-connector-hive and flink-parquet > > > Key: FLINK-23074 > URL: https://issues.apache.org/jira/browse/FLINK-23074 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1, 1.12.4 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.12.5, 1.13.2 > > Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png, > image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png, > image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png, > screenshot-1.png, screenshot-3.png, screenshot-4.png > > > flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but > flink-parquet include 1.11.1. > org.apache.parquet.hadoop.example.GroupWriteSupport > is different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[ https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377786#comment-17377786 ] Rui Li commented on FLINK-23074: Fixed in release-1.13: 8359fa8a9149392d964f52e7492b4dc24d74bb15 Fixed in release-1.12: 09ac6e50da595b080878d1f18815d2a9fb251f13 > There is a class conflict between flink-connector-hive and flink-parquet > > > Key: FLINK-23074 > URL: https://issues.apache.org/jira/browse/FLINK-23074 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1, 1.12.4 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png, > image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png, > image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png, > screenshot-1.png, screenshot-3.png, screenshot-4.png > > > flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but > flink-parquet include 1.11.1. > org.apache.parquet.hadoop.example.GroupWriteSupport > is different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22402) Add hive dialect test in sql-client
[ https://issues.apache.org/jira/browse/FLINK-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377369#comment-17377369 ] Rui Li commented on FLINK-22402: Hey [~frank wang], I don't think anyone is working on this. But there's plan [1] to move the hive tests from sql client to a dedicated e2e test module. So I think it's better to add these tests after that refactor is done. Or let me know if you're interested in helping with the refactor. [1] FLINK-13400 > Add hive dialect test in sql-client > --- > > Key: FLINK-22402 > URL: https://issues.apache.org/jira/browse/FLINK-22402 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Tests >Reporter: Rui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23316) There's no test for custom PartitionCommitPolicy
[ https://issues.apache.org/jira/browse/FLINK-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23316: --- Description: We support custom PartitionCommitPolicy but currently there's no test coverage for that use case. > There's no test for custom PartitionCommitPolicy > > > Key: FLINK-23316 > URL: https://issues.apache.org/jira/browse/FLINK-23316 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Rui Li >Priority: Critical > > We support custom PartitionCommitPolicy but currently there's no test > coverage for that use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23316) There's no test for custom PartitionCommitPolicy
Rui Li created FLINK-23316: -- Summary: There's no test for custom PartitionCommitPolicy Key: FLINK-23316 URL: https://issues.apache.org/jira/browse/FLINK-23316 Project: Flink Issue Type: Improvement Components: Table SQL / Ecosystem Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23178) Raise an error for writing stream data into partitioned hive tables without a partition committer
[ https://issues.apache.org/jira/browse/FLINK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23178. Fix Version/s: 1.13.2 1.14.0 Resolution: Fixed Pushed to master: a94745ec85bf9e8ca3bc2fced5c1a466b836e0be Pushed to release-1.13: f14307b869e1e9b518784276311ce6afb112312d > Raise an error for writing stream data into partitioned hive tables without a > partition committer > - > > Key: FLINK-23178 > URL: https://issues.apache.org/jira/browse/FLINK-23178 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.2 > > > If a user writes streaming data into hive but forgets to specify a partition > commit policy, the job will run successfully but data was never committed to > hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23298) [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
[ https://issues.apache.org/jira/browse/FLINK-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23298. Fix Version/s: 1.14.0 Resolution: Fixed Fixed in master: e9cc49a31fc384bff1c84372788f14a173cd94b0 > [datagen] Normalize parameter names in RandomGeneratorVisitor and > SequenceGeneratorVisitor > -- > > Key: FLINK-23298 > URL: https://issues.apache.org/jira/browse/FLINK-23298 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Ecosystem >Reporter: fhan >Assignee: fhan >Priority: Minor > Labels: pull-request-available > Fix For: 1.14.0 > > > This PR normalized parameter names in RandomGeneratorVisitor and > SequenceGeneratorVisitor. > related methods: > [!https://user-images.githubusercontent.com/5745228/118935994-b47c6080-b97e-11eb-86ef-43c191f602fd.jpg|width=416,height=287!|https://user-images.githubusercontent.com/5745228/118935994-b47c6080-b97e-11eb-86ef-43c191f602fd.jpg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23298) [datagen] Normalize parameter names in RandomGeneratorVisitor and SequenceGeneratorVisitor
[ https://issues.apache.org/jira/browse/FLINK-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23298: -- Assignee: fhan > [datagen] Normalize parameter names in RandomGeneratorVisitor and > SequenceGeneratorVisitor > -- > > Key: FLINK-23298 > URL: https://issues.apache.org/jira/browse/FLINK-23298 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Ecosystem >Reporter: fhan >Assignee: fhan >Priority: Minor > Labels: pull-request-available > > This PR normalized parameter names in RandomGeneratorVisitor and > SequenceGeneratorVisitor. > related methods: > [!https://user-images.githubusercontent.com/5745228/118935994-b47c6080-b97e-11eb-86ef-43c191f602fd.jpg|width=416,height=287!|https://user-images.githubusercontent.com/5745228/118935994-b47c6080-b97e-11eb-86ef-43c191f602fd.jpg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23032) Refactor HiveSource to make it usable in data stream job
[ https://issues.apache.org/jira/browse/FLINK-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376309#comment-17376309 ] Rui Li commented on FLINK-23032: Hi [~frank wang], thanks for your interests in this! However I'm already working on it. You can search for unassigned tickets to make more contributions. > Refactor HiveSource to make it usable in data stream job > > > Key: FLINK-23032 > URL: https://issues.apache.org/jira/browse/FLINK-23032 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-22657) HiveParserDDLSemanticAnalyzer can directly return operations
[ https://issues.apache.org/jira/browse/FLINK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-22657: -- Assignee: luoyuxia > HiveParserDDLSemanticAnalyzer can directly return operations > > > Key: FLINK-22657 > URL: https://issues.apache.org/jira/browse/FLINK-22657 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: luoyuxia >Priority: Major > > There's no need to first generate some "desc" and later convert to operation > with DDLOperationConverter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23032) Refactor HiveSource to make it usable in data stream job
[ https://issues.apache.org/jira/browse/FLINK-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23032: --- Labels: (was: stale-assigned) > Refactor HiveSource to make it usable in data stream job > > > Key: FLINK-23032 > URL: https://issues.apache.org/jira/browse/FLINK-23032 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[ https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375674#comment-17375674 ] Rui Li commented on FLINK-23074: [~ana4] If you want this fix in release 1.13 or 1.12, please open PRs for those branches respectively. > There is a class conflict between flink-connector-hive and flink-parquet > > > Key: FLINK-23074 > URL: https://issues.apache.org/jira/browse/FLINK-23074 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1, 1.12.4 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png, > image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png, > image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png, > screenshot-1.png, screenshot-3.png, screenshot-4.png > > > flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but > flink-parquet include 1.11.1. > org.apache.parquet.hadoop.example.GroupWriteSupport > is different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[ https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23074. Fix Version/s: 1.14.0 Resolution: Fixed Fixed in master: 78bafc55363d11e33143f1769248b85eaec2ee5a > There is a class conflict between flink-connector-hive and flink-parquet > > > Key: FLINK-23074 > URL: https://issues.apache.org/jira/browse/FLINK-23074 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.1, 1.12.4 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png, > image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png, > image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png, > screenshot-1.png, screenshot-3.png, screenshot-4.png > > > flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but > flink-parquet include 1.11.1. > org.apache.parquet.hadoop.example.GroupWriteSupport > is different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375292#comment-17375292 ] Rui Li commented on FLINK-22898: Fixed in master: a765da5dc9bc7f56971d1714abc8bce4796f9610 [~luoyuxia] Please open PRs for release-1.13 and release-1.12. A simple cherry-pick doesn't compile. > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-22884) Select view columns fail when store metadata with hive
[
https://issues.apache.org/jira/browse/FLINK-22884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-22884.
Fix Version/s: 1.13.3
1.14.0
Resolution: Fixed
Fixed in master: 0b686824c33ad976cefb588c5a272a09de4b23ca
Fixed in release-1.13: 28b09e575bd174685a15d93345d0f87a37d6f002
> Select view columns fail when store metadata with hive
> --
>
> Key: FLINK-22884
> URL: https://issues.apache.org/jira/browse/FLINK-22884
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: ELLEX_SHEN
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.14.0, 1.13.3
>
>
> I am use hive for filnk metadata, so select view table will mismatch to hive
> table after create view, I founded is a bug in HiveCatalog.classs, all view
> table is default mark to hive table unexpected.
> after store in hive metadata, view table without "is_generic" or "connector"
> properties.
> bug is here:
> @VisibleForTesting
> public Table getHiveTable(ObjectPath tablePath) throws
> TableNotExistException {
> try {
> Table table = this.client.getTable(tablePath.getDatabaseName(),
> tablePath.getObjectName());
> boolean isHiveTable;
> if (table.getParameters().containsKey("is_generic")) {
> isHiveTable =
> !Boolean.parseBoolean((String)table.getParameters().remove("is_generic"));
> } else {
> isHiveTable = !table.getParameters().containsKey("flink." +
> FactoryUtil.CONNECTOR.key()) &&
> !table.getParameters().containsKey("flink.connector.type");
> }
> if (isHiveTable) {
> table.getParameters().put(FactoryUtil.CONNECTOR.key(),
> "hive");
> }
> return table;
> } catch (NoSuchObjectException var4) {
> throw new TableNotExistException(this.getName(), tablePath);
> } catch (TException var5) {
> throw new CatalogException(String.format("Failed to get table %s
> from Hive metastore", tablePath.getFullName()), var5);
> }
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[
https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375141#comment-17375141
]
Rui Li commented on FLINK-23074:
How about just {{org.apache.hive.shaded.parquet}}, as we're relocating parquet
classes in hive-exec?
> There is a class conflict between flink-connector-hive and flink-parquet
>
>
> Key: FLINK-23074
> URL: https://issues.apache.org/jira/browse/FLINK-23074
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1, 1.12.4
>Reporter: Ada Wong
>Assignee: Ada Wong
>Priority: Major
> Labels: pull-request-available
> Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png,
> image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png,
> image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png,
> screenshot-1.png, screenshot-3.png, screenshot-4.png
>
>
> flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but
> flink-parquet include 1.11.1.
> org.apache.parquet.hadoop.example.GroupWriteSupport
> is different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23178) Raise an error for writing stream data into partitioned hive tables without a partition committer
[ https://issues.apache.org/jira/browse/FLINK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23178: --- Component/s: (was: Connectors / FileSystem) > Raise an error for writing stream data into partitioned hive tables without a > partition committer > - > > Key: FLINK-23178 > URL: https://issues.apache.org/jira/browse/FLINK-23178 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > > If a user writes streaming data into hive but forgets to specify a partition > commit policy, the job will run successfully but data was never committed to > hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23178) Raise an error for writing stream data into partitioned hive tables without a partition committer
[ https://issues.apache.org/jira/browse/FLINK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23178: --- Summary: Raise an error for writing stream data into partitioned hive tables without a partition committer (was: Raise an error for writing stream data into partitioned hive/FS tables without a partition committer) > Raise an error for writing stream data into partitioned hive tables without a > partition committer > - > > Key: FLINK-23178 > URL: https://issues.apache.org/jira/browse/FLINK-23178 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > > If a user writes streaming data into hive but forgets to specify a partition > commit policy, the job will run successfully but data was never committed to > hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23178) Raise an error for writing stream data into partitioned hive/FS tables without a partition committer
[ https://issues.apache.org/jira/browse/FLINK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23178: -- Assignee: Rui Li > Raise an error for writing stream data into partitioned hive/FS tables > without a partition committer > > > Key: FLINK-23178 > URL: https://issues.apache.org/jira/browse/FLINK-23178 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > > If a user writes streaming data into hive but forgets to specify a partition > commit policy, the job will run successfully but data was never committed to > hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23074) There is a class conflict between flink-connector-hive and flink-parquet
[
https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17374459#comment-17374459
]
Rui Li commented on FLINK-23074:
I think we can relocate parquet in sql-connector-hive jars, but the shaded
pattern should be different from {{org.apache.flink.hive.shaded.parquet}}
otherwise it overrides the classes in connector-hive jar.
> There is a class conflict between flink-connector-hive and flink-parquet
>
>
> Key: FLINK-23074
> URL: https://issues.apache.org/jira/browse/FLINK-23074
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1, 1.12.4
>Reporter: Ada Wong
>Assignee: Ada Wong
>Priority: Major
> Labels: pull-request-available
> Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png,
> image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png,
> image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png,
> screenshot-1.png, screenshot-3.png, screenshot-4.png
>
>
> flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but
> flink-parquet include 1.11.1.
> org.apache.parquet.hadoop.example.GroupWriteSupport
> is different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23175) Write data to hive table in batch mode throws FileNotFoundException.
[
https://issues.apache.org/jira/browse/FLINK-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373516#comment-17373516
]
Rui Li commented on FLINK-23175:
[~chrismartin] The staging dir is created on client side and registered as
"delete on exit". So if your client exits after submitting the job, the dir is
deleted and the job will fail. Could you check if this is the cause of your
problem?
> Write data to hive table in batch mode throws FileNotFoundException.
>
>
> Key: FLINK-23175
> URL: https://issues.apache.org/jira/browse/FLINK-23175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Environment: flink 1.12.1 standalone cluster
>Reporter: chrismartin
>Priority: Major
>
> I wanna luanch a batch job to process hive table data and write the result to
> another table(*T1*), and my SQL statements is wriiten like below:
> -- use hive dialectSET table.sql-dialect=hive;-- insert into hive tableinsert
> overwrite table hivetablepartition (dt='20210626')select * from
> pgcatalog.pgschema.table
> The job was success luanched, but it failed on *Sink* operator. On Flink UI
> page I saw all task state is `*FINISHED*`, but *the job failed and it
> restarted again*.
> And I found exception information like below: (*The path was marksed*)
> {code:java}
> //代码占位符
> java.lang.Exception: Failed to finalize execution on master
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1291)
> at
> org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:870)
> at
> org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1125)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1491)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1464)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:497)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.table.api.TableException: Exception in
> finalizeGlobal
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
> ... 31 more
> Caused by: java.io.FileNotFoundException: File
> //XX/XXX/.staging_1621244168369 does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(Distr
[jira] [Created] (FLINK-23224) Support timestamp_ltz type in hive connector
Rui Li created FLINK-23224: -- Summary: Support timestamp_ltz type in hive connector Key: FLINK-23224 URL: https://issues.apache.org/jira/browse/FLINK-23224 Project: Flink Issue Type: New Feature Components: Connectors / Hive Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23168) Catalog shouldn't merge properties for alter DB operation
[ https://issues.apache.org/jira/browse/FLINK-23168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373501#comment-17373501 ] Rui Li edited comment on FLINK-23168 at 7/2/21, 12:54 PM: -- Fixed in master: * 37cd345c729016379a84b6e8083186561e3beb76 * 39acdb762295f6b744c0af0e0449faaef8e73652 Fixed in release-1.13: * d0bf474cec5cc079172160ba070070de1a14e59c * 2075d0facdcfbac9f3f060e8b274dd485f365829 was (Author: lirui): Fixed in master: * 37cd345c729016379a84b6e8083186561e3beb76 * 39acdb762295f6b744c0af0e0449faaef8e73652 Fixed in release-1.13: * d0bf474cec5cc079172160ba070070de1a14e59c * 2075d0facdcfbac9f3f060e8b274dd485f365829 > Catalog shouldn't merge properties for alter DB operation > - > > Key: FLINK-23168 > URL: https://issues.apache.org/jira/browse/FLINK-23168 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / API >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23168) Catalog shouldn't merge properties for alter DB operation
[ https://issues.apache.org/jira/browse/FLINK-23168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23168. Fix Version/s: 1.13.3 1.14.0 Resolution: Fixed Fixed in master: * 37cd345c729016379a84b6e8083186561e3beb76 * 39acdb762295f6b744c0af0e0449faaef8e73652 Fixed in release-1.13: * d0bf474cec5cc079172160ba070070de1a14e59c * 2075d0facdcfbac9f3f060e8b274dd485f365829 > Catalog shouldn't merge properties for alter DB operation > - > > Key: FLINK-23168 > URL: https://issues.apache.org/jira/browse/FLINK-23168 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / API >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23168) Catalog shouldn't merge properties for alter DB operation
[ https://issues.apache.org/jira/browse/FLINK-23168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23168: --- Summary: Catalog shouldn't merge properties for alter DB operation (was: GenericInMemoryCatalog shouldn't merge properties for alter DB operation) > Catalog shouldn't merge properties for alter DB operation > - > > Key: FLINK-23168 > URL: https://issues.apache.org/jira/browse/FLINK-23168 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / API >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-22898) HiveParallelismInference limit return wrong parallelism
[ https://issues.apache.org/jira/browse/FLINK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-22898: -- Assignee: luoyuxia > HiveParallelismInference limit return wrong parallelism > --- > > Key: FLINK-22898 > URL: https://issues.apache.org/jira/browse/FLINK-22898 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Reporter: Jingsong Lee >Assignee: luoyuxia >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23178) Raise an error for writing stream data into partitioned hive/FS tables without a partition committer
[ https://issues.apache.org/jira/browse/FLINK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated FLINK-23178: --- Description: If a user writes streaming data into hive but forgets to specify a partition commit policy, the job will run successfully but data was never committed to hive metastore. > Raise an error for writing stream data into partitioned hive/FS tables > without a partition committer > > > Key: FLINK-23178 > URL: https://issues.apache.org/jira/browse/FLINK-23178 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Connectors / Hive >Reporter: Rui Li >Priority: Major > > If a user writes streaming data into hive but forgets to specify a partition > commit policy, the job will run successfully but data was never committed to > hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23178) Raise an error for writing stream data into partitioned hive/FS tables without a partition committer
Rui Li created FLINK-23178: -- Summary: Raise an error for writing stream data into partitioned hive/FS tables without a partition committer Key: FLINK-23178 URL: https://issues.apache.org/jira/browse/FLINK-23178 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem, Connectors / Hive Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23168) Catalog shouldn't merge properties for alter DB operation
Rui Li created FLINK-23168: -- Summary: Catalog shouldn't merge properties for alter DB operation Key: FLINK-23168 URL: https://issues.apache.org/jira/browse/FLINK-23168 Project: Flink Issue Type: Improvement Components: Connectors / Hive, Table SQL / API Reporter: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-23010) HivePartitionFetcherContextBase::getComparablePartitionValueList can return partitions that don't exist
[
https://issues.apache.org/jira/browse/FLINK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-23010.
Fix Version/s: 1.13.2
1.12.5
1.14.0
Resolution: Fixed
Fixed in:
* master: 53034eacc498d79f08ae17d8c941b4931e619351
* release-1.13: 86581e90fb59289c4ad4be24495a9895d1099176
* release-1.12: 3a7552a3db8d4437a39f128b0a89470e9e2e7750
> HivePartitionFetcherContextBase::getComparablePartitionValueList can return
> partitions that don't exist
> ---
>
> Key: FLINK-23010
> URL: https://issues.apache.org/jira/browse/FLINK-23010
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.12.5, 1.13.2
>
>
> When consume order is {{CREATE_TIME}}, HivePartitionFetcherContextBase lists
> folders under table location to find partitions. This is wrong because HMS is
> the single source of truth to decide whether a partition exists for a hive
> table. Returning a non-existing partition will lead to unnecessary job FO
> when {{HiveContinuousPartitionFetcher}} fails to get the partition from HMS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-22617) Add log when create bulk format
[
https://issues.apache.org/jira/browse/FLINK-22617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-22617.
Fix Version/s: 1.14.0
Resolution: Fixed
Pushed to master: e58f7dd342846549949cdd9946cf0837e1b96cb5
> Add log when create bulk format
>
>
> Key: FLINK-22617
> URL: https://issues.apache.org/jira/browse/FLINK-22617
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive
>Reporter: hehuiyuan
>Assignee: hehuiyuan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: hivesourelog.jpg
>
>
> Hive table sink has some log that tells us whether to use native or mapred .
>
>
> {code:java}
> LOG.info("Hive streaming sink: Use MapReduce RecordWriter writer.");
> LOG.info("Hive streaming sink: Use native parquet&orc writer.");
> LOG.info(
> "Hive streaming sink: Use MapReduce RecordWriter writer because BulkWriter
> Factory not available.");
> {code}
>
> I have some ideas we can add log to make it more obvious when read hive for
> `createBulkFormatForSplit`.
>
> !hivesourelog.jpg|width=788,height=301!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368590#comment-17368590
]
Rui Li commented on FLINK-23096:
This is actually a different issue from FLINK-16688, un-linking it.
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li resolved FLINK-23096.
Fix Version/s: 1.13.2
1.14.0
Resolution: Fixed
Fixed in master: 36a05834f0aa0e949138a52e1bb1aa7116767b1a
Fixed in release-1.13: ea3d085f6b19b6dfb36462650344229c9a3ca70b
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.2
>
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-23096) HiveParser could not attach the sessionstate of hive
[
https://issues.apache.org/jira/browse/FLINK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned FLINK-23096:
--
Assignee: shizhengchao
> HiveParser could not attach the sessionstate of hive
>
>
> Key: FLINK-23096
> URL: https://issues.apache.org/jira/browse/FLINK-23096
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: shizhengchao
>Assignee: shizhengchao
>Priority: Major
>
> My sql code is as follows：
> {code:java}
> //代码占位符
> CREATE CATALOG myhive WITH (
> 'type' = 'hive',
> 'default-database' = 'default',
> 'hive-conf-dir' = '/home/service/upload-job-file/1624269463008'
> );
> use catalog hive;
> set 'table.sql-dialect' = 'hive';
> create view if not exists view_test as
> select
> cast(goods_id as string) as goods_id,
> cast(depot_id as string) as depot_id,
> cast(product_id as string) as product_id,
> cast(tenant_code as string) as tenant_code
> from edw.dim_yezi_whse_goods_base_info/*+
> OPTIONS('streaming-source.consume-start-offset'='dayno=20210621') */;
> {code}
> and the exception is as follows:
> {code:java}
> //代码占位符
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Conf non-local session path expected to be non-null
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.client.cli.CliFrontend$$Lambda$68/330382173.call(Unknown
> Source)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$69/680712932.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.NullPointerException: Conf non-local session path
> expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.clearSessionState(HiveParser.java:376)
> at
> org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:219)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:724)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callFlinkSql(FlinkSqlStreamingPlatform.java:157)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.callCommand(FlinkSqlStreamingPlatform.java:129)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.run(FlinkSqlStreamingPlatform.java:91)
> at
> com.shizhengchao.io.FlinkSqlStreamingPlatform.main(FlinkSqlStreamingPlatform.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 13 common frames omitted
> {code}
> My guess is that sessionstate is not set to threadlocal：
> {code:java}
> //代码占位符
> // @see org.apache.hadoop.hive.ql.session.SessionState.setCurrentSessionState
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
