[jira] [Comment Edited] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248712#comment-17248712 ] badqiu edited comment on FLINK-19146 at 12/14/20, 2:00 AM: --- set `table.exec.mini-batch.enabled`=`true`; set `table.exec.mini-batch.allow-latency`=`5s`; set `table.exec.mini-batch.size`=`2000`; CREATE TABLE test_table ( name STRING ) WITH ( 'connector'='datagen', 'rows-per-second'='1', 'fields.name.length'='1' ); select name,count(1) from test_table group by name; this select sql has no output。 modify set `table.exec.mini-batch.size`=`2`; will output result; was (Author: badqiu): set `table.exec.mini-batch.enabled`=`true`; set `table.exec.mini-batch.allow-latency`=`5s`; set `table.exec.mini-batch.size`=`2000`; CREATE TABLE test_table ( name STRING ) WITH ( 'connector'='datagen', 'rows-per-second'='1', 'fields.name.length'='1' ); select name,count(*) from test_table group by name; this select sql has no output。 modify set `table.exec.mini-batch.size`=`2`; will output result; > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248712#comment-17248712 ] badqiu commented on FLINK-19146: set `table.exec.mini-batch.enabled`=`true`; set `table.exec.mini-batch.allow-latency`=`5s`; set `table.exec.mini-batch.size`=`2000`; CREATE TABLE test_table ( name STRING ) WITH ( 'connector'='datagen', 'rows-per-second'='1', 'fields.name.length'='1' ); select name,count(*) from test_table group by name; this select sql has no output。 modify set `table.exec.mini-batch.size`=`2`; will output result; > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247717#comment-17247717 ] badqiu commented on FLINK-19146: As you said, it seems to work. But the actual test found no effect. Only size is in effect. If the output result of my "group by" is less than 100, mini batch size=100. It will not any output > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247607#comment-17247607 ] badqiu commented on FLINK-19146: This problem is repeated if the output result set is small. > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19390) mysql jdbc join error java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long
[ https://issues.apache.org/jira/browse/FLINK-19390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201448#comment-17201448 ] badqiu commented on FLINK-19390: yes, But why use BigInteger, convert it to avoid errors。 > mysql jdbc join error java.lang.ClassCastException: java.math.BigInteger > cannot be cast to java.lang.Long > - > > Key: FLINK-19390 > URL: https://issues.apache.org/jira/browse/FLINK-19390 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > > join on by t1.order_id = t2.order_id. > and order_id dataType: *bigint*. will error. > > > Caused by: java.lang.RuntimeException: java.lang.ClassCastException: > java.math.BigInteger cannot be cast to java.lang.Long > at > org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:84) > at > org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:71) > at > org.apache.flink.table.functions.TableFunction.collect(TableFunction.java:203) > at > org.apache.flink.connector.jdbc.table.JdbcRowDataLookupFunction.eval(JdbcRowDataLookupFunction.java:162) > at LookupFunction$43.flatMap(Unknown Source) > at > org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:82) > at > org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:36) > at > org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) > at StreamExecCalc$40.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111) > at > org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:352) > at > org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:185) > at > org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:141) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:755) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:201) > Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast > to java.lang.Long > at > org.apache.flink.table.data.GenericRowData.getLong(GenericRowData.java:154) > at TableCalcMapFunction$47.flatMap(Unknown Source) > at > org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:82) > ... 27 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19390) mysql jdbc join error java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long
badqiu created FLINK-19390: -- Summary: mysql jdbc join error java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long Key: FLINK-19390 URL: https://issues.apache.org/jira/browse/FLINK-19390 Project: Flink Issue Type: Bug Affects Versions: 1.11.1 Reporter: badqiu join on by t1.order_id = t2.order_id. and order_id dataType: *bigint*. will error. Caused by: java.lang.RuntimeException: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:84) at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:71) at org.apache.flink.table.functions.TableFunction.collect(TableFunction.java:203) at org.apache.flink.connector.jdbc.table.JdbcRowDataLookupFunction.eval(JdbcRowDataLookupFunction.java:162) at LookupFunction$43.flatMap(Unknown Source) at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:82) at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:36) at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at StreamExecCalc$40.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104) at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111) at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:352) at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:185) at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:141) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:755) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:201) Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long at org.apache.flink.table.data.GenericRowData.getLong(GenericRowData.java:154) at TableCalcMapFunction$47.flatMap(Unknown Source) at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinWithCalcRunner$CalcCollector.collect(LookupJoinWithCalcRunner.java:82) ... 27 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191534#comment-17191534 ] badqiu commented on FLINK-19146: My test result is not like this. search result : *trigger by size* !mini_batch_trigger_by_size.png! search result : *trigger by latency* !mini_batch_trigger_by_latency.png! > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
[ https://issues.apache.org/jira/browse/FLINK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] badqiu updated FLINK-19146: --- Attachment: mini_batch_trigger_by_size.png mini_batch_trigger_by_latency.png > createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and > table.exec.mini-batch.allow-latency > > > Key: FLINK-19146 > URL: https://issues.apache.org/jira/browse/FLINK-19146 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Affects Versions: 1.11.1 >Reporter: badqiu >Priority: Major > Attachments: mini_batch_trigger_by_latency.png, > mini_batch_trigger_by_size.png > > > Using *or* conditions, you can control the total data delay and improve > computing performance. > Increase the batch size to very large, but the data delay is still within the > set range. > > > table.exec.mini-batch.size is true > => > (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19146) createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency
badqiu created FLINK-19146: -- Summary: createMiniBatchTrigger() use OR ，table.exec.mini-batch.size and table.exec.mini-batch.allow-latency Key: FLINK-19146 URL: https://issues.apache.org/jira/browse/FLINK-19146 Project: Flink Issue Type: New Feature Components: API / DataStream Affects Versions: 1.11.1 Reporter: badqiu Using *or* conditions, you can control the total data delay and improve computing performance. Increase the batch size to very large, but the data delay is still within the set range. table.exec.mini-batch.size is true => (table.exec.mini-batch.size or table.exec.mini-batch.allow-latency) is true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19048) support line format for table connector
[ https://issues.apache.org/jira/browse/FLINK-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185081#comment-17185081 ] badqiu commented on FLINK-19048: 这个啥时候支持呢？ 用这个 + UDF解析数据方便很多啊。 > support line format for table connector > --- > > Key: FLINK-19048 > URL: https://issues.apache.org/jira/browse/FLINK-19048 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: badqiu >Priority: Major > Attachments: LineFormatFactory.java, > LineRowDataDeserializationSchema.java, LineRowDeserializationSchema.java, > LineRowFormatFactory.java, LineRowSchemaConverter.java, > LineRowSerializationSchema.java > > > Native string data format. No data conversion is done. > This format is particularly friendly to data without time attributes. With > UDF, writing your own data analysis will be much more convenient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19048) support line format for table connector
[ https://issues.apache.org/jira/browse/FLINK-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] badqiu updated FLINK-19048: --- Summary: support line format for table connector (was: suuport line format for table connector) > support line format for table connector > --- > > Key: FLINK-19048 > URL: https://issues.apache.org/jira/browse/FLINK-19048 > Project: Flink > Issue Type: New Feature > Components: API / Type Serialization System >Reporter: badqiu >Priority: Major > Attachments: LineFormatFactory.java, > LineRowDataDeserializationSchema.java, LineRowDeserializationSchema.java, > LineRowFormatFactory.java, LineRowSchemaConverter.java, > LineRowSerializationSchema.java > > > Native string data format. No data conversion is done. > This format is particularly friendly to data without time attributes. With > UDF, writing your own data analysis will be much more convenient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19048) suuport line format for table connector
badqiu created FLINK-19048: -- Summary: suuport line format for table connector Key: FLINK-19048 URL: https://issues.apache.org/jira/browse/FLINK-19048 Project: Flink Issue Type: New Feature Components: API / Type Serialization System Reporter: badqiu Attachments: LineFormatFactory.java, LineRowDataDeserializationSchema.java, LineRowDeserializationSchema.java, LineRowFormatFactory.java, LineRowSchemaConverter.java, LineRowSerializationSchema.java Native string data format. No data conversion is done. This format is particularly friendly to data without time attributes. With UDF, writing your own data analysis will be much more convenient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
