[jira] [Assigned] (FLINK-12939) Translate "Apache Kafka Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12939: -- Assignee: (was: aloyszhang) > Translate "Apache Kafka Connector" page into Chinese > > > Key: FLINK-12939 > URL: https://issues.apache.org/jira/browse/FLINK-12939 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Priority: Minor > > Translate the page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/kafka.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873903#comment-16873903
]
aloyszhang commented on FLINK-12848:
Hi [~jark]
I made a mistake for flink-1.9 test because I just pay attention to the DEBUG
information of FlinkTypeFactory#buildLogicalRowType.
Actually, there is no problem with flink-1.9.
Let's just see test flink-1.7 under.
Code need in my test display as follow:
SimpleProcessionTimeSource.java
{code:java}
public class SimpleProcessionTimeSource
implements StreamTableSource, DefinedProctimeAttribute {
public static final String PROCTIME_NAME = "timestamp";
private String[] fieldNames;
private TypeInformation[] typeInformations;
private RowTypeInfo typeInfo;
private TableSchema tableSchema;
public SimpleProcessionTimeSource(String[] fieldNames,
TypeInformation[] typeInformations) {
this.fieldNames = fieldNames;
this.typeInformations = typeInformations;
this.typeInfo = new RowTypeInfo(typeInformations, fieldNames);
String [] schemaFiled = new String [fieldNames.length + 1];
TypeInformation [] schemaTypes = new TypeInformation[typeInformations.length
+ 1];
System.arraycopy(fieldNames,0, schemaFiled, 0 ,fieldNames.length);
System.arraycopy(typeInformations,0, schemaTypes, 0 ,typeInformations.length);
schemaFiled[fieldNames.length] = PROCTIME_NAME;
schemaTypes[typeInformations.length] = Types.SQL_TIMESTAMP;
this.tableSchema = new TableSchema(schemaFiled, schemaTypes);
}
@Override public DataStream getDataStream(
StreamExecutionEnvironment execEnv) {
DataStreamSource ds = execEnv.addSource(new SimpleSourceFunction(),
"pbSource", typeInfo);
return ds;
}
@Override public TypeInformation getReturnType() {
return typeInfo;
}
@Override
public TableSchema getTableSchema() {
return tableSchema;
}
@Override public String explainSource() {
return "";
}
@Nullable
@Override
public String getProctimeAttribute() {
return PROCTIME_NAME;
}
class SimpleSourceFunction implements SourceFunction {{
}
@Override
public void run(SourceContext sourceContext) throws Exception {
}
@Override
public void cancel() {
}
}
}{code}
Test code：
{code:java}
@Test
public void test001(){
String [] fields = new String []{"first", "second"};
TypeInformation[] types = new TypeInformation[]{
Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
Types.ROW_NAMED(new String[]{"second002"}, Types.INT)
};
//build flink program
StreamExecutionEnvironment execEnv =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment env =
StreamTableEnvironment.getTableEnvironment(execEnv);
SimpleProcessionTimeSource streamTableSource = new
SimpleProcessionTimeSource(fields, types);
env.registerTableSource("testSource", streamTableSource);
Table sourceTable = env.scan("testSource");
System.out.println("Source table schema : ");
sourceTable.printSchema();
Table table =
sourceTable.select("first.get('first001'),second.get('second002')");
table.printSchema();
try {
execEnv.execute();
} catch (Exception e) {
e.printStackTrace();
}
}
{code}
Test result :
{code:java}
Source table schema :
root
|-- first: Row(first001: Integer)
|-- second: Row(first001: Integer)
|-- timestamp: TimeIndicatorTypeInfo(proctime)
org.apache.flink.table.api.ValidationException: Expression
'second.get(second002) failed on input check: Field name 'second002' could not
be found.
at
org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:156)
at
org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:97)
at
org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:84)
at org.apache.flink.table.plan.TreeNode.postOrderTransform(TreeNode.scala:72)
at org.apache.flink.table.plan.TreeNode$$anonfun$1.apply(TreeNode.scala:46)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at scala.collection.AbstractIterator.to(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
at org.apache.flink.table.plan.TreeNode.childrenTransform$1(TreeNode.scala:66)
at org.apache.flink.table.plan.TreeNode.postOrderTransform(TreeNode.scala:70)
at
[jira] [Closed] (FLINK-12949) remove legacy FlinkKafkaConsumer082
[ https://issues.apache.org/jira/browse/FLINK-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang closed FLINK-12949. -- Resolution: Won't Do > remove legacy FlinkKafkaConsumer082 > --- > > Key: FLINK-12949 > URL: https://issues.apache.org/jira/browse/FLINK-12949 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.9.0 >Reporter: aloyszhang >Assignee: aloyszhang >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12948) remove legacy FlinkKafkaConsumer081
[ https://issues.apache.org/jira/browse/FLINK-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang closed FLINK-12948. -- Resolution: Won't Do > remove legacy FlinkKafkaConsumer081 > --- > > Key: FLINK-12948 > URL: https://issues.apache.org/jira/browse/FLINK-12948 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: aloyszhang >Assignee: aloyszhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12950) remove legacy FlinkKafkaProducer
[ https://issues.apache.org/jira/browse/FLINK-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang closed FLINK-12950. -- Resolution: Won't Do > remove legacy FlinkKafkaProducer > > > Key: FLINK-12950 > URL: https://issues.apache.org/jira/browse/FLINK-12950 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.9.0 >Reporter: aloyszhang >Assignee: aloyszhang >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-12945) Translate "RabbitMQ Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870533#comment-16870533 ] aloyszhang edited comment on FLINK-12945 at 6/23/19 11:51 AM: -- Hi jasper, Thanks for watching this issue. Just like the "Translate Twitter connector" issue here [https://issues.apache.org/jira/projects/FLINK/issues/FLINK-12947?filter=allissues], I have been working on this and almost done. was (Author: aloyszhang): Hi jasper, Thanks for watching this issue. IJust like the "Translate Twitter connector" issue here [https://issues.apache.org/jira/projects/FLINK/issues/FLINK-12947?filter=allissues], I have been working on this and almost done. > Translate "RabbitMQ Connector" page into Chinese > > > Key: FLINK-12945 > URL: https://issues.apache.org/jira/browse/FLINK-12945 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/rabbitmq.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/rabbitmq.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12945) Translate "RabbitMQ Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870533#comment-16870533 ] aloyszhang commented on FLINK-12945: Hi jasper, Thanks for watching this issue. IJust like the "Translate Twitter connector" issue here [https://issues.apache.org/jira/projects/FLINK/issues/FLINK-12947?filter=allissues], I have been working on this and almost done. > Translate "RabbitMQ Connector" page into Chinese > > > Key: FLINK-12945 > URL: https://issues.apache.org/jira/browse/FLINK-12945 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/rabbitmq.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/rabbitmq.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12947) Translate "Twitter Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870529#comment-16870529 ] aloyszhang commented on FLINK-12947: Hi jasper, Thanks for watching this issue. I have been working on this and already done this. I will open a PR tonight or tomorrow. > Translate "Twitter Connector" page into Chinese > --- > > Key: FLINK-12947 > URL: https://issues.apache.org/jira/browse/FLINK-12947 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/twitter.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/twitter.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12950) remove legacy FlinkKafkaProducer
aloyszhang created FLINK-12950: -- Summary: remove legacy FlinkKafkaProducer Key: FLINK-12950 URL: https://issues.apache.org/jira/browse/FLINK-12950 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.9.0 Reporter: aloyszhang Assignee: aloyszhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12949) remove legacy FlinkKafkaConsumer082
aloyszhang created FLINK-12949: -- Summary: remove legacy FlinkKafkaConsumer082 Key: FLINK-12949 URL: https://issues.apache.org/jira/browse/FLINK-12949 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.9.0 Reporter: aloyszhang Assignee: aloyszhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12948) remove legacy FlinkKafkaConsumer081
aloyszhang created FLINK-12948: -- Summary: remove legacy FlinkKafkaConsumer081 Key: FLINK-12948 URL: https://issues.apache.org/jira/browse/FLINK-12948 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Reporter: aloyszhang Assignee: aloyszhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12944) Translate "Streaming File Sink" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12944: -- Assignee: aloyszhang > Translate "Streaming File Sink" page into Chinese > - > > Key: FLINK-12944 > URL: https://issues.apache.org/jira/browse/FLINK-12944 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/streamfile_sink.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/streamfile_sink.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12938) Translate "Streaming Connectors" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12938: -- Assignee: aloyszhang > Translate "Streaming Connectors" page into Chinese > -- > > Key: FLINK-12938 > URL: https://issues.apache.org/jira/browse/FLINK-12938 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Major > > Translate the page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/; into > Chinese. > The doc located in "flink/docs/dev/connectors/index.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12947) Translate "Twitter Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12947: -- Assignee: aloyszhang > Translate "Twitter Connector" page into Chinese > --- > > Key: FLINK-12947 > URL: https://issues.apache.org/jira/browse/FLINK-12947 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/twitter.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/twitter.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12946) Translate "Apache NiFi Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12946: -- Assignee: aloyszhang > Translate "Apache NiFi Connector" page into Chinese > --- > > Key: FLINK-12946 > URL: https://issues.apache.org/jira/browse/FLINK-12946 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/nifi.htmll; > into Chinese. > The doc located in "flink/docs/dev/connectors/nifi.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12945) Translate "RabbitMQ Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12945: -- Assignee: aloyszhang > Translate "RabbitMQ Connector" page into Chinese > > > Key: FLINK-12945 > URL: https://issues.apache.org/jira/browse/FLINK-12945 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/rabbitmq.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/rabbitmq.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12943) Translate "HDFS Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12943: -- Assignee: aloyszhang > Translate "HDFS Connector" page into Chinese > > > Key: FLINK-12943 > URL: https://issues.apache.org/jira/browse/FLINK-12943 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the internal page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/filesystem_sink.html; > into Chinese. > > The doc located in "flink/docs/dev/connectors/filesystem_sink.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12939) Translate "Apache Kafka Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12939: -- Assignee: aloyszhang > Translate "Apache Kafka Connector" page into Chinese > > > Key: FLINK-12939 > URL: https://issues.apache.org/jira/browse/FLINK-12939 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: aloyszhang >Priority: Minor > > Translate the page > "https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html; > into Chinese. > The doc located in "flink/docs/dev/connectors/kafka.zh.md" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang updated FLINK-12848:
---
Fix Version/s: (was: 1.9.0)
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang reopened FLINK-12848:
Assignee: vinoyang
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868224#comment-16868224
]
aloyszhang edited comment on FLINK-12848 at 6/20/19 9:25 AM:
-
Hi Enrico,
Simple add the fieldNames to the equals method of RowTypeInfo is not safe. It
will cause test failed in `ExternalCatalogInsertTest` because of some operator
like `union` use equals in RowTypeInfo to determine whether the two input are
the of same type. So I did not find a way to meet both tableEnv.scan() and
union operator.
-And more , this problem does not appear in flink-1.9.-
Actually, this problem also appeared in flink 1.9, test case like this :
{{{code:title=Test.java|borderStyle=solid}}}
@Test
public void test001(){
String [] fields = new String []\{"first", "second"};
TypeInformation[] types = new TypeInformation[]{
Types.ROW_NAMED(new String[]\{"a0001"}, Types.INT),
Types.ROW_NAMED(new String[]\{"b0002"}, Types.INT)
};
for(TypeInformation type: types){
System.out.println(type);
}
//build flink program
StreamExecutionEnvironment execEnv =
StreamExecutionEnvironment.getExecutionEnvironment();
execEnv.setParallelism(1);
StreamTableEnvironment env = StreamTableEnvironment.create(execEnv);
SimpleProcessionTimeSource streamTableSource = new
SimpleProcessionTimeSource(fields, types);
env.registerTableSource("testSource", streamTableSource);
Table sourceTable = env.scan("testSource");
System.out.println("Source table schema : ");
sourceTable.printSchema();
MemoryTableSourceSinkUtil.UnsafeMemoryAppendTableSink sink = new
MemoryTableSourceSinkUtil.UnsafeMemoryAppendTableSink();
env.registerTableSink("targetTable", sink.configure(fields, types));
sourceTable.select("first,second").insertInto("targetTable");
try {
execEnv.execute();
} catch (Exception e) {
e.printStackTrace();
}
}
{code}
was (Author: aloyszhang):
Hi Enrico,
Simple add the fieldNames to the equals method of RowTypeInfo is not safe. It
will cause test failed in `ExternalCatalogInsertTest` because of some operator
like `union` use equals in RowTypeInfo to determine whether the two input are
the of same type. So I did not find a way to meet both tableEnv.scan() and
union operator.
And more , this problem does not appear in flink-1.9.
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-12721) make flink-json more precisely when handle integer type
[ https://issues.apache.org/jira/browse/FLINK-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-12721: -- Assignee: (was: aloyszhang) > make flink-json more precisely when handle integer type > --- > > Key: FLINK-12721 > URL: https://issues.apache.org/jira/browse/FLINK-12721 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0 >Reporter: aloyszhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > At present， flink-json convert integer type to `Types.BIG_DEC` which will > make some mismatch error when sink to external storage system like MySql, we > can make it more precisely when by process integer as `Types.INT` > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-12721) make flink-json more precisely when handle integer type
[ https://issues.apache.org/jira/browse/FLINK-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang closed FLINK-12721. -- Resolution: Invalid > make flink-json more precisely when handle integer type > --- > > Key: FLINK-12721 > URL: https://issues.apache.org/jira/browse/FLINK-12721 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0 >Reporter: aloyszhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > At present， flink-json convert integer type to `Types.BIG_DEC` which will > make some mismatch error when sink to external storage system like MySql, we > can make it more precisely when by process integer as `Types.INT` > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868224#comment-16868224
]
aloyszhang commented on FLINK-12848:
Hi Enrico,
Simple add the fieldNames to the equals method of RowTypeInfo is not safe. It
will cause test failed in `ExternalCatalogInsertTest` because of some operator
like `union` use equals in RowTypeInfo to determine whether the two input are
the of same type. So I did not find a way to meet both tableEnv.scan() and
union operator.
And more , this problem does not appear in flink-1.9.
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang reassigned FLINK-12848:
--
Assignee: (was: aloyszhang)
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang closed FLINK-12848.
--
Resolution: Fixed
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
[
https://issues.apache.org/jira/browse/FLINK-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang updated FLINK-12848:
---
Fix Version/s: 1.9.0
> Method equals() in RowTypeInfo should consider fieldsNames
> --
>
> Key: FLINK-12848
> URL: https://issues.apache.org/jira/browse/FLINK-12848
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.7.2
>Reporter: aloyszhang
>Assignee: aloyszhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
> process data with RowTypeInfo type there may comes an error of the field
> name.
> {code:java}
> String [] fields = new String []{"first", "second"};
> TypeInformation[] types = new TypeInformation[]{
> Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
> Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment env =
> StreamTableEnvironment.getTableEnvironment(execEnv);
> SimpleProcessionTimeSource streamTableSource = new
> SimpleProcessionTimeSource(fields, types);
> env.registerTableSource("testSource", streamTableSource);
> Table sourceTable = env.scan("testSource");
> System.out.println("Source table schema : ");
> sourceTable.printSchema();
> {code}
> The table shcema will be
> {code:java}
> Source table schema :
> root
> |-- first: Row(first001: Integer)
> |-- second: Row(first001: Integer)
> |-- timestamp: TimeIndicatorTypeInfo(proctime)
> {code}
> the second field has the same name with the first field.
> So, we should consider the fieldnames in RowTypeInfo#equals()
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12848) Method equals() in RowTypeInfo should consider fieldsNames
aloyszhang created FLINK-12848:
--
Summary: Method equals() in RowTypeInfo should consider fieldsNames
Key: FLINK-12848
URL: https://issues.apache.org/jira/browse/FLINK-12848
Project: Flink
Issue Type: Improvement
Affects Versions: 1.7.2
Reporter: aloyszhang
Assignee: aloyszhang
Since the `RowTypeInfo#equals()` does not consider the fieldNames , when
process data with RowTypeInfo type there may comes an error of the field name.
{code:java}
String [] fields = new String []{"first", "second"};
TypeInformation[] types = new TypeInformation[]{
Types.ROW_NAMED(new String[]{"first001"}, Types.INT),
Types.ROW_NAMED(new String[]{"second002"}, Types.INT) };
StreamExecutionEnvironment execEnv =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment env =
StreamTableEnvironment.getTableEnvironment(execEnv);
SimpleProcessionTimeSource streamTableSource = new
SimpleProcessionTimeSource(fields, types);
env.registerTableSource("testSource", streamTableSource);
Table sourceTable = env.scan("testSource");
System.out.println("Source table schema : ");
sourceTable.printSchema();
{code}
The table shcema will be
{code:java}
Source table schema :
root
|-- first: Row(first001: Integer)
|-- second: Row(first001: Integer)
|-- timestamp: TimeIndicatorTypeInfo(proctime)
{code}
the second field has the same name with the first field.
So, we should consider the fieldnames in RowTypeInfo#equals()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-12721) make flink-json more precisely when handle integer type
[ https://issues.apache.org/jira/browse/FLINK-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang updated FLINK-12721: --- Description: At present， flink-json convert integer type to `Types.BIG_DEC` which will make some mismatch error when sink to external storage system like MySql, we can make it more precisely when by process integer as `Types.INT` was: At present， flink-json convert integer type to `Types.BIG_DEC` which will make some mismatch error when sink to some external storage system like MySql, we can make it more precisely when by process integer as `Types.INT` > make flink-json more precisely when handle integer type > --- > > Key: FLINK-12721 > URL: https://issues.apache.org/jira/browse/FLINK-12721 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0 >Reporter: aloyszhang >Assignee: aloyszhang >Priority: Major > > At present， flink-json convert integer type to `Types.BIG_DEC` which will > make some mismatch error when sink to external storage system like MySql, we > can make it more precisely when by process integer as `Types.INT` > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12721) make flink-json more precisely when handle integer type
aloyszhang created FLINK-12721: -- Summary: make flink-json more precisely when handle integer type Key: FLINK-12721 URL: https://issues.apache.org/jira/browse/FLINK-12721 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.8.0 Reporter: aloyszhang Assignee: aloyszhang At present， flink-json convert integer type to `Types.BIG_DEC` which will make some mismatch error when sink to some external storage system like MySql, we can make it more precisely when by process integer as `Types.INT` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12082) Bump up the jython-standalone version
aloyszhang created FLINK-12082: -- Summary: Bump up the jython-standalone version Key: FLINK-12082 URL: https://issues.apache.org/jira/browse/FLINK-12082 Project: Flink Issue Type: Improvement Affects Versions: 1.8.0 Reporter: aloyszhang Assignee: aloyszhang jyhont's CVE : h1. [CVE-2016-4000|https://www.cvedetails.com/cve/CVE-2016-4000/] we can update to 2.7.1b3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-12043) Null value check in array serializers classes
[
https://issues.apache.org/jira/browse/FLINK-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aloyszhang reassigned FLINK-12043:
--
Assignee: aloyszhang
> Null value check in array serializers classes
> -
>
> Key: FLINK-12043
> URL: https://issues.apache.org/jira/browse/FLINK-12043
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.7.2
>Reporter: Quan Shi
>Assignee: aloyszhang
>Priority: Major
>
> Null pointer exception when get length of "_from"_ if _"from"_ is null in
> copy() method:
>
> Involved classes:
> {code:java}
> // code placeholder
> public String[] copy(String[] from) {
>String[] target = new String[from.length];
>System.arraycopy(from, 0, target, 0, from.length);
>return target;
> }
> {code}
> Involved serializer classes in package
> "org.apache.flink.api.common.typeutils.base.array"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-12017) Support translation from Rank/FirstLastRow to StreamTransformation
[ https://issues.apache.org/jira/browse/FLINK-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang updated FLINK-12017: --- Description: Support translation from Rank/FirstLastRow to StreamTransformation, So following sql can be run: 1. SELECT a, b, c FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY b ORDER BY proc DESC) as rowNum FROM T ) WHERE rowNum = 1 2. SELECT * FROM ( SELECT category, shopId, num, ROW_NUMBER() OVER (PARTITION BY category ORDER BY num DESC) as rank_num FROM T) WHERE rank_num <= 2 was: Support translation from Rank/FirstLastRow to StreamTransformation, So following sql can be run: 1. SELECT a, b, c FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY b ORDER BY proc DESC) as rowNum FROM T ) WHERE rowNum = 1 2. SELECT * FROM ( SELECT category, shopId, num, ROW_NUMBER() OVER (PARTITION BY category ORDER BY num DESC) as rank_num FROM T) WHERE rank_num <= 2 > Support translation from Rank/FirstLastRow to StreamTransformation > -- > > Key: FLINK-12017 > URL: https://issues.apache.org/jira/browse/FLINK-12017 > Project: Flink > Issue Type: Task > Components: Table SQL / Planner >Reporter: Jing Zhang >Priority: Major > > Support translation from Rank/FirstLastRow to StreamTransformation, So > following sql can be run: > 1. > SELECT a, b, c > FROM ( > SELECT *, > ROW_NUMBER() OVER (PARTITION BY b ORDER BY proc DESC) as rowNum > FROM T > ) > WHERE rowNum = 1 > 2. > SELECT * > FROM ( > SELECT category, shopId, num, > ROW_NUMBER() OVER (PARTITION BY category ORDER BY num DESC) as rank_num > FROM T) > WHERE rank_num <= 2 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-9363) Bump up the Jackson version
[ https://issues.apache.org/jira/browse/FLINK-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aloyszhang reassigned FLINK-9363: - Assignee: aloyszhang (was: vinoyang) > Bump up the Jackson version > --- > > Key: FLINK-9363 > URL: https://issues.apache.org/jira/browse/FLINK-9363 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Ted Yu >Assignee: aloyszhang >Priority: Major > Labels: security > > CVE's for Jackson : > CVE-2017-17485 > CVE-2018-5968 > CVE-2018-7489 > We can upgrade to 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
