[jira] [Created] (FLINK-6236) Savepoint page needs to include web console possibility
Rami created FLINK-6236: --- Summary: Savepoint page needs to include web console possibility Key: FLINK-6236 URL: https://issues.apache.org/jira/browse/FLINK-6236 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.2.0 Reporter: Rami Priority: Minor Starting Flink 1.2.0 it is also possible to point to the savepoint when starting a job. However, the page only mention the CLI only. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6235) Broken link in documentation
Rami created FLINK-6235: --- Summary: Broken link in documentation Key: FLINK-6235 URL: https://issues.apache.org/jira/browse/FLINK-6235 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.2.0 Reporter: Rami https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/index.html#task-chaining-and-resource-groups The link comes from: https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/savepoints.html At the bottom of the page -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-4029) Multi-field "sum" function just like "keyBy"
[
https://issues.apache.org/jira/browse/FLINK-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364203#comment-15364203
]
Rami commented on FLINK-4029:
-
Sure, thanks for working on this:
So if we have pojo for example with those fields:
int count=1;
int bytesConsumed=32898;
Now I have a stream of those pojos. I want to be able to do:
.sum ("count","bytesConsumed")
which will sum all count values together and all bytesConsumed together. and I
end up with the count of all and the sum of all the bytes consumed ( of course
within the defined window)
> Multi-field "sum" function just like "keyBy"
>
>
> Key: FLINK-4029
> URL: https://issues.apache.org/jira/browse/FLINK-4029
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Rami
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> I can use keyBy as follows:
> stream.keyBy(“pojo.field1”,”pojo.field2”,…)
> Would make sense that I can use sum for example, to do its job for more than
> one field:
> stream.sum(“pojo.field1”,”pojo.field2”,…)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4029) Multi-field "sum" function just like "keyBy"
Rami created FLINK-4029: --- Summary: Multi-field "sum" function just like "keyBy" Key: FLINK-4029 URL: https://issues.apache.org/jira/browse/FLINK-4029 Project: Flink Issue Type: Improvement Components: DataStream API Reporter: Rami Priority: Minor I can use keyBy as follows: stream.keyBy(“pojo.field1”,”pojo.field2”,…) Would make sense that I can use sum for example, to do its job for more than one field: stream.sum(“pojo.field1”,”pojo.field2”,…) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3702) DataStream API PojoFieldAccessor doesn't support nested POJOs
[
https://issues.apache.org/jira/browse/FLINK-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295502#comment-15295502
]
Rami commented on FLINK-3702:
-
This feature would really make a difference in our code. most of the events we
are having needs count, and many of those events share the nested rawevent
pojo, if we could have the count field only in the nested rawevent pojo , it
would be much cleaner code and easier to understand rather than having the
count field duplicated in the event and it's nested rawevent. Any ETA?
> DataStream API PojoFieldAccessor doesn't support nested POJOs
> -
>
> Key: FLINK-3702
> URL: https://issues.apache.org/jira/browse/FLINK-3702
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Affects Versions: 1.0.0
>Reporter: Robert Metzger
>
> The {{PojoFieldAccessor}} (which is used by {{.sum(String)}} and similar
> methods) doesn't support nested POJOs right now.
> As part of FLINK-3697 I'll add a check for a nested POJO and fail with an
> exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3697) keyBy() with nested POJO computes invalid field position indexes
[
https://issues.apache.org/jira/browse/FLINK-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226011#comment-15226011
]
Rami commented on FLINK-3697:
-
It took my few hours to find out that it is flink bug and not my doing. I am
getting:
Exception in thread "main" java.lang.IndexOutOfBoundsException
at
org.apache.flink.api.java.typeutils.PojoTypeInfo.getTypeAt(PojoTypeInfo.java:259)
at
org.apache.flink.streaming.util.keys.KeySelectorUtil.getSelectorForKeys(KeySelectorUtil.java:62)
at
org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:276)
when using the following nested pojo:
public class RawEvent {
public long timeStamp;
public String[] message;
public String type;
public int count=1;
public String productId;
public RawEvent() {
}
}
using this keyby :
.keyBy("rawEvent.productId")
however, it works fine if I rename "productId" to "c".
.keyBy("rawEvent.c")
This sound like the same issue described here. I think it is should be major
bug instead of minor. I am having hard time to explain to my team members that
we can't name the pojo variable to what it needs to be.
> keyBy() with nested POJO computes invalid field position indexes
>
>
> Key: FLINK-3697
> URL: https://issues.apache.org/jira/browse/FLINK-3697
> Project: Flink
> Issue Type: Bug
> Components: DataStream API
>Affects Versions: 1.0.0
> Environment: MacOS X 10.10
>Reporter: Ron Crocker
>Priority: Minor
> Labels: pojo
>
> Using named keys in keyBy() for nested POJO types results in failure. The
> iindexes for named key fields are used inconsistently with nested POJO types.
> In particular, {{PojoTypeInfo.getFlatFields()}} returns the field's position
> after (apparently) flattening the structure but is referenced in the
> unflattened version of the POJO type by {{PojoTypeInfo.getTypeAt()}}.
> In the example below, getFlatFields() returns positions 0, 1, and 14. These
> positions appear correct in the flattened structure of the Data class.
> However, in {{KeySelector getSelectorForKeys(Keys keys,
> TypeInformation typeInfo, ExecutionConfig executionConfig)}}, a call to
> {{compositeType.getTypeAt(logicalKeyPositions[i])}} for the third key results
> {{PojoTypeInfo.getTypeAt()}} declaring it out of range, as it compares the
> length of the directly named fields of the object vs the length of flattened
> version of that type.
> Concrete Example:
> Consider this graph:
> {code}
> DataStream dataStream = see.addSource(new
> FlinkKafkaConsumer08<>(timesliceConstants.topic, new DataDeserialzer(),
> kafkaConsumerProperties));
> dataStream
> .flatMap(new DataMapper())
> .keyBy("aaa", "abc", "wxyz")
> {code}
> {{DataDeserialzer}} returns a "NativeDataFormat" object; {{DataMapper}} takes
> this NativeDataFormat object and extracts individual Data objects: {code}
> public class Data {
> public int aaa;
> public int abc;
> public long wxyz;
> public int t1;
> public int t2;
> public Policy policy;
> public Stats stats;
> public Data() {}
> {code}
> A {{Policy}} object is an instance of this class:
> {code}
> public class Policy {
> public short a;
> public short b;
> public boolean c;
> public boolean d;
> public Policy() {}
> }
> {code}
> A {{Stats}} object is an instance of this class:
> {code}
> public class Stats {
> public long count;
> public float a;
> public float b;
> public float c;
> public float d;
> public float e;
> public Stats() {}
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
