Hi Jörn,
thanks for the little example. Maybe Avro changed the behavior about
maps from the old version we used in 1.3 to the newest version in Flink
1.4.0.
I will investigate this and might open an issue for it.
Regards,
Timo
Am 12/7/17 um 5:07 PM schrieb Joern Kottmann:
Hello Timo,
thanks for your quick response. I can't share the code of that pipeline here.
The Flink version I am using is this one:
http://people.apache.org/~aljoscha/flink-1.4.0-rc3/flink-1.4.0-src.tgz
That was compiled by me with mvn install -DskipTests (for some reason
the tests failed, can also share that, maybe a firewall issue on
Fedora 27).
I managed to isolate the problem and it can be reproduced with a few
lines running in the IDE:
https://github.com/kottmann/flink-avro-issue
And it is indeed like you suspected, the key is a Utf8, and I pass in
a String to the get.
But why did that now break with Flink 1.4.0 and runs on Flink 1.3.2?
Thanks again!
Jörn
On Thu, Dec 7, 2017 at 5:07 PM, Joern Kottmann <kottm...@gmail.com> wrote:
Hello Timo,
thanks for your quick response. I can't share the code of that pipeline here.
The Flink version I am using is this one:
http://people.apache.org/~aljoscha/flink-1.4.0-rc3/flink-1.4.0-src.tgz
That was compiled by me with mvn install -DskipTests (for some reason
the tests failed, can also share that, maybe a firewall issue on
Fedora 27).
I managed to isolate the problem and it can be reproduced with a few
lines running in the IDE:
https://github.com/kottmann/flink-avro-issue
And it is indeed like you suspected, the key is a Utf8, and I pass in
a String to the get.
But why did that now break with Flink 1.4.0 and runs on Flink 1.3.2?
Thanks again!
Jörn
On Thu, Dec 7, 2017 at 3:57 PM, Timo Walther <twal...@apache.org> wrote:
Can you also check the type of the keys in your map. Avro distinguished
between String and Utf8 class. Maybe this is why your key cannot be found.
Regards,
Timo
Am 12/7/17 um 3:54 PM schrieb Timo Walther:
Hi Jörn,
could you tell us a bit more about your job? Did you import the flink-avro
module? How does the Flink TypeInformation for your Avro type look like
using println(ds.getType)? It sounds like a very weird error if the
toString() method shows the key. Can you reproduce the error in your IDE and
enter the `get("theKey")` method using a debugger. I will loop in Aljoscha
that worked on Avro recently.
Regards,
Timo
Am 12/7/17 um 3:19 PM schrieb Joern Kottmann:
Hello,
after having a version mismatch between Avro in Flink 1.3.2 I decided
to see how things work with Flink 1.4.0.
The pipeline I am building runs now, deployed as standalone on YARN
with Flink 1.3.2 and putting it "FIRST" on the classpath (to use Avro
1.8.2 instead of an 1.7.x version).
The default setting for yarn.per-job-cluster.include-user-jar is
ORDERED which in my case was a bit tricky, because I first deployed a
pipeline having a jar starting with "A" so it was placed before the
flink jars, later I worked on a mostly identical pipeline with a name
a bit further down in the alphabet, and that was placed behind the
flink jars causing various issues. I would recommend to change this as
default to LAST. Then the jar name has no impact on how things are
loaded.
Should I open a jira for this?
Now after updating to Flink 1.4.0 and also running on YARN as
standalone I am getting null values in a map field I did set in
earlier steps in the pipeline on Avro objects. The thing is a bit
weird, because avroObject.toString would show the values just fine,
but I get null on retrieval.
I have my own TableInputFormat to read from HBase and output a Tuple2
containing an Avro object.
Next comes a map which tries to access it.
The failing piece looks like this:
ao.getTheMap().get("theKey") (returns null)
Doing an ao.toString would show the "theMap" and the value of "theKey".
Is this a bug in Flink 1.4.0 or is there something wrong with my
dependencies?
Thanks,
Jörn
