Jozef Vilcek created BEAM-4672:
----------------------------------
Summary: Dynamic destination write to HDFS fails with
IntervalWindow in shard template
Key: BEAM-4672
URL: https://issues.apache.org/jira/browse/BEAM-4672
Project: Beam
Issue Type: Bug
Components: io-java-hadoop
Affects Versions: 2.4.0
Reporter: Jozef Vilcek
Assignee: Chamikara Jayalath
I am doing a windowed write with dynamic destinations and it is failing with
error message :
{noformat}
java.lang.IllegalArgumentException: Expect srcResourceIds and destResourceIds
have the same scheme, but received hdfs, window-1970-01-01T00{noformat}
I am writing from IntervalWindow, for which target file name created by
DefaultFilenamePolicy is going to look something like this:
{noformat}
window-1970-01-01T00:00:00.000Z-1970-01-01T01:00:00.000Z-pane-0-last-00000-of-00001.avro{noformat}
When resolving the path, HadoopResourceId is used and this one is backed by
java URI. The filename like above is parsed incorrectly because of ':'
character introduced by window-to-string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
