yes, it's working fine. now not getting compile time error.
But when i trying to run this on cluster or yarn, getting following runtime
error :
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could
not find a file system implementation for scheme 'hdfs'. The scheme is
not directly supported by Flink and no Hadoop file system to support
this scheme could be loaded.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Hadoop File System abstraction does not support scheme 'hdfs'. Either
no file system implementation exists for that scheme, or the relevant
classes are missing from the classpath.
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
... 12 more
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99)
... 13 more
while submitting job it's printing following logs so i think it's
including hdoop libs :
Using the result of 'hadoop classpath' to augment the Hadoop
classpath:
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
On Fri, Dec 8, 2017 at 9:24 PM, shashank agarwal <[email protected]>
wrote:
> Sure i’ll Try that. Thanks
>
> On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen <[email protected]> wrote:
>
>> I would recommend to add "flink-shaded-hadoop2". That is a bundle of all
>> Hadoop dependencies used by Flink.
>>
>>
>> On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <[email protected]>
>> wrote:
>>
>>> I see, thanks for letting us know!
>>>
>>>
>>> On 8. Dec 2017, at 15:42, shashank agarwal <[email protected]>
>>> wrote:
>>>
>>> I had to include two dependencies.
>>>
>>> hadoop-hdfs (this for HDFS configuration)
>>> hadoop-common (this for Path)
>>>
>>>
>>>
>>>
>>> On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek <[email protected]>
>>> wrote:
>>>
>>>> I think hadoop-hdfs might be sufficient.
>>>>
>>>>
>>>> On 8. Dec 2017, at 14:48, shashank agarwal <[email protected]>
>>>> wrote:
>>>>
>>>> Can you specifically guide which dependencies should I add to extend
>>>> this :
>>>>
>>>> https://github.com/apache/flink/blob/release-1.4.0-rc3/
>>>> flink-connectors/flink-connector-filesystem/src/main/
>>>> java/org/apache/flink/streaming/connectors/fs/Bucketer.java
>>>>
>>>> is the Hadoop-core is sufficient.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal <[email protected]
>>>> > wrote:
>>>>
>>>>> It's a compilation error. I think I have to include the Hadoop
>>>>> dependencies.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Is this a compilation error or at runtime. But in general, yes you
>>>>>> have to include the Hadoop dependencies if they're not there.
>>>>>>
>>>>>> Best,
>>>>>> Aljoscha
>>>>>>
>>>>>>
>>>>>> On 8. Dec 2017, at 14:10, shashank agarwal <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am trying to test 1.4.0-RC3, Hadoop libraries removed in this
>>>>>> version. Actually, i have created custom Bucketer for the bucketing sink.
>>>>>> I am extending
>>>>>>
>>>>>>
>>>>>> org.apache.flink.streaming.connectors.fs.bucketing.Bucketer
>>>>>>
>>>>>> in the class, i have to use org.apache.hadoop.fs.Path but as hadoop
>>>>>> libraries removed it's giving error
>>>>>>
>>>>>> "object hadoop is not a member of package org.apache"
>>>>>>
>>>>>> Should i have to include Hadoop client libs in build.sbt dependencies.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Shashank
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks Regards
>>>>>
>>>>> SHASHANK AGARWAL
>>>>> --- Trying to mobilize the things....
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks Regards
>>>>
>>>> SHASHANK AGARWAL
>>>> --- Trying to mobilize the things....
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks Regards
>>>
>>> SHASHANK AGARWAL
>>> --- Trying to mobilize the things....
>>>
>>>
>>>
>> --
> Sent from iPhone 5
>
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the things....
