I see, thank you, Robert.
On 21.11.2015 15:28, Robert Metzger wrote:
> Ah, I see. Maybe it would make sense then for you to use the latest
> Hadoop version we are supporting. This way, you get the most recent
> Hadoop S3 file system implementation.
>
> Note that there might be an issue with starting Flink 0.10.0 for Hadoop
> 2.7.0. We'll fix it with Flink 0.10.1.
> But if everything is working fine ... it might make sense not to change
> it now ("never change a running system").
>
>
> On Sat, Nov 21, 2015 at 3:24 PM, Konstantin Knauf
> <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>> wrote:
>
> Hi Robert,
>
> I am basically only reading from Kafka and S3 and writing to S3 in this
> job. So I am using the Hadoop S3 FileSystem classes, but that's it.
>
> Cheers,
>
> Konstantin
>
>
> On 21.11.2015 15:16, Robert Metzger wrote:
> > Hi,
> >
> > great to hear that its working. I've updated the documentation (for 1.0)
> > and made the word directory bold ;)
> >
> > You should try to match your Hadoop version as closely as possible.
> > Are you not using HDFS at all? Then it doesn't matter which version of
> > Flink you are downloading.
> > When using Hadoop 2.x then I'd recommend at least a Flink version for
> > Hadoop 2.3.0
> >
> >
> > On Sat, Nov 21, 2015 at 3:13 PM, Konstantin Knauf
> > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>
> <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>> wrote:
> >
> > Hi Robert,
> >
> > thanks a lot, it's working now. Actually, it also says "directory"
> in
> > the description. So I should have known :/
> >
> > On additional question though. If I use the flink binary for Hadoop
> > 1.2.1 and run flink in standalone mode, should I use the *-hadoop1
> > dependencies even If I am not interacting with HDFS 1.x?
> >
> > Cheers,
> >
> > Konstantin
> >
> > On 21.11.2015 14:52, Robert Metzger wrote:
> > > Hi,
> > >
> > > It seems that you've set the "fs.hdfs.hadoopconf" configuration
> > > parameter to a file. I think you have to set it the directory
> containing
> > > the configuration.
> > > Sorry, I know that's not very intuitive, but in Hadoop the
> settings for
> > > in different files (hdfs|yarn|core)-site.xml.
> > >
> > >
> > > On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf
> > > <konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>
> > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>>> wrote:
> > >
> > > Hi Ufuk,
> > >
> > > sorry for not getting back to you for so long, and thanks for
> your
> > > answer. The problem persists unfortunately. Running the job
> from the IDE
> > > works (with core-site.xml on classpath), running it in local
> standalone
> > > mode does not. AccessKeyID and SecretAccesKey are not found.
> > >
> > > Attached the jobmanager log on DEBUG level. The core-site.xml
> is
> > > definitely at the configured location.
> > >
> > > I am now on version 0.10.0 and using the binaries for Hadoop
> 1.2.1 to
> > > run the jar in local mode. Do I have to use the Hadoop 2.x
> version for
> > > this to work? I have put hadoop-common-2.3.jar into the flink
> lib
> > > folder.
> > >
> > > I don't know if it is relevant (but it seems to be related),
> when I run
> > > the job from my IDE I get the warning:
> > >
> > > 2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load
> > > native-hadoop library for your platform... using builtin-java
> classes
> > > where applicable
> > >
> > > Cheers and thank you,
> > >
> > > Konstantin
> > >
> > >
> > > On 14.10.2015 11:44, Ufuk Celebi wrote:
> > > >
> > > >> On 10 Oct 2015, at 22:59, snntr
> <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>
> <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>
> > > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>>>
> > wrote:
> > > >>
> > > >> Hey everyone,
> > > >>
> > > >> I was having the same problem with S3 and found this
> thread very
> > > useful.
> > > >> Everything works fine now, when I start Flink from my IDE,
> but
> > > when I run
> > > >> the jar in local mode I keep getting
> > > >>
> > > >> java.lang.IllegalArgumentException: AWS Access Key ID and
> Secret
> > > Access Key
> > > >> must be specified as the username or password
> (respectively) of a
> > > s3n URL,
> > > >> or by setting the fs.s3n.awsAccessKeyId or
> fs.s3n.awsSecretAccessKey
> > > >> properties (respectively).
> > > >>
> > > >> I have set fs.hdfs.hadoopconf to point to a core-site.xml
> on my local
> > > >> machine with the required properties. What am I missing?
> > > >>
> > > >> Any advice is highly appreciated ;)
> > > >
> > > > This looks like a problem with picking up the Hadoop
> config. Can
> > > you look into the logs to check whether the configuration is
> picked
> > > up? Change the log settings to DEBUG in log/log4j.properties
> for
> > > this. And can you provide the complete stack trace?
> > > >
> > > > – Ufuk
> > > >
> > > >
> > >
> > > --
> > > Konstantin Knauf * konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>
> > > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>>> * +49-174-3413182
> <tel:%2B49-174-3413182>
> > <tel:%2B49-174-3413182>
> > > <tel:%2B49-174-3413182>
> > > TNG Technology Consulting GmbH, Betastr. 13a, 85774
> Unterföhring
> > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
> > Dahlke
> > > Sitz: Unterföhring * Amtsgericht München * HRB 135082
> > >
> > >
> >
> > --
> > Konstantin Knauf * konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>
> > <mailto:konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com>> * +49-174-3413182
> <tel:%2B49-174-3413182>
> > <tel:%2B49-174-3413182>
> > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
> Dahlke
> > Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >
> >
>
> --
> Konstantin Knauf * konstantin.kn...@tngtech.com
> <mailto:konstantin.kn...@tngtech.com> * +49-174-3413182
> <tel:%2B49-174-3413182>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>
--
Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082
