[jira] [Created] (FLINK-7768) Load File Systems via Java Service abstraction

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-7768:
---

 Summary: Load File Systems via Java Service abstraction
 Key: FLINK-7768
 URL: https://issues.apache.org/jira/browse/FLINK-7768
 Project: Flink
  Issue Type: Improvement
  Components: Core
Reporter: Stephan Ewen
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7767) Avoid loading Hadoop conf dynamically at runtime

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-7767:
---

 Summary: Avoid loading Hadoop conf dynamically at runtime
 Key: FLINK-7767
 URL: https://issues.apache.org/jira/browse/FLINK-7767
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.4.0


The bucketing sink dynamically loads the Hadoop configuration in various places.

The result of that configuration is not always predictable, as it tries to 
automagically discover the Hadoop config files.

A better approach is to rely on the Flink configuration to find the Hadoop 
configuration, or to directly use the Hadoop configuration used by the Hadoop 
file systems.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7766) Remove obsolete reflection for hflush on HDFS

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-7766:
---

 Summary: Remove obsolete reflection for hflush on HDFS
 Key: FLINK-7766
 URL: https://issues.apache.org/jira/browse/FLINK-7766
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.4.0


This code originally existed for compatibility with Hadoop 1.

Since Hadoop 1 support is dropped, this is no longer necessary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: System resource logger

[Bowen Li]

System and processor info, marked as 'logged once' in gist shared by Piotr,
should still be logged instead of registered as metrics, right?

On Thu, Oct 5, 2017 at 2:38 AM, Till Rohrmann  wrote:

> Thanks for the proposal Piotr. I like it a lot since it will help people to
> better understand their system. I would also be in favour of adding them to
> the system metrics. I think o.a.f.runtime.metrics.util.MetricUtils is the
> right place to start. Given the small dependency footprint and the
> compatible license, I would be in favour of option 1.
>
> Cheers,
> Till
> ​
>
> On Thu, Oct 5, 2017 at 11:19 AM, Piotr Nowojski 
> wrote:
>
> > +1 thanks for pointing this out. It makes sense to just expand those
> > system metrics (I was not aware of them).
> >
> > > On Oct 4, 2017, at 6:07 PM, Greg Hogan  wrote:
> > >
> > > What if we added these as system metrics and added a way to write
> > metrics to a (separate?) log file?
> > >
> > >
> > >> On Oct 4, 2017, at 10:13 AM, Piotr Nowojski 
> > wrote:
> > >>
> > >> Hi,
> > >>
> > >> Lately I was debugging some weird test failures on Travis and I needed
> > to look into metrics like:
> > >> - User, System, IOWait, IRQ CPU usages (based on CPU ticks since
> > previous check)
> > >> - System wide memory consumption (including making sure that swap was
> > disabled)
> > >> - network usage
> > >> - etc…
> > >>
> > >> Without an access to the machines itself. For this purpose I
> > implemented some periodic daemon thread logger. Log output looked like
> this:
> > >>
> > >> https://gist.github.com/pnowojski/8b863abb0fb08ac75b62627feadbd2f7 <
> > https://gist.github.com/pnowojski/8b863abb0fb08ac75b62627feadbd2f7>
> > >>
> > >> I think it would be nice to add this feature to Flink itself, by
> > extending existing MemoryLogger. Same lack of information that I had with
> > travis could easily happen on productional environments. The problem is
> > that there is no easy way to obtain such kind of information without
> using
> > some external libraries (think about cross platform support). I have used
> > for that:
> > >>
> > >> https://github.com/oshi/oshi 
> > >>
> > >> It has some minimal additional dependencies, one thing worth noting is
> > a JNA - it’s JAR weights ~1MB. We would have two options to add this
> > feature:
> > >>
> > >> 1. Include this oshi dependency in flink-runtime
> > >> 2. Wrap oshi into flink-contrib/flink-resource-logger module and make
> > this new module an optional/dynamically loaded  dependency by
> flink-runtime
> > (used only if user manually copies flink-resource-logger.jar to a class
> > path).
> > >>
> > >> I would lean toward 1., since that’s a powerful tool and it’s
> > dependencies are pretty minimal (except this JNA’s jar size). What do you
> > think?
> > >>
> > >> Piotrek
> >
> >
>

Re: Unable to write snapshots to S3 on EMR

[Andy M.]

Hi Till,

I believe this is what you are looking for, classpath is much bigger for
the task manager.  I can also post the whole log file if needed:

2017-10-05 14:17:53,038 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
 -  Classpath:

[jira] [Created] (FLINK-7765) Enable dependency convergence

[Piotr Nowojski (JIRA)]

Piotr Nowojski created FLINK-7765:
-

 Summary: Enable dependency convergence
 Key: FLINK-7765
 URL: https://issues.apache.org/jira/browse/FLINK-7765
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski


For motivation check [#7739]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7764) FlinkKafkaProducer010 does not accept name, uid, or parallelism

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-7764:


 Summary: FlinkKafkaProducer010 does not accept name, uid, or 
parallelism
 Key: FLINK-7764
 URL: https://issues.apache.org/jira/browse/FLINK-7764
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Affects Versions: 1.3.2, 1.4.0
Reporter: Fabian Hueske


As [reported on the user 
list|https://lists.apache.org/thread.html/1e97b79cb5611a942beb609a61b5fba6d62a49c9c86336718a9a0004@%3Cuser.flink.apache.org%3E]:

When I try to use KafkaProducer with timestamps it fails to set name, uid or 
parallelism. It uses default values.

{code}
FlinkKafkaProducer010.FlinkKafkaProducer010Configuration producer = 
FlinkKafkaProducer010
.writeToKafkaWithTimestamps(stream, topicName, schema, props, partitioner);
producer.setFlushOnCheckpoint(flushOnCheckpoint);
producer.name("foo")
.uid("bar")
.setParallelism(5);

return producer;
{code}

As operator name it shows "FlinKafkaProducer 0.10.x” with the typo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7763) TableSinkITCase fails with "object reuse" enabled

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-7763:
---

 Summary: TableSinkITCase fails with "object reuse" enabled
 Key: FLINK-7763
 URL: https://issues.apache.org/jira/browse/FLINK-7763
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Affects Versions: 1.3.2, 1.4.0
Reporter: Aljoscha Krettek
Priority: Blocker
 Fix For: 1.4.0, 1.3.3


Set {{objectReuse}} to {{true}} in {{ExecutionConfig}} to reproduce the failing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Dependency convergence

[Till Rohrmann]

+1 for pulling our dependencies straight and guarding it via the
maven-enforcer-plugin.

On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski 
wrote:

> I meant for whole flink project.
>
> > On Oct 4, 2017, at 8:43 PM, Bowen Li  wrote:
> >
> > +1. This is great, Piotrek!
> >
> > BTW, can you clarify what you mean by 'project wide'? Is it the whole
> > `flink` project or just `flink-connector-kafka`? I think it's useful to
> > be applied to the whole flink project. I've seen dependencies conflict
> > problem like this in flink-connector-kinesis. Enabling this in flink
> would
> > protect us from many hidden issues.
> >
> > Bowen
> >
> >
> >
> > On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski 
> > wrote:
> >
> >> Hi,
> >>
> >> I have spent last couple of days trying to find and fix Kafka tests
> >> instabilities on Travis and I think I have finally found the main
> reason:
> >> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper
> 3.10.
> >> Effect was very subtle, because rarely in some corner cases (but not
> >> always) Netty was deadlocking itself…
> >>
> >> Because of that I would like to enable dependencyConvergence rule in
> >> maven-enforcer-plugin project wide - it catches this error immediately:
> >>
> >> Dependency convergence error for io.netty:netty:3.10.5.Final paths to
> >> dependency are:
> >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
> >>  +-org.apache.kafka:kafka_2.11:0.9.0.1
> >>+-org.apache.zookeeper:zookeeper:3.4.10
> >>  +-io.netty:netty:3.10.5.Final
> >> and
> >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
> >>  +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
> >>+-com.data-artisans:flakka-remote_2.11:2.3-custom
> >>  +-io.netty:netty:3.8.0.Final
> >>
> >> Currently this rule fails with multiple errors, but after those lost
> >> couple of days I’m pretty determined to fix all of them “just in case”.
> >> dependencyConvergence rule would protect us in the future against such
> >> nasty subtle bugs. Does anyone have any objections/issues that I’m not
> >> aware of?
> >>
> >> Piotrek
>
>

Re: Unable to write snapshots to S3 on EMR

[Till Rohrmann]

Hi Andy,

the CliFrontend is not executed via Yarn, thus, it is not affected by
dependencies which are added due to the underlying Yarn cluster. Therefore,
it would be helpful to look at the TaskManager logs. Either you have
enabled log aggregation on your Yarn cluster, then you can obtain the logs
via `yarn logs -applicationId ` or you have to retrieve
them from the machines where they were running (either by going directly
there or via the Yarn web interface).

Cheers,
Till

On Wed, Oct 4, 2017 at 4:27 PM, Andy M.  wrote:

> Hi Till,
>
> That is actually the classpath used by the flink bash script(that launches
> the jar using the java command).  I changed the execute to an echo, and
> grabbed that for the CLI arguments.
>
> I believe this is the class path from the log file(although it might not be
> the taskmanager log, is that any different from what would be in my
> flink-1.3.2/log folder?):
>
> 2017-10-02 20:03:26,450 INFO  org.apache.flink.client.CliFrontend
>  -  Classpath:
> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf:
>
> If that doesn't seem right, and you can point me in the right direction as
> to where the TaskManager logs would be, I would be happy to grab the
> information your looking for.
>
> Thank you
>
> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann 
> wrote:
>
> > Hi Andy,
> >
> > this looks to me indeed like a dependency problem. I assume that EMR or
> > something else is pulling in an incompatible version of Hadoop.
> >
> > The classpath you've posted, is this the one logged in the log files
> > (TaskManager log) or did you compile it yourself? In the latter case, it
> > would also be helpful to get access to the TaskManager logs.
> >
> > Cheers,
> > Till
> >
> > On Mon, Oct 2, 2017 at 10:20 PM, Andy M.  wrote:
> >
> > > Hi Fabian,
> > >
> > > 1) I have looked at the linked docs, and from what I can tell no setup
> > > should really need to be done to get Flink working(Other than
> downloading
> > > the correct binaries, which I believe I did)
> > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin-
> > > hadoop27-scala_2.11.tgz
> > >  > > hadoop27-scala_2.11.tgz>)
> > > This is for hadoop 2.7.X, which matches EMR 5.8.0.
> > >
> > > I appreciate any help or guidance you can provide me in fixing my
> > problems,
> > > please let me know if there is anything else I can provide you.
> > >
> > > Thank you
> > >
> > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske 
> wrote:
> > >
> > > > Hi Andy,
> > > >
> > > > I'm not an AWS expert, so I'll just check on some common issues.
> > > >
> > > > I guess you already had a look at the Flink docs for AWS/EMR but I'll
> > > post
> > > > the link just be to sure [1].
> > > >
> > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1)
> did
> > > you
> > > > built Flink yourself or did you download the binaries?
> > > > Does the Hadoop version of the Flink build match the Hadoop version
> of
> > > EMR
> > > > 5.8.0, i.e., Hadoop 2.7.x?
> > > >
> > > > Best, Fabian
> > > >
> > > > [1]
> > > > https://ci.apache.org/projects/flink/flink-docs-
> > > release-1.3/setup/aws.html
> > > >
> > > > 2017-10-02 21:51 GMT+02:00 Andy M. :
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > Sorry, I just realized I forgot to include that part.  The error
> > > returned
> > > > > is:
> > > > >
> > > > > java.lang.NoSuchMethodError:
> > > > > org.apache.hadoop.conf.Configuration.addResource(
> > > > Lorg/apache/hadoop/conf/
> > > > > Configuration;)V
> > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(
> > > > > EmrFileSystem.java:93)
> > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.
> > > > > initialize(HadoopFileSystem.java:328)
> > > > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(
> > > > > FileSystem.java:350)
> > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.
> java:389)
> > > > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> > > > > at org.apache.flink.runtime.state.filesystem.
> > > > > FsCheckpointStreamFactory.(FsCheckpointStreamFactory.
> java:99)
> > > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend.
> > > > > createStreamFactory(FsStateBackend.java:282)
> > > > > at org.apache.flink.contrib.streaming.state.
> RocksDBStateBackend.
> > > > > createStreamFactory(RocksDBStateBackend.java:273
> > > > >
> > > > > I believe it has something to do with the classpath, but I am
> unsure
> > > why
> > > > or
> > > > > how to fix it.  The classpath being used during the execution is:
> > > 

Re: System resource logger

[Piotr Nowojski]

+1 thanks for pointing this out. It makes sense to just expand those system 
metrics (I was not aware of them). 

> On Oct 4, 2017, at 6:07 PM, Greg Hogan  wrote:
> 
> What if we added these as system metrics and added a way to write metrics to 
> a (separate?) log file?
> 
> 
>> On Oct 4, 2017, at 10:13 AM, Piotr Nowojski  wrote:
>> 
>> Hi,
>> 
>> Lately I was debugging some weird test failures on Travis and I needed to 
>> look into metrics like:
>> - User, System, IOWait, IRQ CPU usages (based on CPU ticks since previous 
>> check)
>> - System wide memory consumption (including making sure that swap was 
>> disabled)
>> - network usage 
>> - etc…
>> 
>> Without an access to the machines itself. For this purpose I implemented 
>> some periodic daemon thread logger. Log output looked like this:
>> 
>> https://gist.github.com/pnowojski/8b863abb0fb08ac75b62627feadbd2f7 
>> 
>> 
>> I think it would be nice to add this feature to Flink itself, by extending 
>> existing MemoryLogger. Same lack of information that I had with travis could 
>> easily happen on productional environments. The problem is that there is no 
>> easy way to obtain such kind of information without using some external 
>> libraries (think about cross platform support). I have used for that:
>> 
>> https://github.com/oshi/oshi 
>> 
>> It has some minimal additional dependencies, one thing worth noting is a JNA 
>> - it’s JAR weights ~1MB. We would have two options to add this feature:
>> 
>> 1. Include this oshi dependency in flink-runtime
>> 2. Wrap oshi into flink-contrib/flink-resource-logger module and make this 
>> new module an optional/dynamically loaded  dependency by flink-runtime (used 
>> only if user manually copies flink-resource-logger.jar to a class path).
>> 
>> I would lean toward 1., since that’s a powerful tool and it’s dependencies 
>> are pretty minimal (except this JNA’s jar size). What do you think?
>> 
>> Piotrek

[jira] [Created] (FLINK-7762) Make WikipediaEditsSourceTest a proper test

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-7762:
---

 Summary: Make WikipediaEditsSourceTest a proper test
 Key: FLINK-7762
 URL: https://issues.apache.org/jira/browse/FLINK-7762
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Aljoscha Krettek
Priority: Minor


{{WikipediaEditsSourceTest}} is currently an ITCase even though it's called 
test. Making it a test reduces runtime and also makes it more stable because we 
don't run a whole Flink job.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)