date:20171006

Re: Unable to write snapshots to S3 on EMR

2017-10-06 Thread Bowen Li

Hi Andy,

I believe it's because you didn't set your s3 impl correctly. Try to set
your core-site.xml by following https://ci.apache.org/
projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3afilesystem-
recommended

Bowen

On Fri, Oct 6, 2017 at 7:59 AM, Andy M.  wrote:

> Hi Till,
>
> Seems like everything is in line there.  hadoop-common.jar ->
> hadoop-common-2.7.3-amzn-3.jar
>
> And when i decompiled that jar I see  public void addResource(Configuration
> conf) in org/apache/hadoop/conf/Configuration.java
>
> I agree that an incorrect version of the jar is probably being run, is
> there a way to limit the classpath for the TaskManager when starting the
> job?
>
> Thank you
>
> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann 
> wrote:
>
> > Hi Andy,
> >
> > could you check which Hadoop version this jar
> > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the
> > contained hadoop Configuration class has the method
> > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V.
> Maybe
> > this jar is the culprit because it comes from a different Hadoop version.
> >
> > Cheers,
> > Till
> > 
> >
> > On Thu, Oct 5, 2017 at 4:22 PM, Andy M.  wrote:
> >
> > > Hi Till,
> > >
> > > I believe this is what you are looking for, classpath is much bigger
> for
> > > the task manager.  I can also post the whole log file if needed:
> > >
> > > 2017-10-05 14:17:53,038 INFO  org.apache.flink.yarn.
> > YarnTaskManagerRunner
> > >  -  Classpath:
> > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-
> > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.
> > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:
> > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/
> > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-
> > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.
> > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/
> > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-
> > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/
> > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/
> > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-
> > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/
> > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/
> > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
> > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.
> > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/
> > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/
> > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.
> > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-
> > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/
> > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.
> > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/
> > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/
> > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-
> > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/
> > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/
> > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.
> > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.
> > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
> > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/
> > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
> > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
> > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/
> > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-
> > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/
> > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/
> > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.
> > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/
> > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/
> > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.
> > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/
> > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
> > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/
> > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/
> > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
> > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/
> > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.
> > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/
> > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/
> > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.
> > >

Issue of Twitter streaming with CustomEndpoint

2017-10-06 Thread Devendra Vishwakarma

Hi,

I am following Apache Flink Twitter streaming example given in
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java
to get the tweets using hashtag, for which I have created a custom
endpoint. But my custom endpoint is not working (When I use default
endpoint of the TwitterSource then it is working fine).

*My CustomEndpoint is -*

public class CustomEndpoint implements EndpointInitializer, Serializable {

  private static final long serialVersionUID = 1L;

  @Override
  public StreamingEndpoint createEndpoint() {
AppleSampleEndpoint endpoint = new AppleSampleEndpoint();
return endpoint;
  }}


*AppleSampleEndpoint.java-*

public class AppleSampleEndpoint extends RawEndpoint  {

  public static final String PATH =
"https://api.twitter.com/1.1/search/tweets.json?q=%23bigdata;;

public AppleSampleEndpoint() {
  super(PATH, HttpConstants.HTTP_GET);
}}



I want to fetch the tweets using hashtag in Apache Flink, for which I have
created a custom endpoint. But my custom endpoint is not working (When I
use apache flink default endpoint it is working).

*CustomEndpoint-*

public class CustomEndpoint implements EndpointInitializer, Serializable {

  private static final long serialVersionUID = 1L;

  @Override
  public StreamingEndpoint createEndpoint() {
// this default endpoint initializer returns the sample endpoint:
// Returning a sample from the firehose (all tweets)
AppleSampleEndpoint endpoint = new AppleSampleEndpoint();
return endpoint;
  }}

*AppleSampleEndpoint.java-*

public class AppleSampleEndpoint extends RawEndpoint  {

  public static final String PATH =
"https://api.twitter.com/1.1/search/tweets.json?q=%23bigdata;;

public AppleSampleEndpoint() {
  super(PATH, HttpConstants.HTTP_GET);
}}

And this is how I am using CustomEndpoint in my Main class-

DataStream streamSource;
if (params.has(TwitterSource.CONSUMER_KEY) &&
params.has(TwitterSource.CONSUMER_SECRET) &&
params.has(TwitterSource.TOKEN) &&
params.has(TwitterSource.TOKEN_SECRET)
) {
TwitterSource source = new TwitterSource(params.getProperties());
source.setCustomEndpointInitializer(new CustomEndpoint());
streamSource = env.addSource(source);

But I am not able to fetch tweets with hashtag apple here.

Thanks

Devendra Vishwakarma

[jira] [Created] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs

2017-10-06 Thread Ted Yu (JIRA)

Ted Yu created FLINK-7775:
-

 Summary: Remove unreferenced method 
PermanentBlobCache#getNumberOfCachedJobs
 Key: FLINK-7775
 URL: https://issues.apache.org/jira/browse/FLINK-7775
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu
Priority: Minor


{code}
  public int getNumberOfCachedJobs() {
return jobRefCounters.size();
  }
{code}
The method is not used.

We should remove it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Unable to write snapshots to S3 on EMR

2017-10-06 Thread Andy M.

Hi Till,

Seems like everything is in line there.  hadoop-common.jar ->
hadoop-common-2.7.3-amzn-3.jar

And when i decompiled that jar I see  public void addResource(Configuration
conf) in org/apache/hadoop/conf/Configuration.java

I agree that an incorrect version of the jar is probably being run, is
there a way to limit the classpath for the TaskManager when starting the
job?

Thank you

On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann  wrote:

> Hi Andy,
>
> could you check which Hadoop version this jar
> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the
> contained hadoop Configuration class has the method
> Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe
> this jar is the culprit because it comes from a different Hadoop version.
>
> Cheers,
> Till
> 
>
> On Thu, Oct 5, 2017 at 4:22 PM, Andy M.  wrote:
>
> > Hi Till,
> >
> > I believe this is what you are looking for, classpath is much bigger for
> > the task manager.  I can also post the whole log file if needed:
> >
> > 2017-10-05 14:17:53,038 INFO  org.apache.flink.yarn.
> YarnTaskManagerRunner
> >  -  Classpath:
> > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-
> > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.
> > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:
> > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/
> > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-
> > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.
> > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/
> > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-
> > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/
> > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/
> > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-
> > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/
> > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/
> > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
> > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.
> > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/
> > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/
> > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.
> > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-
> > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/
> > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.
> > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/
> > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/
> > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-
> > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/
> > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/
> > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.
> > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.
> > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
> > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/
> > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
> > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
> > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/
> > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-
> > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/
> > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/
> > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.
> > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/
> > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/
> > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.
> > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/
> > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
> > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/
> > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/
> > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
> > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/
> > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.
> > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/
> > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/
> > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.
> > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/
> > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-
> > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-
> > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:
> > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/
> > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-
> > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/
> > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/
> >

Re: Dependency convergence

2017-10-06 Thread Aljoscha Krettek

+2 ;-)

> On 5. Oct 2017, at 11:26, Till Rohrmann  wrote:
> 
> +1 for pulling our dependencies straight and guarding it via the
> maven-enforcer-plugin.
> 
> On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski 
> wrote:
> 
>> I meant for whole flink project.
>> 
>>> On Oct 4, 2017, at 8:43 PM, Bowen Li  wrote:
>>> 
>>> +1. This is great, Piotrek!
>>> 
>>> BTW, can you clarify what you mean by 'project wide'? Is it the whole
>>> `flink` project or just `flink-connector-kafka`? I think it's useful to
>>> be applied to the whole flink project. I've seen dependencies conflict
>>> problem like this in flink-connector-kinesis. Enabling this in flink
>> would
>>> protect us from many hidden issues.
>>> 
>>> Bowen
>>> 
>>> 
>>> 
>>> On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski 
>>> wrote:
>>> 
 Hi,
 
 I have spent last couple of days trying to find and fix Kafka tests
 instabilities on Travis and I think I have finally found the main
>> reason:
 dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper
>> 3.10.
 Effect was very subtle, because rarely in some corner cases (but not
 always) Netty was deadlocking itself…
 
 Because of that I would like to enable dependencyConvergence rule in
 maven-enforcer-plugin project wide - it catches this error immediately:
 
 Dependency convergence error for io.netty:netty:3.10.5.Final paths to
 dependency are:
 +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
 +-org.apache.kafka:kafka_2.11:0.9.0.1
   +-org.apache.zookeeper:zookeeper:3.4.10
 +-io.netty:netty:3.10.5.Final
 and
 +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
 +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
   +-com.data-artisans:flakka-remote_2.11:2.3-custom
 +-io.netty:netty:3.8.0.Final
 
 Currently this rule fails with multiple errors, but after those lost
 couple of days I’m pretty determined to fix all of them “just in case”.
 dependencyConvergence rule would protect us in the future against such
 nasty subtle bugs. Does anyone have any objections/issues that I’m not
 aware of?
 
 Piotrek
>> 
>>

[jira] [Created] (FLINK-7774) Deserializers are not cleaned up when closing input streams

2017-10-06 Thread Nico Kruber (JIRA)

Nico Kruber created FLINK-7774:
--

 Summary: Deserializers are not cleaned up when closing input 
streams
 Key: FLINK-7774
 URL: https://issues.apache.org/jira/browse/FLINK-7774
 Project: Flink
  Issue Type: Bug
  Components: Network
Affects Versions: 1.3.2, 1.4.0
Reporter: Nico Kruber
Assignee: Nico Kruber


On cleanup of the {{AbstractRecordReader}}, {{StreamInputProcessor}}, and 
{{StreamTwoInputProcessor}}, the deserializers' current buffers are cleaned up 
but not their internal {{spanningWrapper}} and {{nonSpanningWrapper}} via 
{{RecordDeserializer#clear}}. This call should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7773) Test instability in UtilsTest#testYarnFlinkResourceManagerJobManagerLostLeadership

2017-10-06 Thread Nico Kruber (JIRA)

Nico Kruber created FLINK-7773:
--

 Summary: Test instability in 
UtilsTest#testYarnFlinkResourceManagerJobManagerLostLeadership
 Key: FLINK-7773
 URL: https://issues.apache.org/jira/browse/FLINK-7773
 Project: Flink
  Issue Type: Bug
  Components: Tests, YARN
Affects Versions: 1.4.0
Reporter: Nico Kruber
Priority: Critical


{{UtilsTest#testYarnFlinkResourceManagerJobManagerLostLeadership}} may result 
in the following exception (repeated run in IntelliJ until failure, but also on 
Travis here: https://travis-ci.org/NicoK/flink/jobs/283696974 )

{code}
org.apache.flink.yarn.UtilsTest "Until Failure"

org.mockito.exceptions.misusing.UnfinishedStubbingException: 
Unfinished stubbing detected here:
-> at org.apache.flink.yarn.UtilsTest$1.(UtilsTest.java:171)

E.g. thenReturn() may be missing.
Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod();
Hints:
 1. missing thenReturn()
 2. you are trying to stub a final method, you naughty developer!
 3: you are stubbing the behaviour of another mock inside before 'thenReturn' 
instruction if completed


at org.apache.flink.yarn.UtilsTest$1.(UtilsTest.java:179)
at 
org.apache.flink.yarn.UtilsTest.testYarnFlinkResourceManagerJobManagerLostLeadership(UtilsTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:67)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}

The incriminating code is this:
{code}
doAnswer(new Answer() {
@Override
public Object answer(InvocationOnMock invocation) throws Throwable {
Container container = (Container) invocation.getArguments()[0];
resourceManagerGateway.tell(new 
NotifyResourceStarted(YarnFlinkResourceManager.extractResourceID(container)),
leader1Gateway);
return null;
}
}).when(nodeManagerClient).startContainer(Matchers.any(Container.class), 
Matchers.any(ContainerLaunchContext.class));
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Unable to write snapshots to S3 on EMR

2017-10-06 Thread Till Rohrmann

Hi Andy,

could you check which Hadoop version this jar
/usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the
contained hadoop Configuration class has the method
Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe
this jar is the culprit because it comes from a different Hadoop version.

Cheers,
Till


On Thu, Oct 5, 2017 at 4:22 PM, Andy M.  wrote:

> Hi Till,
>
> I believe this is what you are looking for, classpath is much bigger for
> the task manager.  I can also post the whole log file if needed:
>
> 2017-10-05 14:17:53,038 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
>  -  Classpath:
> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-
> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.
> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:
> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/
> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-
> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.
> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/
> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-
> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/
> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/
> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-
> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/
> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/
> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.
> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/
> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/
> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.
> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-
> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/
> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.
> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/
> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/
> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-
> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/
> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/
> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.
> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.
> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/
> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/
> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-
> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/
> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/
> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.
> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/
> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/
> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.
> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/
> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/
> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/
> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/
> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.
> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/
> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/
> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.
> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/
> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-
> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-
> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:
> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/
> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-
> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/
> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/
> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/
> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4.
> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:
> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.
> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/
> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/
> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.
> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/
> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/
> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson-
>

[jira] [Created] (FLINK-7772) Test instability in BlobCacheDeleteTest

2017-10-06 Thread Nico Kruber (JIRA)

Nico Kruber created FLINK-7772:
--

 Summary: Test instability in BlobCacheDeleteTest
 Key: FLINK-7772
 URL: https://issues.apache.org/jira/browse/FLINK-7772
 Project: Flink
  Issue Type: Bug
  Components: Network
Affects Versions: 1.4.0
Reporter: Nico Kruber
Assignee: Nico Kruber


{{BlobCacheDeleteTest}} did not account for the server executing the delete 
call of a transient BLOB after acknowledging the request. This resulted in the 
{{testDeleteTransientLocalFails*}} tests failing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7771) Make the operator state queryable

2017-10-06 Thread Kostas Kloudas (JIRA)

Kostas Kloudas created FLINK-7771:
-

 Summary: Make the operator state queryable
 Key: FLINK-7771
 URL: https://issues.apache.org/jira/browse/FLINK-7771
 Project: Flink
  Issue Type: Improvement
  Components: Queryable State
Affects Versions: 1.4.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.4.0


There seem to be some requests for making the operator (non-keyed) state 
queryable. This means that the user will specify the *uuid* of the operator and 
the *taskId*, and he will be able to access the state that corresponds to that 
operator and for that specific task.

This issue will serve to document the discussion on the topic, so that 
everybody can participate.

Personally, I think that such a feature should wait until some things on state 
handling are stabilized (_e.g._ replication and checkpoint management). My main 
concerns have to do with the semantics and guarantees that such a feature could 
offer *for now*. 

 At first, operator state is essentially a list state that can be reshuffled 
arbitrarily upon restoring or rescaling. This means that task1 will have at a 
given execution attempt elements _A,B,C_ while after restoring (even without 
rescaling) it may have _D,B,E_ without this implying that something happened to 
states _A_ and _C_. They were simply assigned to another task. This makes it 
hard to reason about the results that you get at any point in time, as it 
provides *no locality/consistency guarantees between executions*.

 The above, in combination with the fact that (for now) it is not possible to 
query the state at a specific point in time (_e.g._ the last checkpointed 
state), means that there is no easy way to get a consistent view of the state 
of an operator. So in the example above, when querying _(operatorA, task1)_ and 
_(operatorA, task2)_, the user can get states belonging to different "points in 
time" which can result to duplicates, lost values and all the problems 
encountered in distributed systems when there are no consistency guarantees.

The above illustrates some of the consistency problems that such a feature 
could face now.
 
I also link [~till.rohrmann] and [~skonto] as he also mentioned that this 
feature could be helpful.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7769) Move Queryable State outside the runtime.

2017-10-06 Thread Kostas Kloudas (JIRA)

Kostas Kloudas created FLINK-7769:
-

 Summary: Move Queryable State outside the runtime.
 Key: FLINK-7769
 URL: https://issues.apache.org/jira/browse/FLINK-7769
 Project: Flink
  Issue Type: Improvement
  Components: Queryable State
Affects Versions: 1.4.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7770) Hide Queryable State behind a proxy.

2017-10-06 Thread Kostas Kloudas (JIRA)

Kostas Kloudas created FLINK-7770:
-

 Summary: Hide Queryable State behind a proxy.
 Key: FLINK-7770
 URL: https://issues.apache.org/jira/browse/FLINK-7770
 Project: Flink
  Issue Type: Improvement
  Components: Queryable State
Affects Versions: 1.4.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Unable to write snapshots to S3 on EMR

Issue of Twitter streaming with CustomEndpoint

[jira] [Created] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs

Re: Unable to write snapshots to S3 on EMR

Re: Dependency convergence

[jira] [Created] (FLINK-7774) Deserializers are not cleaned up when closing input streams

[jira] [Created] (FLINK-7773) Test instability in UtilsTest#testYarnFlinkResourceManagerJobManagerLostLeadership

Re: Unable to write snapshots to S3 on EMR

[jira] [Created] (FLINK-7772) Test instability in BlobCacheDeleteTest

[jira] [Created] (FLINK-7771) Make the operator state queryable

[jira] [Created] (FLINK-7769) Move Queryable State outside the runtime.

[jira] [Created] (FLINK-7770) Hide Queryable State behind a proxy.

12 matches

Site Navigation

Mail list logo

Footer information