Attached log file. (Log level : Trace)
Is this the issue ? Trying with very minimal graph (execution plan is
printed in log file)
WARN akka.remote.transport.netty.NettyTransport -
Remote connection to [/127.0.0.1:44322] failed with
org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException:
Adjusted frame length exceeds 10485760: 1195725860 - discarded
On Wed, Jul 11, 2018 at 5:44 PM Chesnay Schepler <[email protected]> wrote:
> Did/could you enable logging in the submitting code?
>
> On 11.07.2018 13:57, Mohan mohan wrote:
> > Hi,
> >
> > I have started flink in cluster mode. ..flink1.4.2/bin/$
> > ./start-cluster.sh (no config changes ie., default settings)
> > And trying to connect to it,
> > ExecutionEnvironment.createRemoteEnvironment("localhost", 6123,
> > "xxx.jar");
> >
> > I am not seeing any response, did not find anything in jobmanager log.
> > Please guide how to trace the issue. Or Do we need any preconfiguration?
> > In local environment everything works fine.
> >
> > Using : flink-1.4.2-bin-scala_2.11.tgz
> >
> > Thanks in advance.
>
>
>
11 Jul 2018 17:49:08.579 [main] INFO
org.apache.flink.api.java.typeutils.TypeExtractor.isValidPojoField
1876 - class
com.adaequare.etl2.batch.operator.EntityType does not contain a setter for
field targetEntity
11 Jul 2018 17:49:08.583 [main] INFO
org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo
1915 - Class class
com.adaequare.etl2.batch.operator.EntityType cannot be used as a POJO type
because not all fields are valid POJO fields, and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
11 Jul 2018 17:49:09.027 [main] DEBUG
org.apache.flink.api.java.ClosureCleaner.cleanThis0
120 - this$0 is accessed: false
11 Jul 2018 17:49:09.065 [main] INFO
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
999 - The job has 2 registered types and 0
default Kryo serializers
11 Jul 2018 17:49:09.068 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1012 - Registered Kryo types: [class
com.adaequare.etl2.batch.operator.EntityType, interface java.util.List]
11 Jul 2018 17:49:09.068 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1013 - Registered Kryo with Serializers types:
[]
11 Jul 2018 17:49:09.069 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1014 - Registered Kryo with Serializer Classes
types: []
11 Jul 2018 17:49:09.069 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1015 - Registered Kryo default Serializers: []
11 Jul 2018 17:49:09.070 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1016 - Registered Kryo default Serializers
Classes []
11 Jul 2018 17:49:09.070 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1017 - Registered POJO types: []
11 Jul 2018 17:49:09.072 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1020 - Static code analysis mode: DISABLE
11 Jul 2018 17:49:09.085 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
433 - Beginning compilation of program 'plan'
11 Jul 2018 17:49:09.095 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
442 - Using a default parallelism of 4
11 Jul 2018 17:49:09.095 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
443 - Using default data exchange mode PIPELINED
11 Jul 2018 17:49:09.120 [main] DEBUG
org.apache.flink.core.fs.FileSystem.loadFileSystems
937 - Loading extension file systems via
services
11 Jul 2018 17:49:09.124 [main] INFO
org.apache.flink.core.fs.FileSystem.loadHadoopFsFactory
1005 - Hadoop is not in the
classpath/dependencies. The extended set of supported File Systems via Hadoop
is not available.
11 Jul 2018 17:49:09.136 [main] DEBUG
org.apache.flink.api.common.io.FileInputFormat.open
810 - Opening input split
file:/tmp/stream2file8333235519400351128.tmp [0,42]
11 Jul 2018 17:49:09.150 [main] DEBUG
org.apache.flink.api.common.io.FileInputFormat.open
810 - Opening input split
file:/tmp/stream2file8333235519400351128.tmp [21,21]
{
"nodes": [
{
"id": 2,
"type": "source",
"pact": "Data Source",
"contents": "at readFile(ExecutionEnvironment.java:515)
(com.adaequare.etl2.batch.operator.EntityCsvInputFormat)",
"parallelism": "4",
"global_properties": [
{ "name": "Partitioning", "value": "RANDOM_PARTITIONED"
},
{ "name": "Partitioning Order", "value": "(none)" },
{ "name": "Uniqueness", "value": "not unique" }
],
"local_properties": [
{ "name": "Order", "value": "(none)" },
{ "name": "Grouping", "value": "not grouped" },
{ "name": "Uniqueness", "value": "not unique" }
],
"estimates": [
{ "name": "Est. Output Size", "value": "42.00" },
{ "name": "Est. Cardinality", "value": "3.00" }
],
"costs": [
{ "name": "Network", "value": "0.0" },
{ "name": "Disk I/O", "value": "42.00" },
{ "name": "CPU", "value": "0.0" },
{ "name": "Cumulative Network", "value": "0.0" },
{ "name": "Cumulative Disk I/O", "value": "42.00" },
{ "name": "Cumulative CPU", "value": "0.0" }
],
"compiler_hints": [
{ "name": "Output Size (bytes)", "value": "(none)" },
{ "name": "Output Cardinality", "value": "(none)" },
{ "name": "Avg. Output Record Size (bytes)", "value":
"(none)" },
{ "name": "Filter Factor", "value": "(none)" }
]
},
{
"id": 1,
"type": "pact",
"pact": "Map",
"contents": "Map at lambda$configure$0(MapOperator.java:21)",
"parallelism": "4",
"predecessors": [
{"id": 2, "ship_strategy": "Forward", "exchange_mode":
"PIPELINED"}
],
"driver_strategy": "Map",
"global_properties": [
{ "name": "Partitioning", "value": "RANDOM_PARTITIONED"
},
{ "name": "Partitioning Order", "value": "(none)" },
{ "name": "Uniqueness", "value": "not unique" }
],
"local_properties": [
{ "name": "Order", "value": "(none)" },
{ "name": "Grouping", "value": "not grouped" },
{ "name": "Uniqueness", "value": "not unique" }
],
"estimates": [
{ "name": "Est. Output Size", "value": "(unknown)" },
{ "name": "Est. Cardinality", "value": "3.00" }
],
"costs": [
{ "name": "Network", "value": "0.0" },
{ "name": "Disk I/O", "value": "0.0" },
{ "name": "CPU", "value": "0.0" },
{ "name": "Cumulative Network", "value": "0.0" },
{ "name": "Cumulative Disk I/O", "value": "42.00" },
{ "name": "Cumulative CPU", "value": "0.0" }
],
"compiler_hints": [
{ "name": "Output Size (bytes)", "value": "(none)" },
{ "name": "Output Cardinality", "value": "(none)" },
{ "name": "Avg. Output Record Size (bytes)", "value":
"(none)" },
{ "name": "Filter Factor", "value": "(none)" }
]
},
{
"id": 0,
"type": "sink",
"pact": "Data Sink",
"contents":
"com.adaequare.etl2.batch.operator.EntityCsvOutputFormat@3023df74",
"parallelism": "1",
"predecessors": [
{"id": 1, "ship_strategy": "Redistribute",
"exchange_mode": "PIPELINED"}
],
"global_properties": [
{ "name": "Partitioning", "value": "RANDOM_PARTITIONED"
},
{ "name": "Partitioning Order", "value": "(none)" },
{ "name": "Uniqueness", "value": "not unique" }
],
"local_properties": [
{ "name": "Order", "value": "(none)" },
{ "name": "Grouping", "value": "not grouped" },
{ "name": "Uniqueness", "value": "not unique" }
],
"estimates": [
{ "name": "Est. Output Size", "value": "(unknown)" },
{ "name": "Est. Cardinality", "value": "3.00" }
],
"costs": [
{ "name": "Network", "value": "(unknown)" },
{ "name": "Disk I/O", "value": "0.0" },
{ "name": "CPU", "value": "0.0" },
{ "name": "Cumulative Network", "value": "(unknown)" },
{ "name": "Cumulative Disk I/O", "value": "42.00" },
{ "name": "Cumulative CPU", "value": "0.0" }
],
"compiler_hints": [
{ "name": "Output Size (bytes)", "value": "(none)" },
{ "name": "Output Cardinality", "value": "(none)" },
{ "name": "Avg. Output Record Size (bytes)", "value":
"(none)" },
{ "name": "Filter Factor", "value": "(none)" }
]
}
]
}
11 Jul 2018 17:49:09.240 [main] INFO
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
999 - The job has 2 registered types and 0
default Kryo serializers
11 Jul 2018 17:49:09.241 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1012 - Registered Kryo types: [class
com.adaequare.etl2.batch.operator.EntityType, interface java.util.List]
11 Jul 2018 17:49:09.242 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1013 - Registered Kryo with Serializers types:
[]
11 Jul 2018 17:49:09.244 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1014 - Registered Kryo with Serializer Classes
types: []
11 Jul 2018 17:49:09.245 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1015 - Registered Kryo default Serializers: []
11 Jul 2018 17:49:09.247 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1016 - Registered Kryo default Serializers
Classes []
11 Jul 2018 17:49:09.252 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1017 - Registered POJO types: []
11 Jul 2018 17:49:09.253 [main] DEBUG
org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan
1020 - Static code analysis mode: DISABLE
11 Jul 2018 17:49:09.327 [main] WARN
org.apache.flink.configuration.Configuration.getRawValueFromOption
771 - Config uses deprecated configuration key
'jobmanager.rpc.address' instead of proper key 'rest.address'
11 Jul 2018 17:49:09.795 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.logging.InternalLoggerFactory.debug
71 - Using SLF4J as the default logging
framework
11 Jul 2018 17:49:09.799 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.debug
76 - -Dio.netty.eventLoopThreads: 8
11 Jul 2018 17:49:09.833 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug
76 - java.nio.Buffer.address: available
11 Jul 2018 17:49:09.834 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug
76 - sun.misc.Unsafe.theUnsafe: available
11 Jul 2018 17:49:09.834 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug
71 - sun.misc.Unsafe.copyMemory: available
11 Jul 2018 17:49:09.835 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug
76 - java.nio.Bits.unaligned: true
11 Jul 2018 17:49:09.836 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - Java version: 8
11 Jul 2018 17:49:09.836 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - -Dio.netty.noUnsafe: false
11 Jul 2018 17:49:09.836 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - sun.misc.Unsafe: available
11 Jul 2018 17:49:09.837 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - -Dio.netty.noJavassist: false
11 Jul 2018 17:49:09.894 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
71 - Javassist: available
11 Jul 2018 17:49:09.894 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
11 Jul 2018 17:49:09.894 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - -Dio.netty.bitMode: 64
(sun.arch.data.model)
11 Jul 2018 17:49:09.895 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug
76 - -Dio.netty.noPreferDirect: false
11 Jul 2018 17:49:09.907 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.debug
76 - -Dio.netty.noKeySetOptimization: false
11 Jul 2018 17:49:09.908 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.debug
76 - -Dio.netty.selectorAutoRebuildThreshold:
512
11 Jul 2018 17:49:09.913 [main] TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.trace
46 - Instrumented an optimized java.util.Set
into: sun.nio.ch.EPollSelectorImpl@77d2e85
11 Jul 2018 17:49:09.918 [main] INFO
org.apache.flink.runtime.rest.RestClient.<init>
119 - Rest client endpoint started.
11 Jul 2018 17:49:09.920 [main] DEBUG
org.apache.flink.client.program.ClusterClient.getOptimizedPlan
355 - Set parallelism 1, plan default
parallelism 4
11 Jul 2018 17:49:09.921 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
433 - Beginning compilation of program 'Flink
Java Job at Wed Jul 11 17:49:09 IST 2018'
11 Jul 2018 17:49:09.921 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
442 - Using a default parallelism of 4
11 Jul 2018 17:49:09.921 [main] DEBUG
org.apache.flink.optimizer.Optimizer.compile
443 - Using default data exchange mode PIPELINED
11 Jul 2018 17:49:09.922 [main] DEBUG
org.apache.flink.api.common.io.FileInputFormat.open
810 - Opening input split
file:/tmp/stream2file8333235519400351128.tmp [0,42]
11 Jul 2018 17:49:09.923 [main] DEBUG
org.apache.flink.api.common.io.FileInputFormat.open
810 - Opening input split
file:/tmp/stream2file8333235519400351128.tmp [21,21]
11 Jul 2018 17:49:09.980 [main] INFO
org.apache.flink.client.program.rest.RestClusterClient.submitJob
240 - Submitting job
3164bc57c90c80756a4ef962d1d8b5c7 (detached: false).
11 Jul 2018 17:49:09.981 [main] INFO
org.apache.flink.client.program.rest.RestClusterClient.submitJob
316 - Requesting blob server port.
11 Jul 2018 17:49:09.994 [main] DEBUG
org.apache.flink.runtime.rest.RestClient.sendRequest
156 - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:6123/blobserver/port
11 Jul 2018 17:49:10.029 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector.debug
81 -
-Dorg.apache.flink.shaded.netty4.io.netty.leakDetectionLevel: simple
11 Jul 2018 17:49:10.045 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadLocalRandom.debug
71 - -Dio.netty.initialSeedUniquifier:
0x24602cf608346d85 (took 0 ms)
11 Jul 2018 17:49:10.060 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil.debug
76 - -Dio.netty.allocator.type: unpooled
11 Jul 2018 17:49:10.060 [main] DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil.debug
76 - -Dio.netty.threadLocalDirectBufferSize:
65536
11 Jul 2018 17:49:10.099 [flink-rest-client-netty-1-1] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.JavassistTypeParameterMatcherGenerator.debug
76 - Generated:
org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectMatcher
11 Jul 2018 17:49:10.119 [Flink-RestClusterClient-IO-thread-3] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.Recycler.debug
76 - -Dio.netty.recycler.maxCapacity.default:
262144
11 Jul 2018 17:49:10.127 [flink-rest-client-netty-1-1] DEBUG
org.apache.flink.shaded.netty4.io.netty.util.internal.Cleaner0.debug
76 - java.nio.ByteBuffer.cleaner(): available
