Is it required to use exactly the same versions of Hadoop as the
pre-bundled hadoop version?
I'm using Hadoop 2.7.1 cluster with Flink 1.9.1 and the corresponding
Prebundled Hadoop 2.7.5.
When I submit a job using:
[vagrant@node1 flink]$ ./bin/flink run -m yarn-cluster
./examples/streaming/SocketWindowWordCount.jar --port 9001
the tail of the log is empty , and i get the following messages:
vagrant@node1 flink]$ ./bin/flink run -m yarn-cluster
./examples/streaming/SocketWindowWordCount.jar --port 9001
2019-12-07 07:04:15,394 INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at /0.0.0.0:8032
2019-12-07 07:04:15,493 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
- No path for the flink jar passed. Using the location of
class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-07 07:04:15,493 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
- No path for the flink jar passed. Using the location of
class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-07 07:04:16,615 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:17,617 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:18,619 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:19,621 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 3 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:20,629 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 4 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:21,632 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 5 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:22,634 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 6 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:23,639 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 7 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:24,644 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 8 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:25,651 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:56,677 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:57,679 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:58,680 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:04:59,686 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 3 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:00,688 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 4 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:01,690 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 5 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:02,693 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 6 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:03,695 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 7 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:04,704 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 8 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
2019-12-07 07:05:05,715 INFO org.apache.hadoop.ipc.Client
- Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
Thanks!
Pankaj
On Wed, Nov 20, 2019 at 2:45 AM Pankaj Chand <pankajchanda...@gmail.com>
wrote:
> Thank you, Ana and Yang!
>
> On Tue, Nov 19, 2019, 9:29 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi Pankaj,
>>
>> First, you need to prepare a hadoop environment separately, including
>> hdfs and Yarn. If you are familiar
>> with hadoop, you could download the binary[1] and start the cluster on
>> you nodes manually. Otherwise,
>> some tools may help you to deploy a hadoop cluster, ambari[2] and
>> cloudera manager[2].
>>
>> Then, download the Flink with Hadoop pre-bundle. You can submit your
>> flink job now.
>>
>> Please make sure you set the correct HADOOP_CONF_DIR in your flink client
>> before starting a flink cluster.
>>
>>
>>
>> [1].https://hadoop.apache.org/releases.html
>> [2].
>> https://www.cloudera.com/products/open-source/apache-hadoop/apache-ambari.html
>> [3].
>> https://www.cloudera.com/products/product-components/cloudera-manager.html
>>
>> Ana <otakunon...@gmail.com> 于2019年11月20日周三 上午10:12写道：
>>
>>> Hi,
>>>
>>> I was able to run Flink on YARN by installing YARN and Flink separately.
>>>
>>> Thank you.
>>>
>>> Ana
>>>
>>> On Wed, Nov 20, 2019 at 10:42 AM Pankaj Chand <pankajchanda...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I want to run Flink on YARN upon a cluster of nodes. From the
>>>> documentation, I was not able to fully understand how to go about it. Some
>>>> of the archived answers are a bit old and had pending JIRA issues, so I
>>>> thought I would ask.
>>>>
>>>> Am I first supposed to install YARN separately, and then download the
>>>> Flink file and Hadoop pre-bundle? Or does the Hadoop-prebundle that I put
>>>> into Flink's /lib folder provide the entire YARN installation?
>>>>
>>>> Is there any download that bundles a complete *installation of Fink +
>>>> installation of YARN*?
>>>>
>>>> Thank you,
>>>>
>>>> Pankaj
>>>>
>>>
