Please disregard my last question. It is working fine with Hadoop 2.7.5. Thanks
On Sat, Dec 7, 2019 at 2:13 AM Pankaj Chand <pankajchanda...@gmail.com> wrote: > Is it required to use exactly the same versions of Hadoop as the > pre-bundled hadoop version? > > I'm using Hadoop 2.7.1 cluster with Flink 1.9.1 and the corresponding > Prebundled Hadoop 2.7.5. > > When I submit a job using: > > [vagrant@node1 flink]$ ./bin/flink run -m yarn-cluster > ./examples/streaming/SocketWindowWordCount.jar --port 9001 > > the tail of the log is empty , and i get the following messages: > > vagrant@node1 flink]$ ./bin/flink run -m yarn-cluster > ./examples/streaming/SocketWindowWordCount.jar --port 9001 > 2019-12-07 07:04:15,394 INFO org.apache.hadoop.yarn.client.RMProxy > - Connecting to ResourceManager at /0.0.0.0:8032 > 2019-12-07 07:04:15,493 INFO > org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path > for the flink jar passed. Using the location of class > org.apache.flink.yarn.YarnClusterDescriptor to locate the jar > 2019-12-07 07:04:15,493 INFO > org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path > for the flink jar passed. Using the location of class > org.apache.flink.yarn.YarnClusterDescriptor to locate the jar > 2019-12-07 07:04:16,615 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 0 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:17,617 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 1 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:18,619 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 2 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:19,621 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 3 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:20,629 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 4 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:21,632 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 5 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:22,634 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 6 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:23,639 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 7 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:24,644 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 8 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:25,651 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 9 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:56,677 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 0 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:57,679 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 1 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:58,680 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 2 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:04:59,686 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 3 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:00,688 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 4 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:01,690 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 5 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:02,693 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 6 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:03,695 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 7 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:04,704 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 8 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2019-12-07 07:05:05,715 INFO org.apache.hadoop.ipc.Client > - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. > Already tried 9 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > > Thanks! > > Pankaj > > On Wed, Nov 20, 2019 at 2:45 AM Pankaj Chand <pankajchanda...@gmail.com> > wrote: > >> Thank you, Ana and Yang! >> >> On Tue, Nov 19, 2019, 9:29 PM Yang Wang <danrtsey...@gmail.com> wrote: >> >>> Hi Pankaj, >>> >>> First, you need to prepare a hadoop environment separately, including >>> hdfs and Yarn. If you are familiar >>> with hadoop, you could download the binary[1] and start the cluster on >>> you nodes manually. Otherwise, >>> some tools may help you to deploy a hadoop cluster, ambari[2] and >>> cloudera manager[2]. >>> >>> Then, download the Flink with Hadoop pre-bundle. You can submit your >>> flink job now. >>> >>> Please make sure you set the correct HADOOP_CONF_DIR in your flink >>> client before starting a flink cluster. >>> >>> >>> >>> [1].https://hadoop.apache.org/releases.html >>> [2]. >>> https://www.cloudera.com/products/open-source/apache-hadoop/apache-ambari.html >>> [3]. >>> https://www.cloudera.com/products/product-components/cloudera-manager.html >>> >>> Ana <otakunon...@gmail.com> 于2019年11月20日周三 上午10:12写道： >>> >>>> Hi, >>>> >>>> I was able to run Flink on YARN by installing YARN and Flink separately. >>>> >>>> Thank you. >>>> >>>> Ana >>>> >>>> On Wed, Nov 20, 2019 at 10:42 AM Pankaj Chand < >>>> pankajchanda...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I want to run Flink on YARN upon a cluster of nodes. From the >>>>> documentation, I was not able to fully understand how to go about it. Some >>>>> of the archived answers are a bit old and had pending JIRA issues, so I >>>>> thought I would ask. >>>>> >>>>> Am I first supposed to install YARN separately, and then download the >>>>> Flink file and Hadoop pre-bundle? Or does the Hadoop-prebundle that I put >>>>> into Flink's /lib folder provide the entire YARN installation? >>>>> >>>>> Is there any download that bundles a complete *installation of Fink + >>>>> installation of YARN*? >>>>> >>>>> Thank you, >>>>> >>>>> Pankaj >>>>> >>>>
