Stephan it is exactly the same exception -UknownHost bal bla In Jboss for example the external are also not working, only the 0.0.0.0 - this is AWS NAT. We will proceed with VPC and then I will update you about what we get. Thanks for your help.
On Sun, Aug 30, 2015 at 6:05 PM, Stephan Ewen <se...@apache.org> wrote: > Why are the external IPs not working? Any kind of exception you can share? > > On Sun, Aug 30, 2015 at 5:02 PM, Alexey Sapozhnikov <ale...@scalabill.it> > wrote: > > > it will not help, since the internal IPs are changing in AWS from time to > > time and you should use only Public IP, which is not recognizable by > flink. > > Thats why all app servers, for example JBoss or even Flume are using > > "0.0.0.0" > > > > On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen <se...@apache.org> wrote: > > > > > What you can do as a temporary workaround is to actually enter the IP > > > address for "jobmanager.rpc.address" - that circumvents the DNS. > > > > > > Just saw that Akka 2.4 (released some time in the near future) > apparently > > > introduces an option to listen to all network interfaces. > > > > > > On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov < > ale...@scalabill.it > > > > > > wrote: > > > > > > > Fully understand. > > > > 1.My suggestion is to drop Akka and take something else, since this > > issue > > > > is really big > > > > 2.Not hostname not the endpoint are not working, clarifying the VPC > > topic > > > > now. > > > > > > > > On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <se...@apache.org> > > wrote: > > > > > > > > > Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes > > > > > annoying, but I have not found a good way around this. > > > > > The problem is that the address to bind to an the address used by > > > others > > > > to > > > > > send messages to the node is the same. ( > > > > > https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY) > > > > > > > > > > > > > > > All of that aside, we have been using Flink on Cloud Platforms GCE, > > > > Amazon, > > > > > and it worked. > > > > > > > > > > Can you try to use the internal hostname, or the fully qualified > > > external > > > > > hostname? > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov < > > > ale...@scalabill.it > > > > > > > > > > wrote: > > > > > > > > > > > Hi. > > > > > > First off - many thanks for your efforts and prompt help. > > > > > > We will try to find how to do it with DNS server on VPC. > > > > > > however, absence of "0.0.0.0" is definitely a huge bug - just > think > > > > about > > > > > > the current situation : if I dont have a VPC, I cant invoke the > > Flink > > > > > > functionality remotely in Amazon. > > > > > > We will clarify the VPC situation and I will write my impression > of > > > > what > > > > > we > > > > > > get. > > > > > > > > > > > > Thanks, guys > > > > > > > > > > > > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger < > > rmetz...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > From this blog post, it seems that this hostname is not > > resolvable: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html > > > > > > > Can you easily activate a DNS server in the VPC? > > > > > > > > > > > > > > 0.0.0.0 is not supported because of some requirements of the > Akka > > > > > > > framework. > > > > > > > But you should be able to use the host's IP address. > > > > > > > Can you try that as well? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov < > > > > > ale...@scalabill.it > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Here is the exception from the moment we tried to put in > > > > > > > > jobmanager.rpc.address the hostname of the machine which is > > > > > > ip-172-36-98 > > > > > > > > looks like it doesnt recognize this address. > > > > > > > > Why it doesnt support "0.0.0.0"???? > > > > > > > > > > > > > > > > 13:43:14,805 INFO > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------------------- > > > > > > > > 13:43:14,914 INFO > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - Loading configuration from > > > > > > /usr/local/test/flink-0.9.0/bin/../conf > > > > > > > > 13:43:14,942 INFO > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - Security is not enabled. Starting non-authenticated > > > > > JobManager. > > > > > > > > 13:43:14,942 INFO > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - Starting JobManager > > > > > > > > 13:43:14,944 INFO > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - Starting JobManager actor system at ip-172-36-98:6123. > > > > > > > > 13:43:15,047 ERROR > > org.apache.flink.runtime.jobmanager.JobManager > > > > > > > > - Failed to run JobManager. > > > > > > > > java.lang.Exception: Could not create JobManager actor system > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826) > > > > > > > > at > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) > > > > > > > > Caused by: java.net.UnknownHostException: ip-172-36-98: > unknown > > > > error > > > > > > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native > > > > Method) > > > > > > > > at > > > > > > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) > > > > > > > > @ > > > > > > > > > > > > > 21,1 > > > > > > > > 71% > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger < > > > > rmetz...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > How is Flink crashing when you start it on the Linux > machine > > in > > > > > > Amazon? > > > > > > > > > > > > > > > > > > Can you post the exception here? > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov < > > > > > > > ale...@scalabill.it > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello Stephan. > > > > > > > > > > > > > > > > > > > > We run this Linux machine on Amazon, which I predict, > most > > of > > > > the > > > > > > > > people > > > > > > > > > > will do. > > > > > > > > > > We tried to put "0.0.0.0" or Public IP of the machine- > > Flink > > > > > > crashes > > > > > > > on > > > > > > > > > > start, it doesnt recognize himself. > > > > > > > > > > It is very strange that it doesnt work with 0.0.0.0- > > > basically > > > > > this > > > > > > > is > > > > > > > > a > > > > > > > > > > way in Java to make it being seen widely. > > > > > > > > > > We tried to put there the hostname - what you get from > > > > "hostname" > > > > > > > > > command. > > > > > > > > > > It crashes. > > > > > > > > > > It works only with "localhost" and works only locally > > > > > > > > > > So what you suggest we will put there so the remote > client > > > > could > > > > > > > > connect. > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen < > > > > se...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Do you start Flink via YARN? In that case the > > > > > > > > "jobmanager.rpc.address" > > > > > > > > > is > > > > > > > > > > > not used, because YARN assigns containers/nodes. > > > > > > > > > > > > > > > > > > > > > > If you start Flink in "standalone" mode, this should be > > the > > > > > > address > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > node that runs the JobManager. It will be used as the > > > host/IP > > > > > > that > > > > > > > > > Flink > > > > > > > > > > > binds to. The same host should then be used in the > > > > > > > RemoteEnvironment. > > > > > > > > > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer < > > > > > ha...@scalabill.it > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hello all. > > > > > > > > > > > > > > > > > > > > > > > > Firstly- thank you for your valuable advices. > > > > > > > > > > > > We did some very fine tuned pinpoint test and comes > to > > > > > > following > > > > > > > > > > > > conclusions > > > > > > > > > > > > > > > > > > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7 > > > > > > > > > > > > 2.Once we copy our Java client program directy to the > > > > machine > > > > > > and > > > > > > > > run > > > > > > > > > > it > > > > > > > > > > > > directly there it worked very good > > > > > > > > > > > > The program is > > > > > > > > > > > > > > > > > > > > > > > > ..... > > > > > > > > > > > > > > > > > > > > > > > > ExecutionEnvironment envRemote > > > > > > > > > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost", > > > > > > 6123, > > > > > > > > > > > > "\usr\local\HananTestProj.jar"); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.api.java.DataSet text = > > > > > > > > > > > > (org.apache.flink.api.java.DataSet) > > > envRemote.fromElements( > > > > > > > > > > > > "Who's there?", > > > > > > > > > > > > "I think I hear them. Stand, ho! Who's > > > there?"); > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String, > > > > > > Integer>> > > > > > > > > > > > > wordCounts = text > > > > > > > > > > > > .flatMap(new LineSplitter()) > > > > > > > > > > > > .groupBy(0) > > > > > > > > > > > > .sum(1); > > > > > > > > > > > > > > > > > > > > > > > > wordCounts.print(); > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > public static class LineSplitter implements > > > > > > > > > FlatMapFunction<String, > > > > > > > > > > > > Tuple2<String, Integer>> { > > > > > > > > > > > > public void flatMap(String line, > > > > > > Collector<Tuple2<String, > > > > > > > > > > > Integer>> > > > > > > > > > > > > out) { > > > > > > > > > > > > for (String word : line.split(" ")) { > > > > > > > > > > > > out.collect(new Tuple2<String, > > > > Integer>(word, > > > > > > 1)); > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > ..... > > > > > > > > > > > > Program works fine > > > > > > > > > > > > 3.Now we are trying to run this program remotely , > from > > > > > windows > > > > > > > > > machine > > > > > > > > > > > > when the first row looks differently > > > > > > > > > > > > ExecutionEnvironment envRemote > > > > > > > > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4", > > > > > 6123, > > > > > > > > > > > > "C:\\HananTestProj.jar"); > > > > > > > > > > > > when 1.2.3.4 is IP address of fink machine > > > > > > > > > > > > > > > > > > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be > > > > reached > > > > > > bla > > > > > > > > bla > > > > > > > > > > bla > > > > > > > > > > > > > > > > > > > > > > > > 5.in flink configuration we found a following line > > > > > > > > > > > jobmanager.rpc.address: > > > > > > > > > > > > localhost > > > > > > > > > > > > Flink cant be started with any other value > > > > > (hostname/ipaddress > > > > > > ) > > > > > > > > > except > > > > > > > > > > > the > > > > > > > > > > > > localhost > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 6.We assume that probably Flink has a critical bug : > it > > > > cant > > > > > be > > > > > > > > > started > > > > > > > > > > > > from remote machine, only locally. Are we right? Are > we > > > > > wrong? > > > > > > > > Should > > > > > > > > > > we > > > > > > > > > > > > fill JIRA ? > > > > > > > > > > > > Maybe we need somehow to configure Flink differently? > > > > > > > > > > > > > > > > > > > > > > > > Please advice > > > > > > > > > > > > Best regards > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger < > > > > > > > > rmetz...@apache.org > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > The output of the YARN session should look like > this: > > > > > > > > > > > > > > > > > > > > > > > > > > Flink JobManager is now running on > > > > > quickstart.cloudera:39956 > > > > > > > > > > > > > JobManager Web Interface: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/ > > > > > > > > > > > > > Number of connected TaskManagers changed to 1. > Slots > > > > > > > available: 1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen < > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > The only thing I can think of is that you are not > > > using > > > > > the > > > > > > > > right > > > > > > > > > > > > > host/port > > > > > > > > > > > > > > for the JobManager. > > > > > > > > > > > > > > > > > > > > > > > > > > > > When you start the YARN session, it should print > > the > > > > host > > > > > > > where > > > > > > > > > the > > > > > > > > > > > > > > JobManager runs. You also need to take the port > > from > > > > > there, > > > > > > > as > > > > > > > > in > > > > > > > > > > > YARN, > > > > > > > > > > > > > the > > > > > > > > > > > > > > port is usually not 6123. Yarn starts many > services > > > on > > > > > one > > > > > > > > > machine, > > > > > > > > > > > so > > > > > > > > > > > > > the > > > > > > > > > > > > > > ports need to be randomized. > > > > > > > > > > > > > > > > > > > > > > > > > > > > It may be worth adding a YARNExecutionEnvironment > > at > > > > some > > > > > > > > point, > > > > > > > > > > > which > > > > > > > > > > > > > > deals with this transparent (starting the YARN > > > cluster, > > > > > > > > > connecting > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > JobManager). > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer < > > > > > > > > > ha...@scalabill.it> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello. > > > > > > > > > > > > > > > Let me clarify the situation. > > > > > > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We > > > > > connected > > > > > > it > > > > > > > > to > > > > > > > > > > HDFS > > > > > > > > > > > > > > 2.7.1. > > > > > > > > > > > > > > > 2. Locally, our program is working: once we run > > > flink > > > > > as > > > > > > > > > > > > > > ./start-local.sh, > > > > > > > > > > > > > > > we are able to connect and run the > > > > > > createRemoteEnvironment > > > > > > > > and > > > > > > > > > > > > Execute > > > > > > > > > > > > > > > methods. > > > > > > > > > > > > > > > 3.Due to our architecture and basic Flink > feature > > > we > > > > > want > > > > > > > to > > > > > > > > > > invoke > > > > > > > > > > > > > this > > > > > > > > > > > > > > > functionality REMOTELY , when our Java code is > > > > calling > > > > > > the > > > > > > > > > Flink > > > > > > > > > > > > > methods > > > > > > > > > > > > > > > from another server. > > > > > > > > > > > > > > > 4.We tried both > > > > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1", > > > > > > > > > > > > > > > 6123, "TestProj.jar"); and > > > > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment(" > > > > > > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which > is > > > > > > > definitely > > > > > > > > > not > > > > > > > > > > > > right > > > > > > > > > > > > > > > since > > > > > > > > > > > > > > > it should be an IP address) - it crash on the > > "cant > > > > > reach > > > > > > > > > > > JobManager" > > > > > > > > > > > > > > > error. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to us that it can be one of 2 issues. > > > > > > > > > > > > > > > 1.Somehow we need to configure flink to accept > > the > > > > > > > > connections > > > > > > > > > > from > > > > > > > > > > > > the > > > > > > > > > > > > > > > remote machine > > > > > > > > > > > > > > > 2.Flink has a critical showstopper bug that > > > > > jeopardizing > > > > > > a > > > > > > > > > whole > > > > > > > > > > > > > decision > > > > > > > > > > > > > > > to use this technology. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please advise us how we should advance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert > Metzger > > < > > > > > > > > > > > > rmetz...@apache.org> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in the exception you've posted earlier, you > can > > > see > > > > > the > > > > > > > > > > following > > > > > > > > > > > > > root > > > > > > > > > > > > > > > > cause: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor > not > > > > found > > > > > > for: > > > > > > > > > > > > > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL > > > > > > > > > > :6123/), > > > > > > > > > > > > > > > > Path(/user/jobmanager)] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This string > "akka.tcp://flink@FLINK_SERVER_URL > > > > > :6123/" > > > > > > > > > usually > > > > > > > > > > > > looks > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So > it > > > > seems > > > > > > that > > > > > > > > you > > > > > > > > > > are > > > > > > > > > > > > > > > > passing FLINK_SERVER_URL > > > > > > > > > > > > > > > > as the server hostname (or ip). > > > > > > > > > > > > > > > > Can you pass the correct hostname when you > call > > > > > > > > > > > > ExecutionEnvironment. > > > > > > > > > > > > > > > > createRemoteEnvironment(). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer > < > > > > > > > > > > ha...@scalabill.it > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > I'm currently using flink 0.9.0 which by > > maven > > > > > > support > > > > > > > > > > Hadoop > > > > > > > > > > > 1 > > > > > > > > > > > > . > > > > > > > > > > > > > > > > > By using > > > > flink-clients-0.7.0-hadoop2-incubating.jar > > > > > > > with > > > > > > > > > > > > > > > executePlan(Plan > > > > > > > > > > > > > > > > > p) method instead, I'm getting the same > > > > exception > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hanan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan > Meyer > > < > > > > > > > > > > > ha...@scalabill.it > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. I have restarted Flink service via > > > > > > > > > stop/start-loval.sh - > > > > > > > > > > > it > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > restarted successfully ,no errors in log > > > folder > > > > > > > > > > > > > > > > > > 2. default flink port is -6123 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Getting this via Eclips IDE: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.ProgramInvocationException: > > > > > > > > > > > > > Failed > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > resolve JobManager > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:379) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:356) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:349) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789) > > > > > > > > > > > > > > > > > > at Test.main(Test.java:39) > > > > > > > > > > > > > > > > > > Caused by: java.io.IOException: > JobManager > > at > > > > > > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL > > > > > > > :6123/user/jobmanager > > > > > > > > > not > > > > > > > > > > > > > > > reachable. > > > > > > > > > > > > > > > > > > Please make sure that the JobManager is > > > running > > > > > and > > > > > > > its > > > > > > > > > > port > > > > > > > > > > > is > > > > > > > > > > > > > > > > > reachable. > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:376) > > > > > > > > > > > > > > > > > > ... 7 more > > > > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: > Actor > > > not > > > > > > found > > > > > > > > for: > > > > > > > > > > > > > > > > > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL > > > > > > > > > > > > :6123/), > > > > > > > > > > > > > > > > > > Path(/user/jobmanager)] > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > > > > > > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > > > > > > > > > > > > > > > > > > at > > > > > > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > > > > > > > > > > > > > > > > > > at > > > > > > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) > > > > > > > > > > > > > > > > > > at > > > > > > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561) > > > > > > > > > > > > > > > > > > at > > > > > > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) > > > > > > > > > > > > > > > > > > at > > > > > > > akka.actor.ActorCell.terminate(ActorCell.scala:369) > > > > > > > > > > > > > > > > > > at > > > > > > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) > > > > > > > > > > > > > > > > > > at > > > > > > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) > > > > > > > > > > > > > > > > > > at > > > akka.dispatch.Mailbox.run(Mailbox.scala:220) > > > > > > > > > > > > > > > > > > at > > > > akka.dispatch.Mailbox.exec(Mailbox.scala:231) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert > > > > Metzger > > > > > < > > > > > > > > > > > > > > > rmetz...@apache.org> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> I guess you are getting an entire > > exception > > > > > after > > > > > > > the > > > > > > > > > > > > > > > > "org.apache.flink > > > > > > > > > > > > > > > > > >> > > .client.program.ProgramInvocationException: > > > > > Failed > > > > > > > to > > > > > > > > > > > > > > > > > >> resolve JobManager". > > > > > > > > > > > > > > > > > >> Can you post it here to help us > > > understanding > > > > > the > > > > > > > > issue? > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey > > > > > > Sapozhnikov > > > > > > > < > > > > > > > > > > > > > > > > > ale...@scalabill.it> > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > Hello all. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Some clarification: locally everything > > > works > > > > > > > great. > > > > > > > > > > > > > > > > > >> > However once we run our Flink on > remote > > > > linux > > > > > > > > machine > > > > > > > > > > and > > > > > > > > > > > > try > > > > > > > > > > > > > to > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >> > client program from our machine, using > > > > create > > > > > > > remote > > > > > > > > > > > > > > environment- > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > >> > JobManager is raising this exception > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, > Stephan > > > > Ewen > > > > > < > > > > > > > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > If you start the job via the > > "bin/flink" > > > > > > script, > > > > > > > > > then > > > > > > > > > > > > simply > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > >> > > > > > > > > "ExecutionEnvironment.getExecutionEnvironment()" > > > > > > > > > > rather > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > creating > > > > > > > > > > > > > > > > > >> a > > > > > > > > > > > > > > > > > >> > > remote environment manually. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > That way, hosts and ports are > > configured > > > > > > > > > > automatically. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, > > Robert > > > > > > Metzger > > > > > > > < > > > > > > > > > > > > > > > > > rmetz...@apache.org> > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > >> Hi, > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> Which values did you use for > > > > > FLINK_SERVER_URL > > > > > > > and > > > > > > > > > > > > > FLINK_PORT? > > > > > > > > > > > > > > > > > >> > >> Every time you deploy Flink on > YARN, > > > the > > > > > host > > > > > > > and > > > > > > > > > > port > > > > > > > > > > > > > > change, > > > > > > > > > > > > > > > > > >> because > > > > > > > > > > > > > > > > > >> > the > > > > > > > > > > > > > > > > > >> > >> JobManager is started on a > different > > > YARN > > > > > > > > > container. > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, > > Hanan > > > > > Meyer > > > > > > < > > > > > > > > > > > > > > > ha...@scalabill.it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > Hello All > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit > > > Flink > > > > to > > > > > > > Yarn > > > > > > > > > > single > > > > > > > > > > > > > node > > > > > > > > > > > > > > > > > cluster > > > > > > > > > > > > > > > > > >> I'm > > > > > > > > > > > > > > > > > >> > >> > getting : > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > "org.apache.flink.client.program.ProgramInvocationException: > > > > > > > > > > > > > > > > > >> Failed to > > > > > > > > > > > > > > > > > >> > >> > resolve JobManager" > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > Using Flink 0.9.0 > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > The Jar copy a file from one > > location > > > > in > > > > > > Hdfs > > > > > > > > to > > > > > > > > > > > > another > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > works > > > > > > > > > > > > > > > > > >> > fine > > > > > > > > > > > > > > > > > >> > >> > while executed locally on the > > single > > > > node > > > > > > > Yarn > > > > > > > > > > > cluster > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > >> > >> > bin/flink run -c Test > > > > > ./examples/MyJar.jar > > > > > > > > > > > > > > > > > >> > >> > > hdfs://localhost:9000/flink/in.txt > > > > > > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > The code skeleton: > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > ExecutionEnvironment > envRemote > > = > > > > > > > > > > > > > > > > > >> > >> > > > > > > > ExecutionEnvironment.createRemoteEnvironment > > > > > > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK > > > > > > > > PORT,JAR_PATH_ON_CLIENT); > > > > > > > > > > > > > > > > > >> > >> > DataSet<String> data = > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt"); > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt"); > > > > > > > > > > > > > > > > > >> > >> > envRemote.execute(); > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > Please advise, > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > Hanan Meyer > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > *Regards* > > > > > > > > > > > > > > > > > > > > *Alexey Sapozhnikov* > > > > > > > > > > CTO& Co-Founder > > > > > > > > > > Scalabillit Inc > > > > > > > > > > Aba Even 10-C, Herzelia, Israel > > > > > > > > > > M : +972-52-2363823 > > > > > > > > > > E : ale...@scalabill.it > > > > > > > > > > W : http://www.scalabill.it > > > > > > > > > > YT - https://youtu.be/9Rj309PTOFA > > > > > > > > > > Map:http://mapta.gs/Scalabillit > > > > > > > > > > Revolutionizing Proof-of-Concept > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > *Regards* > > > > > > > > > > > > > > > > *Alexey Sapozhnikov* > > > > > > > > CTO& Co-Founder > > > > > > > > Scalabillit Inc > > > > > > > > Aba Even 10-C, Herzelia, Israel > > > > > > > > M : +972-52-2363823 > > > > > > > > E : ale...@scalabill.it > > > > > > > > W : http://www.scalabill.it > > > > > > > > YT - https://youtu.be/9Rj309PTOFA > > > > > > > > Map:http://mapta.gs/Scalabillit > > > > > > > > Revolutionizing Proof-of-Concept > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Regards* > > > > > > > > > > > > *Alexey Sapozhnikov* > > > > > > CTO& Co-Founder > > > > > > Scalabillit Inc > > > > > > Aba Even 10-C, Herzelia, Israel > > > > > > M : +972-52-2363823 > > > > > > E : ale...@scalabill.it > > > > > > W : http://www.scalabill.it > > > > > > YT - https://youtu.be/9Rj309PTOFA > > > > > > Map:http://mapta.gs/Scalabillit > > > > > > Revolutionizing Proof-of-Concept > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Regards* > > > > > > > > *Alexey Sapozhnikov* > > > > CTO& Co-Founder > > > > Scalabillit Inc > > > > Aba Even 10-C, Herzelia, Israel > > > > M : +972-52-2363823 > > > > E : ale...@scalabill.it > > > > W : http://www.scalabill.it > > > > YT - https://youtu.be/9Rj309PTOFA > > > > Map:http://mapta.gs/Scalabillit > > > > Revolutionizing Proof-of-Concept > > > > > > > > > > > > > > > -- > > > > *Regards* > > > > *Alexey Sapozhnikov* > > CTO& Co-Founder > > Scalabillit Inc > > Aba Even 10-C, Herzelia, Israel > > M : +972-52-2363823 > > E : ale...@scalabill.it > > W : http://www.scalabill.it > > YT - https://youtu.be/9Rj309PTOFA > > Map:http://mapta.gs/Scalabillit > > Revolutionizing Proof-of-Concept > > > -- *Regards* *Alexey Sapozhnikov* CTO& Co-Founder Scalabillit Inc Aba Even 10-C, Herzelia, Israel M : +972-52-2363823 E : ale...@scalabill.it W : http://www.scalabill.it YT - https://youtu.be/9Rj309PTOFA Map:http://mapta.gs/Scalabillit Revolutionizing Proof-of-Concept
