ok i could live with setting mapred.job.tracker manually for the code for now.
This way it can connect now to the proper jobtracker.
> The hadoop map tasks will need to connect to the kafka server port (the
> broker uri/port).
i run the hadoop soncumer on the same hostA where the kafka-server is running.
each of the host in the hadoop cluster can telnet/nmap to port 9092 on
hostA where the kafka-server is running.
also HostA can connect to port 5181 on any host in the cluster.
but each task fails with a similar connection issue :
java.io.IOException: java.net.ConnectException: Connection refused
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
at
kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
at
kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
at
kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
at kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
... 11 more
