Okay thanks. I removed zookeeper and kafka logs
/tmp/kafka-logs
/tmp/logs
/var/zookeeper/version-2/
/tmp/zookeeper*.log (on Zk)
restarted zookeeper
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
no topics exist!
Started Broker 2
$KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
[2014-03-11 12:05:32,651] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is overridden
to 104857600 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is overridden
to 1 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is overridden
to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is overridden to
1000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to
192.168.1.120:2181 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden to
168 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.ms is
overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
overridden to 10000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to
536870912 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
(kafka.server.KafkaServer)
[2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log cleaner
every 60000 ms (kafka.log.LogManager)
[2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log flusher
every 3000 ms with the following overrides Map() (kafka.log.LogManager)
[2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092.
(kafka.network.Acceptor)
[2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
(kafka.network.SocketServer)
[2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181
(kafka.server.KafkaZooKeeper)
[2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2014-03-11 12:05:32,817] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client
environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client
environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client
environment:os.version=2.6.32-279.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.name=root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,825] INFO Initiating client connection,
connectString=192.168.1.120:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@721e34d8 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,842] INFO Opening socket connection to server
/192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,846] INFO Socket connection established to
192.168.1.120/192.168.1.120:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,854] INFO Session establishment complete on server
192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2014-03-11 12:05:32,891] INFO Registered broker 2 at path /brokers/ids/2 with
address localhost:9092. (kafka.utils.ZkUtils$)
[2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
192.168.1.120:2181 (kafka.server.KafkaServer)
[2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is not in the
classpath (kafka.utils.Mx4jLoader$)
[2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
(kafka.server.ZookeeperLeaderElector)
[2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
(kafka.server.KafkaServer)
[2014-03-11 12:05:33,377] INFO New leader is 2
(kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
Then started Broker 1
$KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
[2014-03-11 12:07:20,843] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is overridden
to 104857600 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is overridden
to 1 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is overridden
to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is overridden to
1000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property zookeeper.connect is overridden to
192.168.1.120:2181 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property log.retention.hours is overridden to
168 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property num.network.threads is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property zookeeper.connection.timeout.ms is
overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property num.partitions is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
overridden to 10000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,879] INFO Property log.segment.bytes is overridden to
536870912 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
(kafka.server.KafkaServer)
[2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
'/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
[2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log cleaner
every 60000 ms (kafka.log.LogManager)
[2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log flusher
every 3000 ms with the following overrides Map() (kafka.log.LogManager)
[2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092.
(kafka.network.Acceptor)
[2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
(kafka.network.SocketServer)
[2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181
(kafka.server.KafkaZooKeeper)
[2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2014-03-11 12:07:21,029] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client
environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client
environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client
environment:os.version=2.6.32-279.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.name=root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,031] INFO Initiating client connection,
connectString=192.168.1.120:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@45419cee (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,043] INFO Opening socket connection to server
/192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,049] INFO Socket connection established to
192.168.1.120/192.168.1.120:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,058] INFO Session establishment complete on server
192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1 with
address localhost:9092. (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
192.168.1.120:2181 (kafka.server.KafkaServer)
[2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not in the
classpath (kafka.utils.Mx4jLoader$)
[2014-03-11 12:07:21,249] INFO conflict in /controller data: { "brokerid":1,
"timestamp":"1394539641229", "version":1 } stored data: { "brokerid":2,
"timestamp":"1394539533037", "version":1 } (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
(kafka.server.KafkaServer)
Could this be an issue?
INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
INFO Registered broker 1 at path /brokers/ids/1 with address localhost:9092.
> Date: Tue, 11 Mar 2014 09:51:54 -0700
> Subject: Re: Remote Zookeeper
> From: [email protected]
> To: [email protected]
>
> Seems you my have put your cluster in a very confused state with random
> addition and removal of brokers and topics. I'd suggest deleting the
> zookeeper and Kafka logs and starting over using the getting started
> tutorial from the wiki.
>
> On Tuesday, March 11, 2014, A A <[email protected]> wrote:
>
> > I noticed a problem with my topics. The previous two topics would only be
> > present on 1 broker. So I created a new topic
> >
> > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper
> > 192.168.1.120:2181--replica 2 --partition 1 --topic test-rep
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> >
> > No there is no firewall. Even then I stopped the iptables. And then I
> > tried the following
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > replicated message
> >
> > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test-rep
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test-rep
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> >
> >
> >
> >
> > > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > > Subject: Re: Remote Zookeeper
> > > From: [email protected] <javascript:;>
> > > To: [email protected] <javascript:;>
> > >
> > > Is there a firewall thats blocking connections on port 9092? Also, the
> > > broker list should be comma separated.
> > >
> > >
> > > On Tue, Mar 11, 2014 at 9:02 AM, A A <[email protected]> wrote:
> > >
> > > > Sorry one of the brokers for was down. Brought it back up. Tried the
> > > > following
> > > >
> > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > > 192.168.1.124:9092 --topic test
> > > > hello brokers
> > > >
> > > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to<
