Re: java.io.IOException: config()

2011-08-06 Thread jagaran das 
I am accessing through threads in parallel.

What is the concept of Lease in HDFS??

Regards,
JD



From: Harsh J ha...@cloudera.com
To: jagaran das jagaran_...@yahoo.co.in
Sent: Friday, 5 August 2011 11:37 PM
Subject: Re: java.io.IOException: config()


How long are you keeping it open for?


On 06-Aug-2011, at 10:14 AM, jagaran das wrote:

Hi,


I am using CDH3.
I need to stream huge amount of data from our application to hadoop.
I am opening a connection like
 
config.set(fs.default.name,hdfsURI);
FileSystem dfs = FileSystem.get(config);
String path = hdfsURI + connectionKey;
Path destPath = new Path(path);
logger.debug(Path --  + destPath.getName());
outStream = dfs.create(destPath);
and keeping the outStream open for some time and writing continuously through 
it and then closing it.
But it is throwing 


 
5Aug2011 21:36:48,550 DEBUG 
[LeaseChecker@DFSClient[clientName=DFSClient_218151655, ugi=jagarandas]: 
java.lang.Throwable: for testing
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.toString(DFSClient.java:1181)
at org.apache.hadoop.util.Daemon.init(Daemon.java:38)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.put(DFSClient.java:1094)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:547)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:219)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:584)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:565)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:472)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:464)
at 
com.apple.ireporter.common.persistence.ConnectionManager.createConnection(ConnectionManager.java:66)
at 
com.apple.ireporter.common.persistence.HDPPersistor.writeToHDP(HDPPersistor.java:93)
at 
com.apple.ireporter.datatransformer.translator.HDFSTranslator.persistData(HDFSTranslator.java:41)
at 
com.apple.ireporter.datatransformer.adapter.TranslatorAdapter.processData(TranslatorAdapter.java:61)
at 
com.apple.ireporter.datatransformer.DefaultMessageListener.persistValidatedData(DefaultMessageListener.java:276)
at 
com.apple.ireporter.datatransformer.DefaultMessageListener.onMessage(DefaultMessageListener.java:93)
at 
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:506)
at 
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:463)
at 
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at 
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
at 
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:260)
at 
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
at 
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
at java.lang.Thread.run(Thread.java:680)
] (RPC.java:230) - Call: renewLease 4
05Aug2011 21:36:48,550 DEBUG [listenerContainer-1] (DFSClient.java:3274) - 
DFSClient writeChunk allocating new packet seqno=0, 
src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819,
 packetSize=65557, chunksPerPacket=127, bytesCurBlock=0
05Aug2011 21:36:48,551 DEBUG [Thread-11] (DFSClient.java:2499) - Allocating 
new block
05Aug2011 21:36:48,552 DEBUG [sendParams-0] (Client.java:761) - IPC Client 
(47) connection to localhost/127.0.0.1:8020 from jagarandas sending #3
05Aug2011 21:36:48,553 DEBUG [IPC Client (47) connection to 
localhost/127.0.0.1:8020 from jagarandas] (Client.java:815) - IPC Client (47) 
connection to localhost/127.0.0.1:8020 from jagarandas got value #3
05Aug2011 21:36:48,556 DEBUG [Thread-11] (RPC.java:230) - Call: addBlock 4
05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3094) - pipeline = 
127.0.0.1:50010
05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3102) - Connecting to 
127.0.0.1:50010
05Aug2011 21:36:48,559 DEBUG [Thread-11] (DFSClient.java:3109) - Send buf size 
131072
05Aug2011 21:36:48,635 DEBUG [DataStreamer for file 
/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819
 block blk_-5183404460805094255_1042] (DFSClient.java:2533) - DataStreamer 
block blk_-5183404460805094255_1042 wrote packet seqno:0 size:1522 
offsetInBlock:0 lastPacketInBlock:true
05Aug2011 21:36:48,638 DEBUG [ResponseProcessor for block 
blk_-5183404460805094255_1042] (DFSClient.java:2640) - DFSClient Replies for 
seqno 0 are SUCCESS
05Aug2011 21:36:48,639 DEBUG [DataStreamer for file 
/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819
 block blk_-5183404460805094255_1042

java.io.IOException: config()

2011-08-05 Thread jagaran das 
Hi,

I have been struck with this exception:

java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.(Configuration.java:211)
at org.apache.hadoop.conf.Configuration.(Configuration.java:198)
at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:99)
at test.TestApp.main(TestApp.java:19)


  
05Aug2011 20:08:53,303 DEBUG 
[LeaseChecker@DFSClient[clientName=DFSClient_-1591195062, 
ugi=jagarandas,staff,com.apple.sharepoint.group.1,_developer,_lpoperator,_lpadmin,_appserveradm,admin,_appserverusr,localaccounts,everyone,fmsadmin,com.apple.access_screensharing,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3]:
 java.lang.Throwable: for testing

  
05Aug2011 20:08:53,315 DEBUG [listenerContainer-1] (DFSClient.java:3012) - 
DFSClient writeChunk allocating new packet seqno=0, 
src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_222812011-08-05-20-08-52,
 packetSize=65557, chunksPerPacket=127, bytesCurBlock=0

I saw the source code :

 public Configuration(boolean loadDefaults) {
    this.loadDefaults = loadDefaults;
    if (LOG.isDebugEnabled()) {
      LOG.debug(StringUtils.stringifyException(new IOException(config(;
    }
    synchronized(Configuration.class) {
      REGISTRY.put(this, null);
    }
  }

Log is in debug mode.

Can anyone please help me on this??

Regards,
JD

Re: java.io.IOException: config() IMP

2011-08-05 Thread jagaran das 
:8020 from jagarandas got value #4
05Aug2011 21:36:48,648 DEBUG [listenerContainer-1] (RPC.java:230) - Call: 
complete 3

Please help as it a production enhancement for us.

Regards
Jagaran 



From: Harsh J ha...@cloudera.com
To: u...@pig.apache.org; jagaran das jagaran_...@yahoo.co.in
Sent: Friday, 5 August 2011 8:54 PM
Subject: Re: java.io.IOException: config()

Could you explain how/where you're stuck?

That DEBUG log doesn't even seem like a valid throw; its just to get a
strace I believe.

On Sat, Aug 6, 2011 at 8:52 AM, jagaran das jagaran_...@yahoo.co.in wrote:
 Hi,

 I have been struck with this exception:

 java.io.IOException: config()
 at org.apache.hadoop.conf.Configuration.(Configuration.java:211)
 at org.apache.hadoop.conf.Configuration.(Configuration.java:198)
 at 
 org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:99)
 at test.TestApp.main(TestApp.java:19)



 05Aug2011 20:08:53,303 DEBUG 
 [LeaseChecker@DFSClient[clientName=DFSClient_-1591195062, 
 ugi=jagarandas,staff,com.apple.sharepoint.group.1,_developer,_lpoperator,_lpadmin,_appserveradm,admin,_appserverusr,localaccounts,everyone,fmsadmin,com.apple.access_screensharing,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3]:
  java.lang.Throwable: for testing


 05Aug2011 20:08:53,315 DEBUG [listenerContainer-1] (DFSClient.java:3012) - 
 DFSClient writeChunk allocating new packet seqno=0, 
 src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_222812011-08-05-20-08-52,
  packetSize=65557, chunksPerPacket=127, bytesCurBlock=0

 I saw the source code :

  public Configuration(boolean loadDefaults) {
     this.loadDefaults = loadDefaults;
     if (LOG.isDebugEnabled()) {
       LOG.debug(StringUtils.stringifyException(new IOException(config(;
     }
     synchronized(Configuration.class) {
       REGISTRY.put(this, null);
     }
   }

 Log is in debug mode.

 Can anyone please help me on this??

 Regards,
 JD



-- 
Harsh J