Fixed the issue by making the following config changes : tez.am.container.idle.release-timeout-min.millis=30000
tez.am.container.idle.release-timeout-max.millis=90000 hive.prewarm.enabled = true hive.prewarm.numcontainers=2 hive.server2.tez.initialize.default.sessions=true Thank you so much! great relief! On Tue, Mar 20, 2018 at 12:45 PM, Jyotsna G <[email protected]> wrote: > Thank you so much Jonathan for your response. I could point my finger at > the issue a bit more in detail now. Its with the way connection is obtained > to Hive (on Tez) and closing it. I need to tweak my Tez configurations with > respect to sessions. > > When executed with a single connection, it works perfectly fine and > returns the Json object to browser. > > *SINGLE CONNECTION* > > > > private static String driverName = "org.apache.hive.jdbc.HiveDriver"; > > private static String CONNECTION_URL= "jdbc:hive2://slc06ffs.us. > oracle.com:10000"; > > > > try { > > Class.forName(driverName); > > } catch (ClassNotFoundException e) { > > e.printStackTrace(); > > System.exit(1); > > } > > Connection con = DriverManager.getConnection(CONNECTION_URL, "", ""); > > Statement stmt = con.createStatement(); > > //EXECUTE ALL YOUR STATEMENTS HERE > > con.close(); > > > > > > > > The issue is with when they use pool of connections as below. Once > connection is obtained and query executed, results are not shown back on > browser and also hive server 2 refuses any further connections after that > single hung request (for which tez dag executes successfully) > > > > *POOL * > > > > And using the pool is like so: > > > > Connection con = HiveDatasource.getConnectionFromPool(); > > Statement stmt = con.createStatement(); > > // EXECUTE ALL YOUR QUERIES HERE > > con.close(); > > > Hive server 2 log when using a connection pool on closing the > connection(Attached the entire log of debug mode): > > *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > thrift.ThriftCLIService: Session disconnected without closing properly, > close it now* > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > session.SessionState: Removing resource dir /tmp/38896527-5822-4453-9918- > a5f36db09d1a_resources > *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > tez.TezSessionPoolManager: Closing tez session default? false* > *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > tez.TezSessionState: Closing Tez Session* > *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > client.TezClient: Shutting down Tez Session, > sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b, > applicationId=application_1521448362533_0101* > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client: > The ping interval is 60000 ms. > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client: > Connecting to slc03qzg/10.240.178.10:9032 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/ > 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye: > starting, having connections 1 > 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC > Client (368802354) connection to slc03qzg/10.240.178.10:9032 from > thirdeye sending #1811 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/ > 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got > value #1811 > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > ipc.ProtobufRpcEngine: Call: getApplicationReport took 5ms > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > client.TezClientUtils: Application not running, > applicationId=application_1521448362533_0101, > yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED, > trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A, > diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0, > killedDAGs=0 > > *18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > client.TezClient: Failed to shutdown Tez Session via proxy* > *org.apache.tez.dag.api.SessionNotRunning: Application not running, > applicationId=application_1521448362533_0101, > yarnApplicationState=FINISHED, finalApplicationStatus=SUCCEEDED, > trackingUrl=slc03qzg:9088/proxy/application_1521448362533_0101/A, > diagnostics=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0, > killedDAGs=0* > > at org.apache.tez.client.TezClientUtils.getSessionAMProxy( > TezClientUtils.java:798) > at org.apache.tez.client.TezClient.getSessionAMProxy(TezClient.java:633) > at org.apache.tez.client.TezClient.stop(TezClient.java:440) > at org.apache.hadoop.hive.ql.exec.tez.TezSessionState. > close(TezSessionState.java:257) > at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.close( > TezSessionPoolManager.java:185) > at org.apache.hadoop.hive.ql.session.SessionState.close( > SessionState.java:1482) > at org.apache.hive.service.cli.session.HiveSessionImpl.close( > HiveSessionImpl.java:567) > at org.apache.hive.service.cli.session.SessionManager. > closeSession(SessionManager.java:292) > at org.apache.hive.service.cli.CLIService.closeSession( > CLIService.java:237) > at org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext( > ThriftCLIService.java:122) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:300) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > client.TezClient: Could not connect to AM, killing session via YARN, > sessionName=HIVE-12d75efd-ec5d-4e9a-bfc7-3e3dbb48ee6b, > applicationId=application_1521448362533_0101 > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: INFO > impl.YarnClientImpl: Killing application application_1521448362533_0101 > 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC > Client (368802354) connection to slc03qzg/10.240.178.10:9032 from > thirdeye sending #1812 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc03qzg/ > 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye got > value #1812 > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > ipc.ProtobufRpcEngine: Call: forceKillApplication took 2ms > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > service.AbstractService: Service: > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl > entered state STOPPED > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client: > The ping interval is 60000 ms. > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG ipc.Client: > Connecting to slc06ffs/10.245.175.89:55310 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/ > 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye: > starting, having connections 2 > 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC > Client (368802354) connection to slc06ffs/10.245.175.89:55310 from > thirdeye sending #1813 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/ > 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got > value #1813 > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > ipc.ProtobufRpcEngine: Call: delete took 2ms > 18/03/19 23:48:33 [IPC Parameter Sending Thread #2]: DEBUG ipc.Client: IPC > Client (368802354) connection to slc06ffs/10.245.175.89:55310 from > thirdeye sending #1814 > 18/03/19 23:48:33 [IPC Client (368802354) connection to slc06ffs/ > 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye got > value #1814 > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > ipc.ProtobufRpcEngine: Call: delete took 0ms > 18/03/19 23:48:33 [HiveServer2-Handler-Pool: Thread-27]: DEBUG > cli.CLIService: SessionHandle [38896527-5822-4453-9918-a5f36db09d1a]: > closeSession() > 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/ > 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye: > closed > 18/03/19 23:48:43 [IPC Client (368802354) connection to slc03qzg/ > 10.240.178.10:9032 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc03qzg/10.240.178.10:9032 from thirdeye: > stopped, remaining connections 1 > 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/ > 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye: > closed > 18/03/19 23:48:43 [IPC Client (368802354) connection to slc06ffs/ > 10.245.175.89:55310 from thirdeye]: DEBUG ipc.Client: IPC Client > (368802354) connection to slc06ffs/10.245.175.89:55310 from thirdeye: > stopped, remaining connections 0 > > > > On Mon, Mar 19, 2018 at 10:22 PM, Jonathan Eagles <[email protected]> > wrote: > >> I'm glad you checked this with an MR run. That help clarify the issue, >> ruling out the browser compatibility issue. As Tez is an application >> framework, it is not responsible for returning results to the browser. >> Using Hive CLI, it is the Hive CLI that is responsible for displaying the >> results of the query. For HiveServer2, it is HiveServer2's responsibility >> of displaying the results. It is true that the yarn and tez configurations >> for Hive CLI and HiveServer2 can be different, but it is not the >> responsibility of those settings to change how Tez returns results, as it >> is Hive CLI and HiveServer2 that returns results. >> >> Do you have access to the support logs or to a support team that can >> retrieve the relevant parts of those logs that will help to investigate >> this further? The relevant Tez logs you have provided are helpful, but >> only to show that Tez is working correctly. Without having access to the >> the HiveServer2 or thirdeye-services logs, I'm afraid it will be very >> difficult to debug this issue further. >> >> On Mon, Mar 19, 2018 at 11:22 AM, Jyotsna G <[email protected]> wrote: >> >>> Thank you for your response Jonathan. >>> >>> When I change the execution engine to MR, results are fetched perfectly >>> for the rest query, it's only when I change the execution engine to tez >>> that I see the issue occurring. >>> >>> To rephrase my question, are there any further sessions specific >>> configurations for tez that I might have missed? Any yarn related >>> configurations for tez that I might have to look for? >>> >>> >>> >>> On Mon, Mar 19, 2018, 21:37 Jonathan Eagles <[email protected]> wrote: >>> >>>> Thanks for reaching out to us, Jyotsna. >>>> >>>> Let me make sure I understand what your setup and the issue you are >>>> experiencing. It sounds like the setup looks like this. >>>> >>>> Browser -> Third party service "thirdeye-services" -> HiveServer2 -> >>>> Tez >>>> >>>> Since you are showing logs and screenshots of the successful Tez Job. >>>> My initial thought is that HiveServer2 was able to successfully and >>>> correctly get the results from Tez. That would make me believe that either >>>> the connection between the browser to the third party service >>>> "thirdeye-services" isn't working (perhaps browser compatibility issue, >>>> security issue, or service issue) or the connection between the third party >>>> service "thirdeye-services" to HiveServer2 issue (issue retrieving results >>>> from HiveServer2). >>>> >>>> Since our expertise is only with the Tez portion of the setup and that >>>> the Tez portion of the setup seems to be working, the Tez community won't >>>> be the best at addressing this issue. My recommendation would be try the >>>> query in a different browser as a simple test (Chrome, Firefox) checking >>>> with third party "thirdeye-services" browser compatibility. If you have >>>> access to the thirdparty-services logs or logs to the HiveServer2 logs >>>> those will be helpful in debugging. When reaching out to the >>>> thirdeye-services support or HiveServer2 please reference these logs to get >>>> the best support from the community. The hive user list is >>>> [email protected] and I have don't have a support contact for the >>>> thirdparty software thirdeye-services that seems to be part of this setup. >>>> >>>> Thanks again for reaching to the Tez Community. >>>> >>>> Regards, >>>> jeagles >>>> >>>> On Mon, Mar 19, 2018 at 6:47 AM, Jyotsna G <[email protected]> wrote: >>>> >>>>> Hi, >>>>> We are making a rest URI request which hits Hive server 2 and >>>>> translates in to a hive query on TEZ. >>>>> >>>>> Sample uri: >>>>> slc00yeb.us.oracle.com:8989/thirdeye-services/v1.0/OSCS/quer >>>>> y/OSCSRecords.json?COMP_FAMILY=FMW&COMP_NAME=WLS&COMPTYPE1_F >>>>> IRST=RC1&COMPTYPE1_SECOND=RC2&COMPTYPE2_FIRST=manifest.publi >>>>> sh.label.WLS_GENERIC:carb-manifest:12.2.1.3.0-170722. >>>>> 1880245:pom&COMPTYPE2_SECOND=manifest.publish.label.WLS_ >>>>> GENERIC:carb-manifest:12.2.1.3.0-170726.1880744:pom&RELEASE >>>>> _FIRST=12.2.1.3.0&RELEASE_SECOND=12.2.1.3.0&COMPTYPE=MANIFEST >>>>> >>>>> Hadoop version : 2.4.0 >>>>> Hive:1.2.2 >>>>> Tez : 5.4.0 >>>>> >>>>> All my queries run fine in HIVE CLI , also via standalone program >>>>> connecting to hive via JDBC ODBC connection and returns hive query >>>>> results, >>>>> but only when I hit the rest URL, results are not returned. >>>>> >>>>> I have enabled the below settings for tez >>>>> >>>>> tez.session.am.dag.submit.timeout.secs 90 >>>>> >>>>> tez.am.mode.session : true >>>>> tez.am.container.reuse.enable: true >>>>> tez.allow.disabled.timeline-domains : true >>>>> tez.yarn.ats.enabled : true >>>>> tez.use.cluster.hadoop-libs : true >>>>> >>>>> The url on the browser keeps spinning without the json response, even >>>>> after the tez job on yarn cluster has finished successfully. Attached Tez >>>>> job diagnosis screenshot. >>>>> >>>>> From the dag log : >>>>> >>>>> 2018-03-19 11:34:36,367 INFO [AsyncDispatcher event handler] >>>>> impl.DAGImpl: dag_1521448362533_0019_1 transitioned from RUNNING to >>>>> SUCCEEDED >>>>> 2018-03-19 11:34:36,368 INFO [AsyncDispatcher event handler] >>>>> app.DAGAppMaster: DAG completed, dagId=dag_1521448362533_0019_1, >>>>> dagState=SUCCEEDED >>>>> >>>>> {"entity":"dag_1521448362533_0019_1","entitytype":"TEZ_DAG_I >>>>> D","events":[{"ts":1521459276335,"eventtype":"DAG_FINISHED"} >>>>> ],"otherinfo":{"startTime":1521459269619,"endTime":152145927 >>>>> 6335,"timeTaken":6716,"status":"SUCCEEDED","diagnostics":"", >>>>> "counters":{"counterGroups":[{"counterGroupName":"org. >>>>> apache.tez.common.counters.DAGCounter","counterGroupDispl >>>>> ayName":"org.apache.tez.common.counters.DAGCounter"," >>>>> counters":[{"counterName":"NUM_SUCCEEDED_TASKS","counterDisp >>>>> layName":"NUM_SUCCEEDED_TASKS","counterValue":2},{" >>>>> counterName":"TOTAL_LAUNCHED_TASKS","counterDisplayName":" >>>>> TOTAL_LAUNCHED_TASKS","counterValue":2},{"counterName":"DATA_LOCAL_ >>>>> TASKS","counterDisplayName":"DATA_LOCAL_TASKS","counterValue":1}]},{" >>>>> counterGroupName":"org.apache.tez.common.counters.FileSystem >>>>> Counter","counterGroupDisplayName":"File System >>>>> Counters","counters":[{"counterName":"FILE_BYTES_READ","coun >>>>> terDisplayName":"FILE_BYTES_READ","counterValue":41993},{" >>>>> counterName":"FILE_BYTES_WRITTEN","counterDisplayName":" >>>>> FILE_BYTES_WRITTEN","counterValue":83954},{"counterName":" >>>>> FILE_READ_OPS","counterDisplayName":"FILE_READ_OPS"," >>>>> counterValue":0},{"counterName":"FILE_LARGE_READ_OPS"," >>>>> counterDisplayName":"FILE_LARGE_READ_OPS","counterValue" >>>>> :0},{"counterName":"FILE_WRITE_OPS","counterDisplayName":" >>>>> FILE_WRITE_OPS","counterValue":0},{"counterName":"HDFS_BYTES_READ"," >>>>> counterDisplayName":"HDFS_BYTES_READ","counterValue":59231}, >>>>> {"counterName":"HDFS_BYTES_WRITTEN","counterDisplayName": >>>>> "HDFS_BYTES_WRITTEN","counterValue":39631},{" >>>>> counterName":"HDFS_READ_OPS","counterDisplayName":"HDFS_ >>>>> READ_OPS","counterValue":5},{"counterName":"HDFS_LARGE_READ_ >>>>> OPS","counterDisplayName":"HDFS_LARGE_READ_OPS","counterValue":0},{" >>>>> counterName":"HDFS_WRITE_OPS","counterDisplayName":"HDFS_ >>>>> WRITE_OPS","counterValue":2}]},{"counterGroupName":"org. >>>>> apache.tez.common.counters.TaskCounter"," >>>>> counterGroupDisplayName":"org.apache.tez.common.counters. >>>>> TaskCounter","counters":[{"counterName":"REDUCE_INPUT_ >>>>> GROUPS","counterDisplayName":"REDUCE_INPUT_GROUPS"," >>>>> counterValue":190},{"counterName":"REDUCE_INPUT_ >>>>> RECORDS","counterDisplayName":"REDUCE_INPUT_RECORDS"," >>>>> counterValue":191},{"counterName":"COMBINE_INPUT_ >>>>> RECORDS","counterDisplayName":"COMBINE_INPUT_RECORDS"," >>>>> counterValue":0},{"counterName":"SPILLED_RECORDS" >>>>> ,"counterDisplayName":"SPILLED_RECORDS","counterValue >>>>> ":382},{"counterName":"NUM_SHUFFLED_INPUTS","counterDisplayN >>>>> ame":"NUM_SHUFFLED_INPUTS","counterValue":1},{" >>>>> counterName":"NUM_SKIPPED_INPUTS","counterDisplayName":" >>>>> NUM_SKIPPED_INPUTS","counterValue":0},{"counterName":"NUM_FAILED_ >>>>> SHUFFLE_INPUTS","counterDisplayName":"NUM_FAILED_SHUFFLE_ >>>>> INPUTS","counterValue":0},{"counterName":"MERGED_MAP_ >>>>> OUTPUTS","counterDisplayName":"MERGED_MAP_OUTPUTS"," >>>>> counterValue":1},{"counterName":"GC_TIME_MILLIS", >>>>> "counterDisplayName":"GC_TIME_MILLIS","counterValue":84},{" >>>>> counterName":"CPU_MILLISECONDS","counterDisplayName":"CPU_ >>>>> MILLISECONDS","counterValue":5520},{"counterName":" >>>>> PHYSICAL_MEMORY_BYTES","counterDisplayName":"PHYSICAL_ >>>>> MEMORY_BYTES","counterValue":661254144},{"counterName":" >>>>> VIRTUAL_MEMORY_BYTES","counterDisplayName":"VIRTUAL_ >>>>> MEMORY_BYTES","counterValue":4294270976},{"counterName":" >>>>> COMMITTED_HEAP_BYTES","counterDisplayName":"COMMITTED >>>>> _HEAP_BYTES","counterValue":402653184},{"counterName":" >>>>> INPUT_RECORDS_PROCESSED","counterDisplayName":"INPUT_RECORDS >>>>> _PROCESSED","counterValue":8445},{"counterName":"OUTPUT_RECO >>>>> RDS","counterDisplayName":"OUTPUT_RECORDS","counterValue" >>>>> :191},{"counterName":"OUTPUT_BYTES","counterDisplayName":" >>>>> OUTPUT_BYTES","counterValue":41350},{"counterName":"OUTPUT_ >>>>> BYTES_WITH_OVERHEAD","counterDisplayName":"OUTPUT_ >>>>> BYTES_WITH_OVERHEAD","counterValue":41957},{" >>>>> counterName":"OUTPUT_BYTES_PHYSICAL","counterDisplayName" >>>>> :"OUTPUT_BYTES_PHYSICAL","counterValue":41961},{" >>>>> counterName":"ADDITIONAL_SPILLS_BYTES_WRITTEN","counterDisplayName":" >>>>> ADDITIONAL_SPILLS_BYTES_WRITTEN","counterValue":41961},{"cou >>>>> nterName":"ADDITIONAL_SPILLS_BYTES_READ","counterDisplayNam >>>>> e":"ADDITIONAL_SPILLS_BYTES_READ","counterValue":41961},{"co >>>>> unterName":"ADDITIONAL_SPILL_COUNT","counterDisplayName":"A >>>>> DDITIONAL_SPILL_COUNT","counterValue":0},{"counterName":"SHU >>>>> FFLE_BYTES","counterDisplayName":"SHUFFLE_BYTES","counterVal >>>>> ue":41961},{"counterName":"SHUFFLE_BYTES_DECOMPRESSED"," >>>>> counterDisplayName":"SHUFFLE_BYTES_DECOMPRESSED"," >>>>> counterValue":41957},{"counterName":"SHUFFLE_BYTES_ >>>>> TO_MEM","counterDisplayName":"SHUFFLE_BYTES_TO_MEM"," >>>>> counterValue":41961},{"counterName":"SHUFFLE_BYTES_ >>>>> TO_DISK","counterDisplayName":"SHUFFLE_BYTES_TO_DISK"," >>>>> counterValue":0},{"counterName":"SHUFFLE_BYTES_ >>>>> DISK_DIRECT","counterDisplayName":"SHUFFLE_BYTES_DISK_DIRECT >>>>> ","counterValue":0},{"counterName":"NUM_MEM_TO_DISK_MERGES", >>>>> "counterDisplayName":"NUM_MEM_TO_DISK_MERGES","counterValue" >>>>> :0},{"counterName":"NUM_DISK_TO_DISK_MERGES","counterDispla >>>>> yName":"NUM_DISK_TO_DISK_MERGES","counterValue":0}]},{"count >>>>> erGroupName":"HIVE","counterGroupDisplayName":"HIVE","counte >>>>> rs":[{"counterName":"CREATED_FILES","counterDisplayName":" >>>>> CREATED_FILES","counterValue":1},{"counterName":" >>>>> DESERIALIZE_ERRORS","counterDisplayName":"DESERIALIZE_ >>>>> ERRORS","counterValue":0},{"counterName":"RECORDS_IN_Map_1 >>>>> ","counterDisplayName":"RECORDS_I >>>>> >>>>> >>>>> But the json response isnt coming. What am I missing here? >>>>> >>>>> Thanks, >>>>> Jyotsna >>>>> >>>>> >>>>> >>>> >> >
