Check the peer address you specified in the command line. It does not seem to match your remote cluster ZK parent node.
Jerry On Thursday, September 8, 2016, Frank Luo <j...@merkleinc.com> wrote: > I don't think they are pointing to different locations. Both of them > should be /hbase-secure. > > However, the debugger shows that ConnectionManager#retrieveClusterId are > called twice, the first time regards to the source cluster, which works > fine, and > watcher.clusterIdZNode=/hbase-secure/hbaseid, and it is correct. > > The second time for the remote cluster, watcher.clusterIdZNode=/hbase/hbaseid, > which should be incorrect. > > What I am suspecting is ZooKeeperWatcher, method setNodeNames. It reads: > > private void setNodeNames(Configuration conf) { > baseZNode = conf.get(HConstants.ZOOKEEPER_ZNODE_PARENT, > HConstants.DEFAULT_ZOOKEEPER_ZNODE_PARENT); > > I am not sure the conf is corrected fetched from the remote cluster. If > not, the default value is given, which is /hbase and incorrect. > > By the way, below is the root znodes for zookeepers: > > The source cluster: > [hbase-secure, hiveserver2, hive, hbase-unsecure, templeton-hadoop, > hadoop-ha, zookeeper] > > The target cluster: > [hbase-secure, hive, hiveserver2, hbase-unsecure, hadoop-ha, zookeeper] > > -----Original Message----- > From: Esteban Gutierrez [mailto:este...@cloudera.com <javascript:;>] > Sent: Thursday, September 08, 2016 1:02 PM > To: user@hbase.apache.org <javascript:;> > Subject: Re: CopyTable fails on copying between two secured clusters > > Is it possible that in your destination cluster zookeeper.znode.parent > points to a different location than /hbase ? If both clusters are under the > same kerberos realm then there is no need to worry about > zookeeper.security.auth_to_local. > > > > -- > Cloudera, Inc. > > > On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <j...@merkleinc.com > <javascript:;>> wrote: > > > Thanks Esteban for replying. > > > > The Kerberos realm is shared between the two clusters. > > > > I searched zookeeper config and couldn't find the rule, so where it > > is set? > > > > Having said that, I looked at parameters passed to getData call, and > > it doesn't look like security related. > > > > PS. I am using hbase 1.1.2. > > > > Here is the log: > > > > com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyT > > able Connected to the target VM, address: '127.0.0.1:50669', > > transport: > > 'socket' > > 0 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - > > hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 247 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to > > load native-hadoop library for your platform... using builtin-java > > classes where applicable > > 1492 [main] INFO org.apache.hadoop.security.UserGroupInformation - > > Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using > > keytab file \Users\jluo\tidsvc_exphbase.keytab > > 11462 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 164493 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 167720 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 172108 [main] WARN > > org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory > > - The short-circuit local reads feature cannot be used because UNIX > > Domain sockets are not available on Windows. > > 224735 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 225165 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper > > ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > 225190 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 > > GMT > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > host.name=JLUOWIN7.pclc0.merkle.local > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.version=1.7.0_51 > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.vendor=Oracle Corporation > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.home=C:\work\jdk1.7.0_51\jre > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\ > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.compiler=<NA> > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > os.name=Windows 7 > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:os.arch=amd64 > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:os.version=6.1 > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > user.name=jluo > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:user.home=C:\Users\jluo > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance > > 225197 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181, > > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0, > > quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, > > baseZNode=/hbase-secure > > 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to > > authenticate using SASL (unknown error) > > 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session > > 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = > > 0x357067c5442410e, negotiated timeout = 120000 > > 231758 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper > > ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > 231758 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181, > > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0, > > quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, > > baseZNode=/hbase-secure > > 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to > > authenticate using SASL (unknown error) > > 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session > > 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = > > 0x357067c54424116, negotiated timeout = 120000 > > 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Client session timed out, have not > > heard from server in 96993ms for sessionid 0x357067c5442410e, closing > > socket connection and attempting reconnect > > 374923 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x357067c54424116 closed > > 374924 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to > > authenticate using SASL (unknown error) > > 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session > > 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid = > > 0x357067c5442410e, negotiated timeout = 120000 > > 376538 [main] INFO > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa > > tion > > - Closing zookeeper sessionid=0x357067c5442410e > > 376581 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x357067c5442410e closed > > 376581 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 376746 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 376749 [main] INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > 376839 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 376841 [main] INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > 376843 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper > > ensemble=hqetghdpadm91.pclc0.merkle.local:2181, > hqetghdpadm90.pclc0.merkle. > > local:2181 > > 376843 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, > > connectString=hqetghdpadm91.pclc0.merkle.local:2181, > > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 > > watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0. > > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, > > baseZNode=/hbase > > 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt > > to authenticate using SASL (unknown error) > > 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session > > 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid = > > 0x2565c552d296982, negotiated timeout = 40000 > > 377140 [main] INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry > > - ClusterId read in ZooKeeper is null > > 377141 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper > > ensemble=hqetghdpadm91.pclc0.merkle.local:2181, > hqetghdpadm90.pclc0.merkle. > > local:2181 > > 377141 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, > > connectString=hqetghdpadm91.pclc0.merkle.local:2181, > > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 > > watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0. > > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, > > baseZNode=/hbase > > 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt > > to authenticate using SASL (unknown error) > > 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session > > 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid = > > 0x1565c552cfd6b03, negotiated timeout = 40000 > > 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Client session timed out, have not > > heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing > > socket connection and attempting reconnect > > 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Client session timed out, have not > > heard from server in 636900ms for sessionid 0x2565c552d296982, closing > > socket connection and attempting reconnect > > 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to server > > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt > > to authenticate using SASL (unknown error) > > 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session > > 1236762 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x1565c552cfd6b03 closed > > 1236762 [main] INFO > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa > > tion > > - Closing zookeeper sessionid=0x2565c552d296982 > > 1236762 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 1236908 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x2565c552d296982 closed > > 1236909 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > > > java.io.IOException: Failed to get cluster ID > > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > getAuthToken(TokenUtil.java:365) > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > addTokenForJob(TokenUtil.java:327) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initCredentials(TableMapReduceUtil.java:451) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initTableReducerJob(TableMapReduceUtil.java:658) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initTableReducerJob(TableMapReduceUtil.java:591) > > at org.apache.hadoop.hbase.mapreduce.CopyTable. > > createSubmittableJob(CopyTable.java:168) > > at org.apache.hadoop.hbase.mapreduce.CopyTable.run( > > CopyTable.java:348) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > at org.apache.hadoop.hbase.mapreduce.CopyTable.main( > > CopyTable.java:341) > > at com.merkleinc.cr.hbase_maintenance.tableexport. > > CopyTableTest.export(CopyTableTest.java:44) > > at com.merkleinc.cr.hbase_maintenance.tableexport. > > CopyTableTest.testCopyTable(CopyTableTest.java:25) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:57) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > FrameworkMethod.java:50) > > at org.junit.internal.runners.model.ReflectiveCallable.run( > > ReflectiveCallable.java:12) > > at org.junit.runners.model.FrameworkMethod.invokeExplosively( > > FrameworkMethod.java:47) > > at org.junit.internal.runners.statements.InvokeMethod. > > evaluate(InvokeMethod.java:17) > > at com.intellij.junit4.JUnit4TestRunnerUtil$ > > IgnoreIgnoredTestJUnit4ClassRunner.runChild( > JUnit4TestRunnerUtil.java:349) > > at org.junit.runners.BlockJUnit4ClassRunner.runChild( > > BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > > at org.junit.runners.ParentRunner$1.schedule( > ParentRunner.java:71) > > at org.junit.runners.ParentRunner.runChildren( > > ParentRunner.java:288) > > at org.junit.runners.ParentRunner.access$000( > ParentRunner.java:58) > > at org.junit.runners.ParentRunner$2.evaluate( > > ParentRunner.java:268) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > > at > > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs( > > JUnit4IdeaTestRunner.java:69) > > at com.intellij.rt.execution.junit.JUnitStarter. > > prepareStreamsAndStart(JUnitStarter.java:234) > > at com.intellij.rt.execution.junit.JUnitStarter.main( > > JUnitStarter.java:74) > > > > Disconnected from the target VM, address: '127.0.0.1:50669', transport: > > 'socket' > > > > Process finished with exit code -1 > > > > -----Original Message----- > > From: Esteban Gutierrez [mailto:este...@cloudera.com <javascript:;>] > > Sent: Thursday, September 08, 2016 12:37 PM > > To: user@hbase.apache.org <javascript:;> > > Subject: Re: CopyTable fails on copying between two secured clusters > > > > Frank, > > > > Do you have a cross realm setup? if yes, do you have ZK to use the > > proper rules via the -Dzookeeper.security.auth_to_local flag? If you > > could share additional logs that would be helpful for us. > > > > Thanks! > > esteban. > > > > -- > > Cloudera, Inc. > > > > > > On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <j...@merkleinc.com > <javascript:;>> wrote: > > > > > I couldn’t manage to get the CopyTable to work between two secured > > > clusters and hope someone can shed some light. > > > > > > I have a table created on both clusters, and I am running the > > > CopyTable command from the source cluster. > > > > > > The parameters are the following: > > > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase", > > > "--new.name=TargetTable", "SourceTable" > > > > > > I have ensured that the user has all permissions on both tables. > > > When I run it, I got: > > > > > > ===================== > > > java.io.IOException: Failed to get cluster ID > > > > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > > getAuthToken(TokenUtil.java:365) > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > > addTokenForJob(TokenUtil.java:327) > > > ===================== > > > > > > Debugging into it, I found that it is trying to get the clusterId > > > from my target cluster using address of zookeeper1/2, and the code > > > below returns > > > null: > > > > > > > > > ===================== > > > > > > ZKUtil.getData(watcher, watcher.clusterIdZNode); > > > ===================== > > > > > > Any hints please? > > > > > > > > > > > > Access the Q2 2016 Digital Marketing Report for a fresh set of > > > trends and benchmarks in digital > > > marketing<http://www2.merkleinc.com/l/47252/2016-07- > > > 26/47gt7c> > > > > > > Download our latest report titled “The Case for Change: Exploring > > > the Myths of Customer-Centric Transformation” > > > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> > > > > > > This email and any attachments transmitted with it are intended for > > > use by the intended recipient(s) only. If you have received this > > > email in error, please notify the sender immediately and then delete > > > it. If you are not the intended recipient, you must not keep, use, > > > disclose, copy or distribute this email without the author’s prior > permission. > > > We take precautions to minimize the risk of transmitting software > > > viruses, but we advise you to perform your own virus checks on any > > > attachment to this message. We cannot accept liability for any loss > > > or damage caused by software viruses. The information contained in > > > this communication may be confidential and may be subject to the > > attorney-client privilege. > > > > > Access the Q2 2016 Digital Marketing Report for a fresh set of trends > > and benchmarks in digital > > marketing<http://www2.merkleinc.com/l/47252/2016-07- > > 26/47gt7c> > > > > Download our latest report titled “The Case for Change: Exploring the > > Myths of Customer-Centric Transformation” > > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email and > > any attachments transmitted with it are intended for use by the > > intended recipient(s) only. If you have received this email in error, > > please notify the sender immediately and then delete it. If you are > > not the intended recipient, you must not keep, use, disclose, copy or > > distribute this email without the author’s prior permission. We take > > precautions to minimize the risk of transmitting software viruses, but > > we advise you to perform your own virus checks on any attachment to > > this message. We cannot accept liability for any loss or damage caused > > by software viruses. The information contained in this communication > > may be confidential and may be subject to the attorney-client privilege. > > > Access the Q2 2016 Digital Marketing Report for a fresh set of trends and > benchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07- > 26/47gt7c> > > Download our latest report titled “The Case for Change: Exploring the > Myths of Customer-Centric Transformation” <http://www2.merkleinc.com/l/ > 47252/2016-08-04/4b9p7c> > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. >