I know I have given a dummy zookeeper name for sake of simplicity. But the actual address should be correct. Otherwise I will be receiving a different error.
-----Original Message----- From: Jerry He [mailto:jerry...@gmail.com] Sent: Thursday, September 08, 2016 6:57 PM To: user@hbase.apache.org Subject: Re: CopyTable fails on copying between two secured clusters Check the peer address you specified in the command line. It does not seem to match your remote cluster ZK parent node. Jerry On Thursday, September 8, 2016, Frank Luo <j...@merkleinc.com> wrote: > I don't think they are pointing to different locations. Both of them > should be /hbase-secure. > > However, the debugger shows that ConnectionManager#retrieveClusterId > are called twice, the first time regards to the source cluster, which > works fine, and watcher.clusterIdZNode=/hbase-secure/hbaseid, and it > is correct. > > The second time for the remote cluster, > watcher.clusterIdZNode=/hbase/hbaseid, > which should be incorrect. > > What I am suspecting is ZooKeeperWatcher, method setNodeNames. It reads: > > private void setNodeNames(Configuration conf) { > baseZNode = conf.get(HConstants.ZOOKEEPER_ZNODE_PARENT, > HConstants.DEFAULT_ZOOKEEPER_ZNODE_PARENT); > > I am not sure the conf is corrected fetched from the remote cluster. > If not, the default value is given, which is /hbase and incorrect. > > By the way, below is the root znodes for zookeepers: > > The source cluster: > [hbase-secure, hiveserver2, hive, hbase-unsecure, templeton-hadoop, > hadoop-ha, zookeeper] > > The target cluster: > [hbase-secure, hive, hiveserver2, hbase-unsecure, hadoop-ha, > zookeeper] > > -----Original Message----- > From: Esteban Gutierrez [mailto:este...@cloudera.com <javascript:;>] > Sent: Thursday, September 08, 2016 1:02 PM > To: user@hbase.apache.org <javascript:;> > Subject: Re: CopyTable fails on copying between two secured clusters > > Is it possible that in your destination cluster zookeeper.znode.parent > points to a different location than /hbase ? If both clusters are > under the same kerberos realm then there is no need to worry about > zookeeper.security.auth_to_local. > > > > -- > Cloudera, Inc. > > > On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <j...@merkleinc.com > <javascript:;>> wrote: > > > Thanks Esteban for replying. > > > > The Kerberos realm is shared between the two clusters. > > > > I searched zookeeper config and couldn't find the rule, so where it > > is set? > > > > Having said that, I looked at parameters passed to getData call, and > > it doesn't look like security related. > > > > PS. I am using hbase 1.1.2. > > > > Here is the log: > > > > com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCop > > yT able Connected to the target VM, address: '127.0.0.1:50669', > > transport: > > 'socket' > > 0 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil - > > hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 247 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable > > to load native-hadoop library for your platform... using > > builtin-java classes where applicable > > 1492 [main] INFO org.apache.hadoop.security.UserGroupInformation - > > Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using > > keytab file \Users\jluo\tidsvc_exphbase.keytab > > 11462 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 164493 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 167720 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 172108 [main] WARN > > org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory > > - The short-circuit local reads feature cannot be used because UNIX > > Domain sockets are not available on Windows. > > 224735 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 225165 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper > > ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > 225190 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 > > 09:09 GMT > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > host.name=JLUOWIN7.pclc0.merkle.local > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.version=1.7.0_51 > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.vendor=Oracle Corporation > > 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.home=C:\work\jdk1.7.0_51\jre > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\ > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:java.compiler=<NA> > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > os.name=Windows 7 > > 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:os.arch=amd64 > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:os.version=6.1 > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: > > user.name=jluo > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:user.home=C:\Users\jluo > > 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client > > environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance > > 225197 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181, > > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:218 > > 1 > > sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0, > > quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, > > baseZNode=/hbase-secure > > 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session > > 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = > > 0x357067c5442410e, negotiated timeout = 120000 > > 231758 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper > > ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 > > 231758 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181, > > hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:218 > > 1 > > sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0, > > quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02. > > pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181, > > baseZNode=/hbase-secure > > 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session > > 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid = > > 0x357067c54424116, negotiated timeout = 120000 > > 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Client session timed out, have > > not heard from server in 96993ms for sessionid 0x357067c5442410e, > > closing socket connection and attempting reconnect > > 374923 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x357067c54424116 closed > > 374924 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session > > 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid = > > 0x357067c5442410e, negotiated timeout = 120000 > > 376538 [main] INFO > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplemen > > ta > > tion > > - Closing zookeeper sessionid=0x357067c5442410e > > 376581 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x357067c5442410e closed > > 376581 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 376746 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 376749 [main] INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > 376839 [main] WARN > > org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil > > - hbase.regionserver.global.memstore.upperLimit is deprecated by > > hbase.regionserver.global.memstore.size > > 376841 [main] INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > 376843 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper > > ensemble=hqetghdpadm91.pclc0.merkle.local:2181, > hqetghdpadm90.pclc0.merkle. > > local:2181 > > 376843 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, > > connectString=hqetghdpadm91.pclc0.merkle.local:2181, > > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 > > watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0. > > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, > > baseZNode=/hbase > > 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating > > session > > 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid > > = 0x2565c552d296982, negotiated timeout = 40000 > > 377140 [main] INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry > > - ClusterId read in ZooKeeper is null > > 377141 [main] INFO > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper > > - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper > > ensemble=hqetghdpadm91.pclc0.merkle.local:2181, > hqetghdpadm90.pclc0.merkle. > > local:2181 > > 377141 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating > > client connection, > > connectString=hqetghdpadm91.pclc0.merkle.local:2181, > > hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000 > > watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0. > > merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181, > > baseZNode=/hbase > > 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Socket connection established to > > hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating > > session > > 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO > > org.apache.zookeeper.ClientCnxn - Session establishment complete on > > server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid > > = 0x1565c552cfd6b03, negotiated timeout = 40000 > > 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] > > INFO org.apache.zookeeper.ClientCnxn - Client session timed out, > > have not heard from server in 636835ms for sessionid > > 0x1565c552cfd6b03, closing socket connection and attempting > > reconnect > > 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] > > INFO org.apache.zookeeper.ClientCnxn - Client session timed out, > > have not heard from server in 636900ms for sessionid > > 0x2565c552d296982, closing socket connection and attempting > > reconnect > > 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] > > INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to > > server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] > > INFO org.apache.zookeeper.ClientCnxn - Socket connection > > established to hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, > > initiating session > > 1236762 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x1565c552cfd6b03 closed > > 1236762 [main] INFO > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplemen > > ta > > tion > > - Closing zookeeper sessionid=0x2565c552d296982 > > 1236762 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > 1236908 [main] INFO org.apache.zookeeper.ZooKeeper - Session: > > 0x2565c552d296982 closed > > 1236909 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - > > EventThread shut down > > > > java.io.IOException: Failed to get cluster ID > > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > getAuthToken(TokenUtil.java:365) > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > addTokenForJob(TokenUtil.java:327) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initCredentials(TableMapReduceUtil.java:451) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initTableReducerJob(TableMapReduceUtil.java:658) > > at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil. > > initTableReducerJob(TableMapReduceUtil.java:591) > > at org.apache.hadoop.hbase.mapreduce.CopyTable. > > createSubmittableJob(CopyTable.java:168) > > at org.apache.hadoop.hbase.mapreduce.CopyTable.run( > > CopyTable.java:348) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > at org.apache.hadoop.hbase.mapreduce.CopyTable.main( > > CopyTable.java:341) > > at com.merkleinc.cr.hbase_maintenance.tableexport. > > CopyTableTest.export(CopyTableTest.java:44) > > at com.merkleinc.cr.hbase_maintenance.tableexport. > > CopyTableTest.testCopyTable(CopyTableTest.java:25) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:57) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > > FrameworkMethod.java:50) > > at org.junit.internal.runners.model.ReflectiveCallable.run( > > ReflectiveCallable.java:12) > > at > > org.junit.runners.model.FrameworkMethod.invokeExplosively( > > FrameworkMethod.java:47) > > at org.junit.internal.runners.statements.InvokeMethod. > > evaluate(InvokeMethod.java:17) > > at com.intellij.junit4.JUnit4TestRunnerUtil$ > > IgnoreIgnoredTestJUnit4ClassRunner.runChild( > JUnit4TestRunnerUtil.java:349) > > at org.junit.runners.BlockJUnit4ClassRunner.runChild( > > BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > > at org.junit.runners.ParentRunner$1.schedule( > ParentRunner.java:71) > > at org.junit.runners.ParentRunner.runChildren( > > ParentRunner.java:288) > > at org.junit.runners.ParentRunner.access$000( > ParentRunner.java:58) > > at org.junit.runners.ParentRunner$2.evaluate( > > ParentRunner.java:268) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > > at > > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs( > > JUnit4IdeaTestRunner.java:69) > > at com.intellij.rt.execution.junit.JUnitStarter. > > prepareStreamsAndStart(JUnitStarter.java:234) > > at com.intellij.rt.execution.junit.JUnitStarter.main( > > JUnitStarter.java:74) > > > > Disconnected from the target VM, address: '127.0.0.1:50669', transport: > > 'socket' > > > > Process finished with exit code -1 > > > > -----Original Message----- > > From: Esteban Gutierrez [mailto:este...@cloudera.com <javascript:;>] > > Sent: Thursday, September 08, 2016 12:37 PM > > To: user@hbase.apache.org <javascript:;> > > Subject: Re: CopyTable fails on copying between two secured clusters > > > > Frank, > > > > Do you have a cross realm setup? if yes, do you have ZK to use the > > proper rules via the -Dzookeeper.security.auth_to_local flag? If you > > could share additional logs that would be helpful for us. > > > > Thanks! > > esteban. > > > > -- > > Cloudera, Inc. > > > > > > On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <j...@merkleinc.com > <javascript:;>> wrote: > > > > > I couldn’t manage to get the CopyTable to work between two secured > > > clusters and hope someone can shed some light. > > > > > > I have a table created on both clusters, and I am running the > > > CopyTable command from the source cluster. > > > > > > The parameters are the following: > > > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase", > > > "--new.name=TargetTable", "SourceTable" > > > > > > I have ensured that the user has all permissions on both tables. > > > When I run it, I got: > > > > > > ===================== > > > java.io.IOException: Failed to get cluster ID > > > > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > > getAuthToken(TokenUtil.java:365) > > > at org.apache.hadoop.hbase.security.token.TokenUtil. > > > addTokenForJob(TokenUtil.java:327) > > > ===================== > > > > > > Debugging into it, I found that it is trying to get the clusterId > > > from my target cluster using address of zookeeper1/2, and the code > > > below returns > > > null: > > > > > > > > > ===================== > > > > > > ZKUtil.getData(watcher, watcher.clusterIdZNode); > > > ===================== > > > > > > Any hints please? > > > > > > > > > > > > Access the Q2 2016 Digital Marketing Report for a fresh set of > > > trends and benchmarks in digital > > > marketing<http://www2.merkleinc.com/l/47252/2016-07- > > > 26/47gt7c> > > > > > > Download our latest report titled “The Case for Change: Exploring > > > the Myths of Customer-Centric Transformation” > > > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> > > > > > > This email and any attachments transmitted with it are intended > > > for use by the intended recipient(s) only. If you have received > > > this email in error, please notify the sender immediately and then > > > delete it. If you are not the intended recipient, you must not > > > keep, use, disclose, copy or distribute this email without the > > > author’s prior > permission. > > > We take precautions to minimize the risk of transmitting software > > > viruses, but we advise you to perform your own virus checks on any > > > attachment to this message. We cannot accept liability for any > > > loss or damage caused by software viruses. The information > > > contained in this communication may be confidential and may be > > > subject to the > > attorney-client privilege. > > > > > Access the Q2 2016 Digital Marketing Report for a fresh set of > > trends and benchmarks in digital > > marketing<http://www2.merkleinc.com/l/47252/2016-07- > > 26/47gt7c> > > > > Download our latest report titled “The Case for Change: Exploring > > the Myths of Customer-Centric Transformation” > > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email > > and any attachments transmitted with it are intended for use by the > > intended recipient(s) only. If you have received this email in > > error, please notify the sender immediately and then delete it. If > > you are not the intended recipient, you must not keep, use, > > disclose, copy or distribute this email without the author’s prior > > permission. We take precautions to minimize the risk of transmitting > > software viruses, but we advise you to perform your own virus checks > > on any attachment to this message. We cannot accept liability for > > any loss or damage caused by software viruses. The information > > contained in this communication may be confidential and may be subject to > > the attorney-client privilege. > > > Access the Q2 2016 Digital Marketing Report for a fresh set of trends > and benchmarks in digital > marketing<http://www2.merkleinc.com/l/47252/2016-07- > 26/47gt7c> > > Download our latest report titled “The Case for Change: Exploring the > Myths of Customer-Centric Transformation” > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email and > any attachments transmitted with it are intended for use by the > intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are > not the intended recipient, you must not keep, use, disclose, copy or > distribute this email without the author’s prior permission. We take > precautions to minimize the risk of transmitting software viruses, but > we advise you to perform your own virus checks on any attachment to > this message. We cannot accept liability for any loss or damage caused > by software viruses. The information contained in this communication > may be confidential and may be subject to the attorney-client privilege. > Access the Q2 2016 Digital Marketing Report for a fresh set of trends and benchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c> Download our latest report titled “The Case for Change: Exploring the Myths of Customer-Centric Transformation” <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c> This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.