I think I know the cause now.
The code tries to get "baseZNode" from the config. and the latter is obtained
from Connection#getConfiguration(). Now we have two connections, one from local
hbase, the other remote. The local hbase's connection has the configuration set
perfectly, while the one on the remote connection barely has anything, hence
not able to get a value of "baseZNode".
So based on this theory, CopyTable will never work if the remote is a secured
cluster, is that a right assessment? Does anyone have luck to get it work?
-----Original Message-----
From: Frank Luo
Sent: Thursday, September 08, 2016 6:45 PM
To: user@hbase.apache.org
Subject: RE: CopyTable fails on copying between two secured clusters
I don't think they are pointing to different locations. Both of them should be
/hbase-secure.
However, the debugger shows that ConnectionManager#retrieveClusterId are called
twice, the first time regards to the source cluster, which works fine, and
watcher.clusterIdZNode=/hbase-secure/hbaseid, and it is correct.
The second time for the remote cluster, watcher.clusterIdZNode=/hbase/hbaseid,
which should be incorrect.
What I am suspecting is ZooKeeperWatcher, method setNodeNames. It reads:
private void setNodeNames(Configuration conf) {
baseZNode = conf.get(HConstants.ZOOKEEPER_ZNODE_PARENT,
HConstants.DEFAULT_ZOOKEEPER_ZNODE_PARENT);
I am not sure the conf is corrected fetched from the remote cluster. If not,
the default value is given, which is /hbase and incorrect.
By the way, below is the root znodes for zookeepers:
The source cluster:
[hbase-secure, hiveserver2, hive, hbase-unsecure, templeton-hadoop, hadoop-ha,
zookeeper]
The target cluster:
[hbase-secure, hive, hiveserver2, hbase-unsecure, hadoop-ha, zookeeper]
-----Original Message-----
From: Esteban Gutierrez [mailto:este...@cloudera.com]
Sent: Thursday, September 08, 2016 1:02 PM
To: user@hbase.apache.org
Subject: Re: CopyTable fails on copying between two secured clusters
Is it possible that in your destination cluster zookeeper.znode.parent points
to a different location than /hbase ? If both clusters are under the same
kerberos realm then there is no need to worry about
zookeeper.security.auth_to_local.
--
Cloudera, Inc.
On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <j...@merkleinc.com> wrote:
> Thanks Esteban for replying.
>
> The Kerberos realm is shared between the two clusters.
>
> I searched zookeeper config and couldn't find the rule, so where it
> is set?
>
> Having said that, I looked at parameters passed to getData call, and
> it doesn't look like security related.
>
> PS. I am using hbase 1.1.2.
>
> Here is the log:
>
> com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyT
> able Connected to the target VM, address: '127.0.0.1:50669',
> transport:
> 'socket'
> 0 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 247 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to
> load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 1492 [main] INFO org.apache.hadoop.security.UserGroupInformation -
> Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using
> keytab file \Users\jluo\tidsvc_exphbase.keytab
> 11462 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 164493 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 167720 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 172108 [main] WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory
> - The short-circuit local reads feature cannot be used because UNIX
> Domain sockets are not available on Windows.
> 224735 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 225165 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 225190 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> host.name=JLUOWIN7.pclc0.merkle.local
> 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.version=1.7.0_51
> 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.vendor=Oracle Corporation
> 225191 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.home=C:\work\jdk1.7.0_51\jre
> 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\
> 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.compiler=<NA>
> 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> os.name=Windows 7
> 225193 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.arch=amd64
> 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.version=6.1
> 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:
> user.name=jluo
> 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.home=C:\Users\jluo
> 225194 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance
> 225197 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 231758 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 231758 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c54424116, negotiated timeout = 120000
> 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Client session timed out, have not
> heard from server in 96993ms for sessionid 0x357067c5442410e, closing
> socket connection and attempting reconnect
> 374923 [main] INFO org.apache.zookeeper.ZooKeeper - Session:
> 0x357067c54424116 closed
> 374924 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
> EventThread shut down
> 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session
> 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 376538 [main] INFO
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> tion
> - Closing zookeeper sessionid=0x357067c5442410e
> 376581 [main] INFO org.apache.zookeeper.ZooKeeper - Session:
> 0x357067c5442410e closed
> 376581 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
> EventThread shut down
> 376746 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376749 [main] INFO org.apache.hadoop.conf.Configuration.deprecation
> - io.bytes.per.checksum is deprecated. Instead, use
> dfs.bytes-per-checksum
> 376839 [main] WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376841 [main] INFO org.apache.hadoop.conf.Configuration.deprecation
> - io.bytes.per.checksum is deprecated. Instead, use
> dfs.bytes-per-checksum
> 376843 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 376843 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> client connection,
> connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> baseZNode=/hbase
> 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session
> 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid =
> 0x2565c552d296982, negotiated timeout = 40000
> 377140 [main] INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry
> - ClusterId read in ZooKeeper is null
> 377141 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 377141 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
> client connection,
> connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> baseZNode=/hbase
> 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid =
> 0x1565c552cfd6b03, negotiated timeout = 40000
> 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Client session timed out, have not
> heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing
> socket connection and attempting reconnect
> 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Client session timed out, have not
> heard from server in 636900ms for sessionid 0x2565c552d296982, closing
> socket connection and attempting reconnect
> 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 1236762 [main] INFO org.apache.zookeeper.ZooKeeper - Session:
> 0x1565c552cfd6b03 closed
> 1236762 [main] INFO
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> tion
> - Closing zookeeper sessionid=0x2565c552d296982
> 1236762 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
> EventThread shut down
> 1236908 [main] INFO org.apache.zookeeper.ZooKeeper - Session:
> 0x2565c552d296982 closed
> 1236909 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
> EventThread shut down
>
> java.io.IOException: Failed to get cluster ID
>
> at org.apache.hadoop.hbase.security.token.TokenUtil.
> getAuthToken(TokenUtil.java:365)
> at org.apache.hadoop.hbase.security.token.TokenUtil.
> addTokenForJob(TokenUtil.java:327)
> at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initCredentials(TableMapReduceUtil.java:451)
> at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:658)
> at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:591)
> at org.apache.hadoop.hbase.mapreduce.CopyTable.
> createSubmittableJob(CopyTable.java:168)
> at org.apache.hadoop.hbase.mapreduce.CopyTable.run(
> CopyTable.java:348)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.hbase.mapreduce.CopyTable.main(
> CopyTable.java:341)
> at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.export(CopyTableTest.java:44)
> at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.testCopyTable(CopyTableTest.java:25)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(
> ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.
> evaluate(InvokeMethod.java:17)
> at com.intellij.junit4.JUnit4TestRunnerUtil$
> IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:349)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(
> ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(
> ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(
> JUnit4IdeaTestRunner.java:69)
> at com.intellij.rt.execution.junit.JUnitStarter.
> prepareStreamsAndStart(JUnitStarter.java:234)
> at com.intellij.rt.execution.junit.JUnitStarter.main(
> JUnitStarter.java:74)
>
> Disconnected from the target VM, address: '127.0.0.1:50669', transport:
> 'socket'
>
> Process finished with exit code -1
>
> -----Original Message-----
> From: Esteban Gutierrez [mailto:este...@cloudera.com]
> Sent: Thursday, September 08, 2016 12:37 PM
> To: user@hbase.apache.org
> Subject: Re: CopyTable fails on copying between two secured clusters
>
> Frank,
>
> Do you have a cross realm setup? if yes, do you have ZK to use the
> proper rules via the -Dzookeeper.security.auth_to_local flag? If you
> could share additional logs that would be helpful for us.
>
> Thanks!
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <j...@merkleinc.com> wrote:
>
> > I couldn’t manage to get the CopyTable to work between two secured
> > clusters and hope someone can shed some light.
> >
> > I have a table created on both clusters, and I am running the
> > CopyTable command from the source cluster.
> >
> > The parameters are the following:
> > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase",
> > "--new.name=TargetTable", "SourceTable"
> >
> > I have ensured that the user has all permissions on both tables.
> > When I run it, I got:
> >
> > =====================
> > java.io.IOException: Failed to get cluster ID
> >
> > at org.apache.hadoop.hbase.security.token.TokenUtil.
> > getAuthToken(TokenUtil.java:365)
> > at org.apache.hadoop.hbase.security.token.TokenUtil.
> > addTokenForJob(TokenUtil.java:327)
> > =====================
> >
> > Debugging into it, I found that it is trying to get the clusterId
> > from my target cluster using address of zookeeper1/2, and the code
> > below returns
> > null:
> >
> >
> > =====================
> >
> > ZKUtil.getData(watcher, watcher.clusterIdZNode);
> > =====================
> >
> > Any hints please?
> >
> >
> >
> > Access the Q2 2016 Digital Marketing Report for a fresh set of
> > trends and benchmarks in digital
> > marketing<http://www2.merkleinc.com/l/47252/2016-07-
> > 26/47gt7c>
> >
> > Download our latest report titled “The Case for Change: Exploring
> > the Myths of Customer-Centric Transformation”
> > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c>
> >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this
> > email in error, please notify the sender immediately and then delete
> > it. If you are not the intended recipient, you must not keep, use,
> > disclose, copy or distribute this email without the author’s prior
> > permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss
> > or damage caused by software viruses. The information contained in
> > this communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Access the Q2 2016 Digital Marketing Report for a fresh set of trends
> and benchmarks in digital
> marketing<http://www2.merkleinc.com/l/47252/2016-07-
> 26/47gt7c>
>
> Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation”
> <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email and
> any attachments transmitted with it are intended for use by the
> intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are
> not the intended recipient, you must not keep, use, disclose, copy or
> distribute this email without the author’s prior permission. We take
> precautions to minimize the risk of transmitting software viruses, but
> we advise you to perform your own virus checks on any attachment to
> this message. We cannot accept liability for any loss or damage caused
> by software viruses. The information contained in this communication
> may be confidential and may be subject to the attorney-client privilege.
>
Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
benchmarks in digital
marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c>
Download our latest report titled “The Case for Change: Exploring the Myths of
Customer-Centric Transformation”
<http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c>
This email and any attachments transmitted with it are intended for use by the
intended recipient(s) only. If you have received this email in error, please
notify the sender immediately and then delete it. If you are not the intended
recipient, you must not keep, use, disclose, copy or distribute this email
without the author’s prior permission. We take precautions to minimize the risk
of transmitting software viruses, but we advise you to perform your own virus
checks on any attachment to this message. We cannot accept liability for any
loss or damage caused by software viruses. The information contained in this
communication may be confidential and may be subject to the attorney-client
privilege.
