Re: 4.0 best feature/fix?

Fri, 07 May 2021 09:18:59 -0700 

So I'm confused.
I get this on startup from the client:
2021-05-07 15:27:48,119 WARNÂ [com.dat.oss.dri.int.cor.poo.ChannelPool] (s1-admin-1) [s1|hercules/172.16.100.253:9042] Fatal error w hile initializing pool, forcing the node down: com.datastax.oss.driver.api.core.UnsupportedProtocolVersionException: [hercules/172.1 
6.100.253:9042] *Host does not support protocol version V5*
        at com.datastax.oss.driver.api.core.UnsupportedProtocolVersionException.forSingleAttempt(UnsupportedProtocolVersionException 
.java:46)
        at com.datastax.oss.driver.internal.core.channel.ProtocolInitHandler$InitRequest.onResponse(ProtocolInitHandler.java:335)         at com.datastax.oss.driver.internal.core.channel.ChannelHandlerRequest.onResponse(ChannelHandlerRequest.java:94)         at com.datastax.oss.driver.internal.core.channel.InFlightHandler.channelRead(InFlightHandler.java:257)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
        at java.base/java.lang.Thread.run(Thread.java:834)

If I specify in the application.conf to use v4, I get:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:48)         at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:25) Caused by: java.lang.IllegalArgumentException: Unknown protocol version name: v4 

Complete application.conf:

datastax-java-driver {
  basic.request.timeout = 60 seconds
  basic.request.consistency = LOCAL_QUORUM
  basic.contact-points = ["hercules:9042", "chaos:9042"]
  basic.load-balancing-policy {
        local-datacenter = datacenter1
  }
  advanced.protocol {
    version = v4
  }
}

-Joe


On 5/7/2021 12:05 PM, Sam Tunnicliffe wrote:
That's a driver error using protocol V5, which is the default from 4.0-rc1 but only recently added to the drivers. Can you try specifying protocol V4 with all the same parameters? Also, if it's at all possible (which it may not be, given the divergence between driver versions 3 & 4), could you try with protocol V5 and driver version 3.11.0? 

Thanks,
Sam
On 7 May 2021, at 16:12, Joe Obernberger <joseph.obernber...@gmail.com> wrote: 

I can retry Java 11.
I am seeing this error a lot - still debugging, but I'll throw it out there - using 4.11.1 driver and a 4 node RC1 cluster.  I'm seeing warning in the cassandra logs about slow queries, but no errors.  This error is client side.
Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: All 4 node(s) tried for the query failed (showing first 3 nodes, use getAllErrors() for more): Node(endPoint=/172.16.100.39:9042, hostId=93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47, hashCode=345a8431): [java.lang.NullPointerException], Node(endPoint=/172.16.100.36:9042, hostId=d9702f96-256e-45ae-8e12-69a42712be50, hashCode=4c7ac5bb): [java.lang.NullPointerException], Node(endPoint=chaos/172.16.100.37:9042, hostId=08a19658-40be-4e55-8709-812b3d4ac750, hashCode=7ba07f0e): [java.lang.NullPointerException]         at com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:141)         at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)         at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)         at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)         at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)         at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)         at com.ngc.helios.heliosingestservice.IngestService.splitOrigData(IngestService.java:55)         at com.ngc.helios.heliosingestservice.IngestService_ClientProxy.splitOrigData(IngestService_ClientProxy.zig:157)         at com.ngc.helios.heliosingestservice.IngestResource.splitOrigData(IngestResource.java:27)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.base/java.lang.reflect.Method.invoke(Method.java:566)         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)         at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)         at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)         at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492) 
        ... 15 more
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 


On 5/7/2021 10:00 AM, Jeff Jirsa wrote:
Cassandra 4.0 should work fine with java 11, including zgc (though zgc in jdk11 isn't meant to be production ready). 

The things I care most about:
- Much faster streaming, which you care about if you're not using EBS/Disaggregated storage - Virtual tables that make observability much more consistent (less JMX, more CQL) 
- Incremental repair finally actually works (correctly)
- There's a bunch of new defensive rate limiters and hot-tunable properties in the database that people will enjoy once they need to use them 
- JDK11�
On Fri, May 7, 2021 at 6:05 AM Joe Obernberger <joseph.obernber...@gmail.com> wrote: 

    Hi Sean - I'm using RC1 now in a research environment on bare
    metal.  The biggest drawback of Cassandra for me is that
    Cassandra has issues working with modern large servers - a
    server with >32TBytes of SSD seems to be a non-starter.

    I tried running Cassandra with java 11, and that doesn't appear
    to work.

    -Joe

    On 5/7/2021 8:47 AM, Durity, Sean R wrote:


    There is not enough 4.0 chatter here. What feature or fix of
    the 4.0 release is most important for your use
    case(s)/environment? What is working well so far? What needs
    more work? Is there anything that needs more explanation?

    �

    <image001.png>

        

    Sean Durity

    Staff Systems Engineer – Cassandra

    #cassandra - for the latest news and updates

    �
    �

    ------------------------------------------------------------------------

    The information in this Internet Email is confidential and may
    be legally privileged. It is intended solely for the addressee.
    Access to this Email by anyone else is unauthorized. If you are
    not the intended recipient, any disclosure, copying,
    distribution or any action taken or omitted to be taken in
    reliance on it, is prohibited and may be unlawful. When
    addressed to our clients any opinions or advice contained in
    this Email are subject to the terms and conditions expressed in
    any applicable governing The Home Depot terms of business or
    client engagement letter. The Home Depot disclaims all
    responsibility and liability for the accuracy and content of
    this attachment and for any damages or losses arising from any
    inaccuracies, errors, viruses, e.g., worms, trojan horses,
    etc., or other items of a destructive nature, which may be
    contained in this attachment and shall not be liable for
    direct, indirect, consequential or special damages in
    connection with this e-mail message or its attachment.

    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
        Virus-free. www.avg.com
    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>


    <x-msg://7/#m_654382365824200109_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Reply via email to