Re: C++ Broker Performance with large messages

[Chuck Rolke]

- Original Message -
> From: "Gordon Sim" 
> To: users@qpid.apache.org
> Sent: Tuesday, February 20, 2018 3:36:37 PM
> Subject: Re: C++ Broker Performance with large messages
> 
> On 20/02/18 16:51, andi welchlin wrote:
> > Hello all,
> > 
> > I tested throughput of the Qpid C++ Broker (compiled as Release).
> > 
> > It was tested on a virtual machine with 15 GB RAM.
> > 
> > First I sent a 100 MB message into a persistent queue. From sender to
> > receiver it took 16 seconds for one message.
> > 
> > Afterwards I sent a 300 MB message, this one took 125 seconds. So this is
> > not 3 times more (as I would have expected but 7,5 times more).
> > 
> > Any suggestions how to improve throughput?
> > 
> > Do you have an idea why 300 MB is 7,5 times slower than 100 MB?
> 
> Is this over AMQP 1.0? Also what clients have you tested with and is
> there any differences there?
> 

Some AMQP 1.0 qpid-proton -based sending clients before 0.19 suffered from large
output buffer issues. See https://issues.apache.org/jira/browse/PROTON-1687
and related issues.

> 
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
> 
> 

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: C++ Broker Performance with large messages

[Gordon Sim]

On 20/02/18 16:51, andi welchlin wrote:

Hello all,

I tested throughput of the Qpid C++ Broker (compiled as Release).

It was tested on a virtual machine with 15 GB RAM.

First I sent a 100 MB message into a persistent queue. From sender to
receiver it took 16 seconds for one message.

Afterwards I sent a 300 MB message, this one took 125 seconds. So this is
not 3 times more (as I would have expected but 7,5 times more).

Any suggestions how to improve throughput?

Do you have an idea why 300 MB is 7,5 times slower than 100 MB?


Is this over AMQP 1.0? Also what clients have you tested with and is 
there any differences there?



-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

[VOTE] Release Qpid Dispatch Router 1.0.1 (RC1)

[Ted Ross]

Please vote on this thread to release qpid-dispatch 1.0.1-rc1 as the
official 1.0.1.

The release can be found here:

https://dist.apache.org/repos/dist/dev/qpid/dispatch/1.0.1-rc1/

The following defects were fixed in this release:

DISPATCH-874 - unable to load .json or .woff2 files from local
file system from http port
DISPATCH-881 - Inbound pre-settled messages causes memory leak of deliveries
DISPATCH-882 - router buffers messages for slow presettled receiver
DISPATCH-883 - Router crashes when it processes management request
for connections
DISPATCH-887 - Dispatch reestablishes connection inspite of
deleting the connector
DISPATCH-889 - linkRoute patterns beginning with #/string match
substrings after the /
DISPATCH-895 - qpid-dispatch crashes with a SEGFAULT in libqpid-proton
DISPATCH-900 - Memory leak when repeatedly opening and closing connections
DISPATCH-908 - Router loses dispositions over receive link on
qpid-interop-test 2-node test
DISPATCH-914 - qd_connector_t leaks mutexes
DISPATCH-920 - Enabled policy blocks inter-router links

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Install on mac osx

[George David]

I also couldn't install using only pip install.

http://qpid.2158936.n2.nabble.com/trouble-connecting-to-Azure-Service-Bus-from-OS-X-td7672153.html

I ended up installing the qpid proton libraries using macports, then I was
able to install using pip. However, I was still unable to use it because it
was unable to connect to Azure Service Bus. I did get it work on Ubuntu
16.04 and Centos 7.4. The easiest way I found to use it on my mac was
through docker. Using the official python image I pip installed
python-qpid-proton without issue. I mounted the root directory of my
project and could test the code on that image. It's wasn't the best
development experience. I used PyCharm to develop the code and used docker
to test it. Luckily for me it was a small project. All I needed to do was
read messages from a service bus and output them to stdout.

-g$



On Tue, Feb 20, 2018 at 3:57 AM Chris Richardson  wrote:

> Hi Joshua,
>
> A similar issue was raised a few weeks ago, you might find that thread
> useful:
>
> http://qpid.2158936.n2.nabble.com/Error-installing-python-qpid-proton-0-20-0-tp7672530p7672660.html
>
> Unfortunately I'm no more able to test this on a Mac than Ken, so that
> information is all I can offer! I think you'll need to attempt an
> installation from source rather than via pip.
>
> Chris
>
> On 20 February 2018 at 05:08, Joshua Agudo  wrote:
>
> > Hi,
> >
> > Is it possible to install python qpid proton on a mac?
> > I ran "pip3 install python-qpid-proton" but failed with:
> >
> > "/private/var/folders/f0/8mtk875s79jcgcy1l590z69mgn
> > /T/pip-build-Er8aX6/python-qpid-proton/proton-c/src/
> > proactor/epoll.c:48:10:
> > fatal error: 'sys/timerfd.h' file not found"
> >
> > Is qpid proton only supported on linux/windows?
> >
> >
> > Regards
> > Josh
> >
>
>
>
> --
>
> *Chris Richardson*, System Architect
> c...@fourc.eu
>
>
> *FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norwaywww.fourc.eu
> *
>
> *Follow us on LinkedIn , Facebook
> , Google+  and Twitter
> !*
>

C++ Broker Performance with large messages

[andi welchlin]

Hello all,

I tested throughput of the Qpid C++ Broker (compiled as Release).

It was tested on a virtual machine with 15 GB RAM.

First I sent a 100 MB message into a persistent queue. From sender to
receiver it took 16 seconds for one message.

Afterwards I sent a 300 MB message, this one took 125 seconds. So this is
not 3 times more (as I would have expected but 7,5 times more).

Any suggestions how to improve throughput?

Do you have an idea why 300 MB is 7,5 times slower than 100 MB?

Kind Regards,
Andreas

Re: Dispatch router 1.0.0 seems to not always propagate link credit when using link routing

[Ted Ross]

Marcel,

I don't see anything that you are doing wrong in this case.  I will
see if I can reproduce your result.

-Ted

On Tue, Feb 20, 2018 at 9:57 AM, Marcel Meulemans
 wrote:
> I am experiencing a problem while using the dispatch router as a proxy to
> the artemis broker. I have done some investigation and it looks like the
> dispatch router is not always forwarding link credit to the broker.
>
> I have the following setup:
> proton clients 0.9.0 (old, i know :p) <---> qdrouterd 1.0.0 <> artemis
> 2.4.0
>
> The clients are doing some anycast messaging between themselves and artemis
> is providing some queues through which these messages are flowing. Qpid
> dispatch is sitting in between and doing link routing (see the
> qdrouterd.conf below). At around 100 clients (all connecting at the same
> time) things start to go wrong, as in some message do not reach their
> destinations (about 10% of the clients are effected). The messages actually
> "hang" in artemis because artemis cannot deliver the messages because the
> link the message should be delivered on has no credit. After some
> investigation is looks like qdrouterd is not always forwarding the credit
> given by the clients to the link-routing link it set up with artemis. I have
> attached two trace logs from qdrouterd (formatted a bit so you can put the
> side to side) illustrating the problem. The flow.log is the filtered trace
> output showing a successful session, the no-flow.log is a failing session.
> You can see in the failing session that the flow frames for two newly
> attached links are not "forwarded" over the link-route link as I expect they
> should (and which is the case in the successful session).
>
> Am I doing something wrong or is this a bug?
>
> Cheers,
> Marcel
>
>
> // qdrouterd.conf
>
> router {
> mode: standalone
> id: arcbus.proxy
> }
>
> listener {
> host: 0.0.0.0
> port: 5672
> authenticatePeer: no
> saslMechanisms: ANONYMOUS
> }
>
> connector {
> name: broker-service
> host: 127.0.0.1
> port: 5670
> role: route-container
> saslMechanisms: ANONYMOUS
> }
>
> linkRoute {
> name: anycast-to-broker
> prefix: anycast.
> dir: out
> connection: broker-service
> }
>
> linkRoute {
> name: anycast-from-broker
> prefix: anycast.
> dir: in
> connection: broker-service
> }
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Dispatch router 1.0.0 seems to not always propagate link credit when using link routing

[Marcel Meulemans]

I am experiencing a problem while using the dispatch router as a proxy to
the artemis broker. I have done some investigation and it looks like the
dispatch router is not always forwarding link credit to the broker.

I have the following setup:
proton clients 0.9.0 (old, i know :p) <---> qdrouterd 1.0.0 <> artemis
2.4.0

The clients are doing some anycast messaging between themselves and artemis
is providing some queues through which these messages are flowing. Qpid
dispatch is sitting in between and doing link routing (see the
qdrouterd.conf below). At around 100 clients (all connecting at the same
time) things start to go wrong, as in some message do not reach their
destinations (about 10% of the clients are effected). The messages actually
"hang" in artemis because artemis cannot deliver the messages because the
link the message should be delivered on has no credit. After some
investigation is looks like qdrouterd is not always forwarding the credit
given by the clients to the link-routing link it set up with artemis. I
have attached two trace logs from qdrouterd (formatted a bit so you can put
the side to side) illustrating the problem. The flow.log is the filtered
trace output showing a successful session, the no-flow.log is a failing
session. You can see in the failing session that the flow frames for two
newly attached links are not "forwarded" over the link-route link as I
expect they should (and which is the case in the successful session).

Am I doing something wrong or is this a bug?

Cheers,
Marcel


// qdrouterd.conf

router {
mode: standalone
id: arcbus.proxy
}

listener {
host: 0.0.0.0
port: 5672
authenticatePeer: no
saslMechanisms: ANONYMOUS
}

connector {
name: broker-service
host: 127.0.0.1
port: 5670
role: route-container
saslMechanisms: ANONYMOUS
}

linkRoute {
name: anycast-to-broker
prefix: anycast.
dir: out
connection: broker-service
}

linkRoute {
name: anycast-from-broker
prefix: anycast.
dir: in
connection: broker-service
}
2018-02-20 13:23:29.503949 +0100 SERVER (trace) [57]:0 <- @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b""]
2018-02-20 13:23:29.504047 +0100 SERVER (trace) [57]:0 -> @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:ANONYMOUS]]
2018-02-20 13:23:29.504078 +0100 SERVER (trace) [57]:0 -> @sasl-outcome(68) [code=0]
2018-02-20 13:23:29.504489 +0100 SERVER (trace) [57]:0 <- @open(16) [container-id="625e98e0-1672-4589-9216-2e8bdf2114bf_40327", hostname="127.0.0.1:5672", max-frame-size=32768, idle-time-out=15]
2018-02-20 13:23:29.504641 +0100 SERVER (trace) [57]:0 -> @open(16) [container-id="arcbus.proxy", max-frame-size=16384, channel-max=32767, idle-time-out=8000, offered-capabilities=:"ANONYMOUS-RELAY", properties={:product="qpid-dispatch-router", :version="1.0.0"}]

2018-02-20 13:23:29.510188 +0100 SERVER (trace) [57]:0 <- @begin(17) [next-outgoing-id=0, incoming-window=32, outgoing-window=0]
2018-02-20 13:23:29.510289 +0100 SERVER (trace) [57]:0 -> @begin(17) [remote-channel=0, next-outgoing-id=0, incoming-window=2147483647, outgoing-window=2147483647]
2018-02-20 13:23:29.657048 +0100 SERVER (trace) [57]:0 <- @attach(18) [name="I0", handle=0, role=true, snd-settle-mode=0, rcv-settle-mode=0, source=@source(40) [address="anycast.device.example.1-7249-625e98e0-1672-4589-9216-2e8bdf2114bf", durable=0, timeout=0, dynamic=false, capabilities=@PN_SYMBOL[:"create-on-demand"]], target=@target(41) [address="anycast.device.example.1-7249-625e98e0-1672-4589-9216-2e8bdf2114bf", durable=2, timeout=0, dynamic=false], initial-delivery-count=0]
2018-02-20 13:23:29.657286 +0100 SERVER (trace) [57]:0 <- @attach(18) [name="I1", handle=1, role=true, snd-settle-mode=0, rcv-settle-mode=0, source=@source(40) [address="anycast.device.example.1-7249", durable=0, timeout=0, dynamic=false, capabilities=@PN_SYMBOL[:"create-on-demand"]], target=@target(41) [address="anycast.device.example.1-7249", durable=2, timeout=0, dynamic=false], initial-delivery-count=0]
2018-02-20 13:23:29.657414 +0100 SERVER (trace) [57]:0 <- @flow(19) [next-incoming-id=0, incoming-window=32, next-outgoing-id=0, outgoing-window=0, handle=0, delivery-count=0, link-credit=16, drain=false]
2018-02-20 13:23:29.657462 +0100 SERVER (trace) [57]:0 <- @flow(19) [next-incoming-id=0, incoming-window=32, next-outgoing-id=0, outgoing-window=0, handle=1, delivery-count=0, link-credit=16, drain=false]

2018-02-20 13:23:29.667097 +0100 SERVER (trace) [1]:104 -> @begin(17) [next-outgoing-id=0, incoming-window=2147483647, outgoing-window=2147483647]
2018-02-20 13:23:29.667121 +0100 SERVER (trace) [1]:105 -> @begin(17) [next-outgoing-id=0, incoming-window=2147483647, outgoing-window=2147483647]
2018-02-20 13:23:29.668295 +0100 SERVER (trace) [1]:104 -> @attach(18) [name="I0", handle=0, role=true, snd-settle-mode=2, rcv-settle-mode=0, source=@source(40) [address="anycast.device.example.1-7249-625e98e0-1672-4589-9216-2e8bdf2114bf", 

Re: qpid-broker-j-7.0.0 failover is slow

[bryand]

Answers to your questions..

Could you please clarify whether you have seen "Gave up waiting" 
warnings with synchronous or asynchronous recoverer?  
- definitely when using synchronous but also pretty sure it was after I made
the context change to use asynchronous recovery also.  I've been making
several changes to get everything cleaned up so I can't remember about
asynchronous for sure.

Do you have full broker log with the warnings?  Can you share it? 
qpid.log   

Did you connect consumers to DLQs whilst the VH messages were 
asynchronously recovered? 
- I never had any consumers connected to any of the DLQs







--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Save the date: ApacheCon North America, September 24-27 in Montréal

[Rich Bowen]

Dear Apache Enthusiast,

(You’re receiving this message because you’re subscribed to a user@ or 
dev@ list of one or more Apache Software Foundation projects.)


We’re pleased to announce the upcoming ApacheCon [1] in Montréal, 
September 24-27. This event is all about you — the Apache project community.


We’ll have four tracks of technical content this time, as well as lots 
of opportunities to connect with your project community, hack on the 
code, and learn about other related (and unrelated!) projects across the 
foundation.


The Call For Papers (CFP) [2] and registration are now open. Register 
early to take advantage of the early bird prices and secure your place 
at the event hotel.


Important dates
March 30: CFP closes
April 20: CFP notifications sent
	August 24: Hotel room block closes (please do not wait until the last 
minute)


Follow @ApacheCon on Twitter to be the first to hear announcements about 
keynotes, the schedule, evening events, and everything you can expect to 
see at the event.


See you in Montréal!

Sincerely, Rich Bowen, V.P. Events,
on behalf of the entire ApacheCon team

[1] http://www.apachecon.com/acna18
[2] https://cfp.apachecon.com/conference.html?apachecon-north-america-2018

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: How to define RulesBased ACLs using Qpid Broker-J 7.0.0

[bryand]

Thanks for the info.  I was able to get the acls defined via the REST API
Update.  There were a few little quirks I ran into where the documentation
on the ACL page
(https://qpid.apache.org/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-AccessControlProviders.html#Java-Broker-Security-AccessControlProviders-ACLRules)
didn't match what some of the enum values were such as:

Failed: RestfulWsException [responseStatusCode=422, responseBody={
  "errorMessage" : "No enum constant
org.apache.qpid.server.security.access.plugins.RuleOutcome.ALLOW-LOG"
  
  
Failed: RestfulWsException [responseStatusCode=422, responseBody={
  "errorMessage" : "No enum constant
org.apache.qpid.server.security.access.config.ObjectProperties.Property.ROUTINGKEY"
  

but I was able to transform values as necessary for the REST API call



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Install on mac osx

[Chris Richardson]

Hi Joshua,

A similar issue was raised a few weeks ago, you might find that thread
useful:
http://qpid.2158936.n2.nabble.com/Error-installing-python-qpid-proton-0-20-0-tp7672530p7672660.html

Unfortunately I'm no more able to test this on a Mac than Ken, so that
information is all I can offer! I think you'll need to attempt an
installation from source rather than via pip.

Chris

On 20 February 2018 at 05:08, Joshua Agudo  wrote:

> Hi,
>
> Is it possible to install python qpid proton on a mac?
> I ran "pip3 install python-qpid-proton" but failed with:
>
> "/private/var/folders/f0/8mtk875s79jcgcy1l590z69mgn
> /T/pip-build-Er8aX6/python-qpid-proton/proton-c/src/
> proactor/epoll.c:48:10:
> fatal error: 'sys/timerfd.h' file not found"
>
> Is qpid proton only supported on linux/windows?
>
>
> Regards
> Josh
>



-- 

*Chris Richardson*, System Architect
c...@fourc.eu


*FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norwaywww.fourc.eu
*

*Follow us on LinkedIn , Facebook
, Google+  and Twitter
!*

Re: qpid-broker-j-7.0.0 failover is slow

[Oleksandr Rudyy]

Hi Bryan,

Could you please clarify whether you have seen "Gave up waiting"
warnings with synchronous or asynchronous recoverer?
Do you have full broker log with the warnings? Can you share it?
Did you connect consumers to DLQs whilst the VH messages were
asynchronously recovered?

Kind Regards,
Alex

On 19 February 2018 at 14:27, bryand  wrote:
> I was utilizing DLQs (Alternate Binding) for most of my queues and I think
> that is why the Virtual Host Node startup time was so slow (1 minute 45
> seconds).  After I deleted all those DLQs, the startup time was down to 11
> seconds - a huge difference especially when dealing with a failover
> situation.
>
> Here are some log messages I was seeing in the Broker-j log file regarding
> the DLQs (note that all my DLQs were empty):
>
> 2018-02-12 10:42:24,719 WARN  [VirtualHostNode-spgqpiddev3-Config]
> (o.a.q.s.m.AbstractConfiguredObject) - Gave up waiting for Queue
> 'app_attach_workqueue_DLQ' to attain state. Check object's state via
> Management.
>
> So I'm wondering if there is some type of issue with broker-j regarding
> DLQs/Alternate Bindings
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Install on mac osx

[Joshua Agudo]

Hi,

Is it possible to install python qpid proton on a mac?
I ran "pip3 install python-qpid-proton" but failed with:

"/private/var/folders/f0/8mtk875s79jcgcy1l590z69mgn/T/pip-build-Er8aX6/python-qpid-proton/proton-c/src/proactor/epoll.c:48:10:
fatal error: 'sys/timerfd.h' file not found"

Is qpid proton only supported on linux/windows?


Regards
Josh