Seminario Riesgo Bancario

[Miguel Angel Sanchez]

Buen día,



Deseo colocar a su disposición información general  del Seminario 
"GESTIÓN INTEGRAL DE RIESGO BANCARIO". El objetivo de esta 
capacitación es que el participante pueda evaluar las solicitudes 
de créditos aplicando los conceptos de Gestión Integral de Riesgo 
Financiero para poder determinar la tasa de retorno estimada para 
el banco, según el nivel de riesgo que representa cada cliente.
Por medio de presentaciones y ejercicios prácticos lograr 
transmitir los conceptos básicos, teóricos y prácticos de 
evaluación de riesgo y cálculo de retornos financieros.

Y aprovechamos  la ocasión para agradecer la confianza mostrada 
en nuestros programas, donde le ofreceremos innovadoras 
estrategias de capacitación, que le permitirán a su empresa 
mejorar su operatividad  y llevarla a  niveles de mayor 
excelencia.

DURACIÓN:


16 Horas (9:00am a 5:00pm)

FECHA: 


26  y 27 de Mayo 2016

LUGAR:


Hotel  GHL Princess  al lado de World Trade Center.

INVERSIÓN:


$630.00 USD más ITBMS .

PRONTO PAGO:  


$500.00 USD mas ITBMS antes del 04 de Marzo 2016.  
$550.00 USD mas ITBMS antes del 11 de Marzo 2016.

Precios corporativos a partir de  3 personas!!!

INCLUYE:


La exposición de un experto Consultor con trayectoria 
Internacional, material de trabajo, plumas, diplomas con valor 
curricular para cada participante, coffee break y almuerzo tipo 
Buffet.


 Para la confirmación,  requerimos por favor llene el formato 
oficial de INSCRIPCIONES y lo envía escaneado a  
alond...@trainingpartnersdmc.com



¡MUCHAS GRACIAS!  Estamos aquí para ustedes



Atentamente,

Lic. Angela Maria Londoño
Panamá Partners - Training Partners DMC, S.A.

Tel: (507) 217-9182 /2179167
Cel: (507) 65349388

Correo electrónico:

ange...@panamapartnersdmc.net

Nuestro compromiso es brindarle un Seminario práctico y efectivo, 
con la total garantía de satisfacción.

Re: Haproxy running on 100% CPU and slow downloads

[Sachin Shetty]

agree to both the points.

Thanks
Sachin

On 4/7/16, 11:24 PM, "Willy Tarreau"  wrote:

>On Thu, Apr 07, 2016 at 10:59:24PM +0530, Sachin Shetty wrote:
>> Hi Willy,
>> 
>> Sorry for the confusion. I wrote to you much before in my
>>investigation. I
>> will take care going forward.
>
>OK but in general the point remains, and it's not just for you but for
>everyone in general, the mailing list is here to reach around 1000 persons
>at once, so once your message is posted, you have to keep in mind that
>several of them will start to think about your problem even if they don't
>respond, which is why it is very important to be transparent about any
>progress made on parallel investigation or parallel contacts. Just like
>when you ask something to two distinct coworkers, one gives you a fast
>response, the other ones comes the next day and says "I've setup a lab
>yesterday to check what you asked me and I found this last night". You'll
>feel bad telling him "Oh I already got the response, thank you anyway".
>
>> Only now I realized that I messed up the version numbers because it
>>seems
>> we have different versions in our cluster.
>
>OK similarly there's nothing wrong telling errors in bug reports, we all
>do this because we test lots of stuff and we end up confusing things. But
>once you notice something was wrong, simply respond again and fix the
>information. Reliable versions helps eliminate candidate patches and also
>help people joining saying "same problem here".
>
>> We are now testing with 1.6.4 and trying to fast track it.
>
>OK thanks for the feedback!
>
>Willy
>

Re: Haproxy running on 100% CPU and slow downloads

[Willy Tarreau]

On Thu, Apr 07, 2016 at 10:59:24PM +0530, Sachin Shetty wrote:
> Hi Willy,
> 
> Sorry for the confusion. I wrote to you much before in my investigation. I
> will take care going forward.

OK but in general the point remains, and it's not just for you but for
everyone in general, the mailing list is here to reach around 1000 persons
at once, so once your message is posted, you have to keep in mind that
several of them will start to think about your problem even if they don't
respond, which is why it is very important to be transparent about any
progress made on parallel investigation or parallel contacts. Just like
when you ask something to two distinct coworkers, one gives you a fast
response, the other ones comes the next day and says "I've setup a lab
yesterday to check what you asked me and I found this last night". You'll
feel bad telling him "Oh I already got the response, thank you anyway".

> Only now I realized that I messed up the version numbers because it seems
> we have different versions in our cluster.

OK similarly there's nothing wrong telling errors in bug reports, we all
do this because we test lots of stuff and we end up confusing things. But
once you notice something was wrong, simply respond again and fix the
information. Reliable versions helps eliminate candidate patches and also
help people joining saying "same problem here".

> We are now testing with 1.6.4 and trying to fast track it.

OK thanks for the feedback!

Willy

Re: Increased CPU usage after upgrading 1.5.15 to 1.5.16

[Lukas Tribus]

Hi,


Am 07.04.2016 um 17:43 schrieb James Brown:
Calling DES functions is kind of suspicious? I'd expect any clients 
made in the last decade or so to be negotiating AES (which is much, 
/much/ faster than DES) with either the default settings or any 
reasonably-secure custom settings. Can you check what cipher suites 
you've negotiated in production? If something is causing you to 
negotiate  a 3DES-based cipher suite instead of an AES (preferably 
AES-GCM)-based cipher suite, that would definitely explain increased 
CPU usage.


This is just a lab repro here, with 1 client in 1 session and 1 GET 
request. The client is indeed an ancient wget, negotiating 3DES 
(TLS_RSA_WITH_3DES_EDE_CBC_SHA).


However, since I'm testing with the same client both with and without 
this commit, the result should be the same. Also, the slower and 
inefficient crypto is, the better, since I'm trying to create a testcase.


In fact, a recent client will negotiate AES-GCM, which is so efficient, 
most of perf trace consist in read_tsc kernel calls and a comparison of 
the efficiency with and without the commit is rather difficult.



I will keep playing with this.


lukas

Re: Haproxy running on 100% CPU and slow downloads

[Sachin Shetty]

Hi Willy,

Sorry for the confusion. I wrote to you much before in my investigation. I
will take care going forward.

Only now I realized that I messed up the version numbers because it seems
we have different versions in our cluster.

We are now testing with 1.6.4 and trying to fast track it.

Thanks
Sachin

On 4/7/16, 6:31 PM, "Willy Tarreau"  wrote:

>Hi Sachin,
>
>On Thu, Apr 07, 2016 at 02:21:16PM +0200, Lukas Tribus wrote:
>> Hi,
>> 
>> Am 05.04.2016 um 09:38 schrieb Sachin Shetty:
>> >Hi Lukas, Pavlos,
>> >
>> >Thanks for your response, more info as requested.
>> >
>> >1. Attached conf with some obfuscation
>> >2. Haproxy -vv
>> >HA-Proxy version 1.5.4 2014/09/02
>> >Copyright 2000-2014 Willy Tarreau 
>> >
>> 
>> I would upgrade to something more recent, the number of bugfixes
>> since 1.5.4 amount to more than 100!
>(...)
>
>I'm just discovering that you opened this thread twice in parallel,
>once with me in private and once with the ML, resulting in everyone
>doing the work twice and giving you the same advices twice. Please
>avoid this in the future, it wastes everyone's time and discourages
>people from responding to such questions. The place to ask is the ML,
>and if you contact someone privately please at least point to the
>public question so that the response is public and it saves others'
>valuable time.
>
>Also the version you reported to me was different :
>
>   HA-Proxy version 1.5.9 2014/11/25
>
>Thanks,
>Willy
>

Re: [PATCH] OPTIM/MINOR: session: abort if possible before connecting to the backend

[Willy Tarreau]

Hi Frederik,

On Thu, Apr 07, 2016 at 09:42:13AM -0700, Frederik Deweerdt wrote:
> Hi Willy,
> 
> The approach to look at the state of the connection in
> sess_update_stream_int() does work as expected. Please see the
> attached patch.

Excellent! Patch applied now, thanks!

Willy

[PATCH] OPTIM/MINOR: session: abort if possible before connecting to the backend

[Frederik Deweerdt]

Hi Willy,

The approach to look at the state of the connection in
sess_update_stream_int() does work as expected. Please see the
attached patch.

Thanks,
Frederik

On Wed, Apr 6, 2016 at 10:45 AM, Frederik Deweerdt  wrote:
> On Wed, Apr 6, 2016 at 9:53 AM, Willy Tarreau  wrote:
> [...]
>
>> So this means that in TCP mode we're aware of the abort earlier than in
>> HTTP mode. Thus we theorically have everything needed to decide not to
>> connect if possible.
>>
> /me nods, it appears so.
>
>> This one will result in truncated transfers when shutting down at the end
>> so we must not do it this way, but I see your point. I guess one reason
>> why TCP's abort is not caught is that we don't enter process_stream() once
>> the connection is established and we don't have any more analysers (which
>> is true in TCP). In HTTP we have several opportunities to get back there
>> and possibly to abort the connection early.
>>
>> I think that we'd need to add tests for PR_O_ABRT_CLOSE before deciding
>> to establish a connection, it's done in sess_update_stream_int(), for
>> state SI_ST_ASS. In certain states such as SI_ST_QUE (queue) or SI_ST_TAR
>> (turn-around, wait before retrying), the test is already performed, but
>> its missing here before the call to connect_server(). Do you want to try
>> this instead ? If it works, simply send a patch with a description
>> according to the CONTRIBUTING file and I'll happily merge it.
>>
> That sounds good, I'll try that.
>
> Thanks!
> Frederik


0001-OPTIM-MINOR-session-abort-if-possible-before-connect.patch
Description: Binary data

Re: Q: about HTTP/2

[Willy Tarreau]

Hi Aleks,

On Fri, Apr 01, 2016 at 12:18:54PM +0200, Aleksandar Lazic wrote:
> Hi Willy & other core devs/pms.
> 
> I know that HTTP/2 is on the road-map but not ready yet.
> 
> Would you be so kind and share some of your thoughts, stats and plans for
> HTTP/2.

Well, the plan is to have *at least* HTTP/2 with the client and HTTP/1 with
the server. Maybe we'll find that doing H2->H2 is easy enough but given our
past experiences guessing that two sides are only slightly more difficult
than a single one, I'd prefer to remain careful.

Regarding the timing, I'm trying hard to get something for 1.7 next fall.
But to be very honnest, 90% of my time spent on haproxy is spent chasing
bugs these days, the last 10% are spent on code review. We've reached a
level of complexity that is high enough to keep bug hunters busy and that's
slowing us down. For the first time we even released a version with several
known bugs that were not yet addressed by lack of time. So I'm trying to
compose between fixing bugs and developing.

I have some paper drafts about what to do, when I read the date on them I
realize that time flies (2014 for some of them). A significant part of the
internal architecture is ready (split between streams and sessions), some
of it still needs to be done (make an applet able to use normal load balancing
just like a regular client), we need to implement the H2<->H1 MUX which will
itself work almost like a proxy with various states depending what side closes
first etc. And then to address all the shortcomings that will result from this
(eg: tcp-request contents having to be applied on the clear text only, etc).

Also among the requirements we can count one which is that the applets are
fixed regarding the issue we currently have with peers which can stall. H2
will have the same problem so we must ensure we find a correct fix for the
peers before going full throttle the H2 way if we don't want to modify the
architecture again.

That's the most accurate vision I can give for the moment.

Cheers,
Willy

Re: Increased CPU usage after upgrading 1.5.15 to 1.5.16

[Willy Tarreau]

On Thu, Apr 07, 2016 at 03:07:55PM +0200, Willy Tarreau wrote:
> It definitely is, as any piece of information we can find around this. I was
> wondering whether there was something in relation with the max TLS record
> size maybe. The other possibility would be that the fix uncovered another
> bug which was hidden by this fix. Unfortunately for now I failed to find
> which one by just reading the code, and I still couldn't find any way to
> reproduce the issue to try to analyze it deeper :-(

So I did a new series of tests, with servers slower than client and
conversely, with SSL on either and both sides, but couldn't get anywhere
near the problem. However I'm having a few thoughts and now I need to
re-audit the 1.5 code regarding this. The buffer full semantics were
a bit different in 1.5 compared to 1.6. The channel_full() function as
it is now indicates whether or not the buffer is full with *incoming*
data more or less the reserve. The previous one would indicate whether
it was full, and only the data in transit were only deduced from the
reserve. In 1.5 this is used to decide when to enable polling for reading,
and buffer_max_len() is used to determine how much can be read into a
buffer, either from an applet or from the network.

What I'm suspecting is that I didn't fully understand the extent of the
difference between 1.5 and 1.6 in this area and that I possibly broke
either buffer_max_len(), channel_full() or both in some corner cases,
resulting in the following scenario and which seems compatible with the
trace Janusz provided :

  - the buffer is in a certain state that is still left to be figured out
  - channel_full() returns zero because there are enough data in transit
to compensate for the reserve, which is already full
  - stream_interface says "good, there's some room left, let's poll for read"
  - poll returns "read ready".
  - the stream interface data handler is called to perform the read, calls
bi_avail() to determine how much it can read, founds zero indicating the
buffer is full, then says "stop telling me to read, it's full".
  - stream interface then finishes a few adjustments, sees thanks to
channel_full() that there's some room left and enables reading again.
  - goto $-3

If someone who can reliably reproduce the issue could check whether 1.6 has
the same issue, it would help me cut the problem in half. That obviously
excludes all those running sensitive production of course.

Cheers,
Willy

Re: Increased CPU usage after upgrading 1.5.15 to 1.5.16

[James Brown]

Calling DES functions is kind of suspicious? I'd expect any clients made in
the last decade or so to be negotiating AES (which is much, *much* faster
than DES) with either the default settings or any reasonably-secure custom
settings. Can you check what cipher suites you've negotiated in production?
If something is causing you to negotiate  a 3DES-based cipher suite instead
of an AES (preferably AES-GCM)-based cipher suite, that would definitely
explain increased CPU usage.

On Thu, Apr 7, 2016 at 5:25 AM, Lukas Tribus  wrote:

> Hi,
>
> Am 05.04.2016 um 10:17 schrieb Nenad Merdanovic:
>
>>
>> I am not sure, as I haven't even be able to reliably reproduce it on 1.5
>> (though we are running with some backports from 1.6) as it seems to be
>> traffic-pattern related. On one workload I exhibit instant and constant
>> jump in CPU usage (from 40% to 80-100%, about 50:50 sys:usr), but on
>> other, there are just some very short spikes to 100%.
>>
>
> I've played around with an unscientific testcase (single session, large
> 10MB response), perf
> and ltrace, and while the number of SSL_Write calls are the same, OpenSSL
> seems to
> be doing more low level stuff in functions like _x86_DES_encrypt and
> _x86_DES_decrypt.
>
> So this commit does make OpenSSL uncomfortable in some way, although it is
> probably
> not related to the number of SSL_write calls.
>
> Not sure if this is helpful.
>
>
> cheers,
> lukas
>
>
>


-- 
James Brown
Engineer

Re: Increased CPU usage after upgrading 1.5.15 to 1.5.16

[Willy Tarreau]

On Thu, Apr 07, 2016 at 02:25:32PM +0200, Lukas Tribus wrote:
> Hi,
> 
> Am 05.04.2016 um 10:17 schrieb Nenad Merdanovic:
> >
> >I am not sure, as I haven't even be able to reliably reproduce it on 1.5
> >(though we are running with some backports from 1.6) as it seems to be
> >traffic-pattern related. On one workload I exhibit instant and constant
> >jump in CPU usage (from 40% to 80-100%, about 50:50 sys:usr), but on
> >other, there are just some very short spikes to 100%.
> 
> I've played around with an unscientific testcase (single session, large 10MB
> response), perf
> and ltrace, and while the number of SSL_Write calls are the same, OpenSSL
> seems to
> be doing more low level stuff in functions like _x86_DES_encrypt and
> _x86_DES_decrypt.
> 
> So this commit does make OpenSSL uncomfortable in some way, although it is
> probably
> not related to the number of SSL_write calls.
> 
> Not sure if this is helpful.

It definitely is, as any piece of information we can find around this. I was
wondering whether there was something in relation with the max TLS record
size maybe. The other possibility would be that the fix uncovered another
bug which was hidden by this fix. Unfortunately for now I failed to find
which one by just reading the code, and I still couldn't find any way to
reproduce the issue to try to analyze it deeper :-(

Cheers,
Willy

Re: Haproxy running on 100% CPU and slow downloads

[Willy Tarreau]

Hi Sachin,

On Thu, Apr 07, 2016 at 02:21:16PM +0200, Lukas Tribus wrote:
> Hi,
> 
> Am 05.04.2016 um 09:38 schrieb Sachin Shetty:
> >Hi Lukas, Pavlos,
> >
> >Thanks for your response, more info as requested.
> >
> >1. Attached conf with some obfuscation
> >2. Haproxy -vv
> >HA-Proxy version 1.5.4 2014/09/02
> >Copyright 2000-2014 Willy Tarreau 
> >
> 
> I would upgrade to something more recent, the number of bugfixes
> since 1.5.4 amount to more than 100!
(...)

I'm just discovering that you opened this thread twice in parallel,
once with me in private and once with the ML, resulting in everyone
doing the work twice and giving you the same advices twice. Please
avoid this in the future, it wastes everyone's time and discourages
people from responding to such questions. The place to ask is the ML,
and if you contact someone privately please at least point to the
public question so that the response is public and it saves others'
valuable time.

Also the version you reported to me was different :

   HA-Proxy version 1.5.9 2014/11/25

Thanks,
Willy

Re: Increased CPU usage after upgrading 1.5.15 to 1.5.16

[Lukas Tribus]

Hi,

Am 05.04.2016 um 10:17 schrieb Nenad Merdanovic:


I am not sure, as I haven't even be able to reliably reproduce it on 1.5
(though we are running with some backports from 1.6) as it seems to be
traffic-pattern related. On one workload I exhibit instant and constant
jump in CPU usage (from 40% to 80-100%, about 50:50 sys:usr), but on
other, there are just some very short spikes to 100%.


I've played around with an unscientific testcase (single session, large 
10MB response), perf
and ltrace, and while the number of SSL_Write calls are the same, 
OpenSSL seems to
be doing more low level stuff in functions like _x86_DES_encrypt and 
_x86_DES_decrypt.


So this commit does make OpenSSL uncomfortable in some way, although it 
is probably

not related to the number of SSL_write calls.

Not sure if this is helpful.


cheers,
lukas

Re: Haproxy running on 100% CPU and slow downloads

[Lukas Tribus]

Hi,

Am 05.04.2016 um 09:38 schrieb Sachin Shetty:

Hi Lukas, Pavlos,

Thanks for your response, more info as requested.

1. Attached conf with some obfuscation
2. Haproxy -vv
HA-Proxy version 1.5.4 2014/09/02
Copyright 2000-2014 Willy Tarreau 



I would upgrade to something more recent, the number of bugfixes
since 1.5.4 amount to more than 100!

That said, I've not stumbled upon a particular bug explaining what
you are seeing.

My suggestion would be to go back to nbproc 1 (its easier to
troubleshoot), and run the 100% spinning process through
strace -tt -p and post the output.




Thanks,

Lukas