Re: [twsocket] Sending data from multiple threads on SSL doesn't work

[Éric Fleming Bonilha]

I understand

But non-threaded is not an option.. our application is a high performance
video management system, it must be multi-threaded, in fact we have a very
highly threaded architecture, and it would create huge bottlenecks making
it single threaded (for network message processing). This specific part of
our application, is the part that sends video from server to clients. I
assign the socket to a message processing thread that manages sending and
receiving data for this section. I don't necessarily have to send data from
different threads, data can be sent from a single thread (but not main
thread) that is assigned to the socket.

But.. I found a solution to fix my initial problem (sending data from two
different threads).

Can you please validate if this would cause any issues?

I have a custom client class (for TWSocketServer) that is derived
from TWSocketClient that is instantiated by the server for every client
socket that connects to it.
I overwrote the Send function with the following code that checks for Ssl,
if Ssl is disabled then it would call inherited.. but it it is enabled,
instead of calling  TryToSend  directly from the calling thread (that is
causing the issue), it would only add the data to the send buffer and then
post FD_WRITE message, and the thread assigned to the client socket would
pick it up later and send the buffered data from its context.

I didn't change any ICS code, but only my derived class... do you think
that by always adding to the send buffer (PutDataInSendBuffer) and posting
FD_WRITE instead of calling TryToSend could cause any issues? Otherwise,
this fixes the main issue of socket stopping writting data... I will still
investigate on disconnections...


  // When working with SSL, we can't send data from different threads,
  // apparently OpenSSL does not like it. It could be an issue with ICS as
well.
  // To overcome the problem, when sending data with SSL enabled, instead of
  // directly sending the data from the calling thread, we just put the data
  // to the send buffer and post a message so it will be sent later from the
  // thread that is processing the socket messages
  if (not SslEnable) or
 ((FState <> wsConnected) and (FState <> wsSocksConnected)) then
  begin
inherited;
Exit;
  end;

  // Add to send buffer
  if Len <= 0 then
Exit(0)
  else
  begin
Result := Len;
PutDataInSendBuffer(Data, Len);
  end;

  // Post message for sending data
  PostMessage(Handle,
  FMsg_WM_ASYNCSELECT,
  WParam(FHSocket),{ V8.08 }
  IcsMakeLong(FD_WRITE, 0));




Thanks
Eric


On Wed, Oct 3, 2018 at 3:10 PM Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> > It happens if I'm sending from a thread or even only from the
> > main thread (from a timer), please check this demo app:
>
> Sorry, won't be looking for more bugs in ICS until the V8.57 release is
> done, it's weeks later already.
>
> I can only suggest trying to avoid threads, I don't use them and don't
> have any problems in my commercial applications.
>
> Angus
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Sending data from multiple threads on SSL doesn't work

[Éric Fleming Bonilha]

Angus

Funny thing, when the error happens, getlasterror from SSL returns:

error:1408F10B:SSL routines:ssl3_get_record:wrong version number

The disconnection is easily reproduceable by spawning a few threads

Eric



On Wed, Oct 3, 2018 at 2:01 PM Éric Fleming Bonilha 
wrote:

> Angus
>
> I'm trying to make a demo application to test this behavior, but now I'm
> stuck with another problem.. random disconnects..
>
> I'm using latest ICS 8.53 with OpenSSL that is shipped with the ICS
>
> Whenever I start sending data from server -> client, the client will
> disconnect randomly.
>
> It happens if I'm sending from a thread or even only from the main thread
> (from a timer), please check this demo app:
>
> http://www.digifort.com/temp/SSLMultithread.rar
>
> I've found that the client disconnection is happening here:
> TCustomSslWSocket.Do_FD_READ
>
> Res := my_BIO_read(FSslBio, @Dummy, 0); //Pointer(1)
> if Res < 0 then begin
> if not my_BIO_should_retry(FSslBio) then begin
>
>
> my_BIO_read returns -1 and then my_BIO_should_retry returns FALSE
>
> I see this comment on the code:
> { Get number of bytes we can receive and store in the network
> input bio.  }
> { New call to BIO_ctrl_get_read_request in order to read only the
> amount  }
> { of data from the socket that is needed to satisfy a read
> request, if}
> { any. Without that call I had random errors on bi-directional
> shutdowns. }
>
>
> Do you have any ideas why the client is being disconnected on this demo
> app?
>
> This demo app is very simple.. I simply send data to the connected client,
> and when data is received I increment a number of received bytes per second
>
> Eric
>
>
> On Wed, Oct 3, 2018 at 12:38 PM Angus Robertson - Magenta Systems Ltd <
> an...@magsys.co.uk> wrote:
>
>> > I'm currently using OpenSSL 1.0.2h and I tried with latest ICS I
>> > downloaded yesterday (8.53)
>>
>> Does your application include OverbyteIcsSslThrdLock.pas?  That is
>> needed to make locking work with 1.0.2.
>>
>> If you use 1.1.0 or 1.1.1, that unit is not needed.
>>
>> But I don't know much about OverbyteIcsSslThrdLock, never used it
>> myself. It is used by OverbyteIcsWSocketTS.pas which I assume you are
>> using, but I've never looked at that unit, don't know if the OpenSSL
>> stuff is up to date.  There is only one demo that uses it, a simple
>> socket server, not HTTP or anything easy to test.
>>
>> I have been meaning to look at a threaded web server, but never had the
>> time.
>>
>> Angus
>>
>> --
>> To unsubscribe or change your settings for TWSocket mailing list
>> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
>> Visit our website at http://www.overbyte.be
>>
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Sending data from multiple threads on SSL doesn't work

[Éric Fleming Bonilha]

Hi Angus

Thanks for the reply. This is indeed a strange problem, because there's no
deadlock or locking problem, and the problem does not happen in a
deterministic way.. its kind of random, so I have to keep it running for a
few seconds or minutes, then it will stop, maybe when a certain condition /
workflow is met

I'm currently using OpenSSL 1.0.2h and I tried with latest ICS I downloaded
yesterday (8.53)

Maybe I should update my OpenSSL library for testing?

I will update it, if it still happen I will try to make a demo program that
can simulate this issue.. This is beyond my pay grade.. I don't have any
experience with OpenSSL

Thanks
Eric



On Wed, Oct 3, 2018 at 3:25 AM Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> > I understand that ICS is thread safe when sending data because
> > adding and removing data to send buffers are locked with critical
> > sections..  but maybe OpenSSL does not like when we send data
> > from two different threads?
>
> OpenSSL is thread safe, or at least has been in the past, I've tested
> an FTP application sending hundreds of SSL sessions each using a
> separate thread.
>
> But OpenSSL has changed a lot in the last two years, there is a ICS
> unit OverbyteIcsSslThrdLock which used to provide OpenSSL with thread
> locking for 1.0.2 and earlier, but is no longer used for 1.1.0 and
> later since OpenSSL does locking internally.
>
> I've not tested heavy thread use for a while, since it is rarely
> necessary with ICS, it just complicates applications and introduces
> problems.
>
> Which OpenSSL and ICS versions are you using?
>
> Angus
>
>
>
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] Sending data from multiple threads on SSL doesn't work

[Éric Fleming Bonilha]

Hi

I'm experiencing a strange issue that I think could be related to OpenSSL,
but I want to make sure..

In a simple explanation, I have a TWSocketServer which serves sockets on
SSL. I have 1 client connected to it and I send video data through this
socket to the client connected to my server. If I send data from two
different threads (Video from 1 threads, and keep-alive messages from
another thread), eventually the socket will stop sending data. This only
happens if I try to send data from these two threads.

This works fine with non-SSL socket (I'm just implementing SSL in our
application), and it has always worked fine

I found that when this problem is happening my_BIO_write always returns -1
on TryToSend from
TCustomSslWSocket

Count := my_BIO_write(FSslbio, Data, Len);

I understand that ICS is thread safe when sending data because adding and
removing data to send buffers are locked with critical sections.. but maybe
OpenSSL does not like when we send data from two different threads?

I think I can make a sample application to test this case. But does anyone
has any clue?

Thanks

Eric
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] OpenSSL 1.1.0d and 1.0.2k support

[Éric Fleming Bonilha]

> The code you say you added is already there, but after the > 
> WSocket_Synchronized_WSAASyncSelect call, so what needs to be understood > is 
> if this call, and the related in the finally section of the try > block, is 
> really needed when not in a SSL request, as the non-SSL > version of the 
> component doesn't call this WSAASyncSelect method, and > this inherited 
> Do_FD_READ(msg) is the only code that will be called in a > non-SSL situation 
> anyway.

And this is exactly the problem... calling
WSocket_Synchronize_WSAAsyncSelect in Do_FD_READ when NOT in SSL.. by
just adding the SSL socket to the inheritance of classe (Activating
USE_SSL) will change the behavior or Do_FD_READ of NON SSL sockets..
and this screws our whole software and I bet it must screw other users
as well. I have a theory that it might be only affecting sockets with
Naggle disabled (which we do by default), but I have to test and
confirm it (because nobody else reported this issue and it is a very
very visible problem for us)

I added that verification code BEFORE
WSocket_Synchronize_WSAAsyncSelect so it won't be called if not in
SSL, but apparently Angus did not like it...

 > The move of this WSocket_Synchronized_WSAASyncSelect calls to this
> method, done in the V8.22, is about a fix in SSL, so I suppose this
> "Fixed SSL bug where two consecutive requests from a client would
leave > a server in a waiting state and not process any other
requests" is not > needed in non-SSL requests?

I agree that they must have added to fix some SSL problem, but that is
causing issue for NON-SSL sockets... while on SSL it looks like it is
working fine for us as well, but it screws the non-ssl sockets
entirely for us.

Eric
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] OpenSSL 1.1.0d and 1.0.2k support (?ric Fleming Bonilha)

[Éric Fleming Bonilha]

I may have overreacted... I trully admire the work of ICS team and am very
thankful for all the work the team has done for free for all these years to
provide a great communications library..

Im just very disapointed right now for being disregarded as I feel that
other users will have issues as well



On Feb 4, 2017 9:07 PM, "Éric Fleming Bonilha" <e...@digifort.com.br> wrote:

Well do you know how long I've been asking them to fix this broken code? It
has been months... The code is clearly broken and I've even provided demo
programs showing one of the issues.

Im already fed up... So I just fixed my local code and don't care anymore
if they will fix it or not... This bug has rendered ICS completely usuable
for us, and I bet many other users will have the same problems but may not
realise where the problem lies...

Angus might have done a lot, but he is simply ignoring this critical bug
for too long now.

I've provided a very good way to fix the code but he simply wont do it..

He even told in one message that he wont change anything because of only 1
user...

Or simply maybe nobody uses ICS anymore and nobody else has reported this...

Anyway.. I don't care anymore... I've tried to be friendly and that got me
nowhere... I will just keep fixing my local copy until this bug crashes
some other important ICS user and then maybe they will look into it...

On Feb 4, 2017 12:03 PM, <za...@hmisys.com> wrote:

>
> 
>
> > Or you can just ignore it and disregard your users...
> > This was my last attempt to give you an advice to fix a broken code...
> > I miss that old times when ICS team actually cared about the users...
>
> Wow, I would tell you to take a hike.
> Angus does great work for ICS and ICS users and you are being ...
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] OpenSSL 1.1.0d and 1.0.2k support (?ric Fleming Bonilha)

[Éric Fleming Bonilha]

Well do you know how long I've been asking them to fix this broken code? It
has been months... The code is clearly broken and I've even provided demo
programs showing one of the issues.

Im already fed up... So I just fixed my local code and don't care anymore
if they will fix it or not... This bug has rendered ICS completely usuable
for us, and I bet many other users will have the same problems but may not
realise where the problem lies...

Angus might have done a lot, but he is simply ignoring this critical bug
for too long now.

I've provided a very good way to fix the code but he simply wont do it..

He even told in one message that he wont change anything because of only 1
user...

Or simply maybe nobody uses ICS anymore and nobody else has reported this...

Anyway.. I don't care anymore... I've tried to be friendly and that got me
nowhere... I will just keep fixing my local copy until this bug crashes
some other important ICS user and then maybe they will look into it...

On Feb 4, 2017 12:03 PM,  wrote:

>
> 
>
> > Or you can just ignore it and disregard your users...
> > This was my last attempt to give you an advice to fix a broken code...
> > I miss that old times when ICS team actually cared about the users...
>
> Wow, I would tell you to take a hike.
> Angus does great work for ICS and ICS users and you are being ...
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] OpenSSL 1.1.0d and 1.0.2k support

[Éric Fleming Bonilha]

Angus

Did you have a chance to check that SSL problem on FD_READ that we have?

I have fixed my local copy of ICS and it works fine for both with and
without SSL (We are using SSL for one equipment we are testing).

inside TCustomSslWsocket.Do_FD_READ:

I added

if (not FSslEnable) or (FSocksState <> socksData) or
   (FHttpTunnelState <> htsData) then begin
inherited Do_FD_READ(msg);
Exit;
end;

before

WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
 FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
FD_CONNECT);

So, first thing FD_READ will do is checking if Ssl is enabled, if it is not
enabled it will simply call inherited without calling
WSocket_Synchronized_WSAASyncSelect that is causing all the issues we have,
and if SSL is active it will work as you designed

Anyway, I fixed my copy and it is working fine for us.. I was just
wondering if you are going to fix the official release because I'm sure
other users will have similar experiences as we are having and they won't
know what is happening, and what is causing their software to behave odd..

Or you can just ignore it and disregard your users...

This was my last attempt to give you an advice to fix a broken code... if
you think it is not broken, just ignore it... I've helped ICS team before
and some bugs that I reported were fixed by Arno Garrels... I miss that old
times when ICS team actually cared about the users...

Eric



On Thu, Feb 2, 2017 at 10:23 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> Four new zips for Win32 and Win64 versions of OpenSSL 1.1.0d and
> 1.0.2k should be downloadable from the Wiki at:
>
> http://wiki.overbyte.be/wiki/index.php/ICS_Download
>
> Unfortunately the web site itself is currently misbehaving so I've not
> been able to update the web page itself.
>
> Meanwhile, the new zips can be downloaded from my web site instead.
>
> https://www.magsys.co.uk/delphi/magics.asp
>
> SVN and the overnight zip have also been updated to include OpenSSL
> 1.1.0d and can also be downloaded from my web site.
>
> The new OpenSSL releases fix three moderate risk and one low risk
> security bugs that could potentially cause a server to crash.
>
> Angus
>
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

"Please see my reply yesterday, or perhaps you are not reading this list."

Angus, I'm sorry, maybe the list is not sending all e-mails to my mail
because I did not receive your reply about our web server, I had to
check on TWSocket achive to see that you replied. I think something
might be blocking the e-mails or the list might not be working well.

Anyway, I'm reading the thread from TWSocket archive now...

Thanks for letting me know our server has issue with .rar files I will
take a look to check it, but glad you already downloaded the file


Eric




On Mon, Dec 5, 2016 at 5:48 PM, Éric Fleming Bonilha <e...@digifort.com.br>
wrote:

> Angus
>
> I have also uploaded the file to Dropbox: https://www.dropbox.
> com/s/ces533zsvkseghs/ShowModal.rar?dl=0
>
> Eric
>
>
> On Fri, Dec 2, 2016 at 11:44 AM, Éric Fleming Bonilha <
> e...@digifort.com.br> wrote:
>
>> Angus
>>
>> Were you able to download the file?
>>
>> The file at WeTransfer will expire in 4 days
>>
>> Thanks
>>
>>
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

I have also uploaded the file to Dropbox:
https://www.dropbox.com/s/ces533zsvkseghs/ShowModal.rar?dl=0

Eric


On Fri, Dec 2, 2016 at 11:44 AM, Éric Fleming Bonilha <e...@digifort.com.br>
wrote:

> Angus
>
> Were you able to download the file?
>
> The file at WeTransfer will expire in 4 days
>
> Thanks
>
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

Were you able to download the file?

The file at WeTransfer will expire in 4 days

Thanks
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

I don't understand what you mean.. I can download the file from
http://www.digifort.com/files/temp/ShowModal.rar normally using Chrome or
Edge

And I did not see any message from you asking for that... If you sent any
message to this list, it was not received..

Eric



On Thu, Dec 1, 2016 at 3:38 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> > Did you have some time to evaluate this issue?
>
> Please don't send messages to my private email, I do read this list.
>
> I'm waiting for a reply to my message of 23 November, you need to
> set-up your web server MIME mapping properly so binary files can be
> downloaded.
>
> Angus
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Hello Angus

Did you have some time to evaluate this issue?

Thanks



On Wed, Nov 23, 2016 at 12:20 AM, Éric Fleming Bonilha <e...@digifort.com.br
> wrote:

> Angus
>
> I have created a very simple demo application that shows how USE_SSL
> changes the behavior of OnDataAvailable:
>
> Please download at
> http://www.digifort.com/files/temp/ShowModal.rar
>
> I included the source code and 2 compiled projects in Delphi XE7 with ICS
> from April 17, but I have tested with latest version and still the same
> problem happens
>
> This demo shows how OnDataAvailable affects calling Modal forms directly
> or indirectly from OnDataAvailable event, blocking all the socket
> communication while Modal form is open (You can see that MB sent keeps
> increasing, but receiving socket never process new windows socket messages
> until modal closes)
>
>
> I'm still trying to simulate the other most critical issue (That increased
> the CPU usage of our software H.264 video data parsing by 200%) but this is
> much more complicated to simulate because it is triggered / affected by
> peaks in CPU usage, but I will find a way to simulate and show how USE_SSL
> enabled has affected my whole system performance.
>
> Please remember that I'm not using SSL at this moment, the whole problem
> is because by adding TCustomSslWSocket to the inheritance changes the
> behavior of OnDataReceive even if I'm not using SSL and breaks our software
> (And I bet I should not be the only user being affected that this...)
>
> Thank you for taking a closer look at this issue, this is very critical to
> us
>
> Eric
>
>
>
> On Tue, Nov 22, 2016 at 3:45 AM, Angus Robertson - Magenta Systems Ltd <
> an...@magsys.co.uk> wrote:
>
>> I have already previously investigated your problem and have been
>> unable to reproduce it.
>>
>> I've explained previously why your fix is unacceptable.
>>
>> I have limited time for ICS, and need to spend it on development that
>> benefits all ICS users, not just one.
>>
>> The only way I will look at this yet again is if you supply a very
>> simple Delphi application that illustrates the problem and does not
>> require any external libraries, ie copy all twsocket units needed to
>> the application directory, create all components in code.
>>
>> It really would be so much easier if you rewrote your application to
>> remove blocking modal dialogs from event handlers.
>>
>> Angus
>>
>>
>>  Original Message 
>>
>> *Subject:* Re: [twsocket] Compiling with SSL active changes socket
>> behavior
>> *From:* Éric Fleming Bonilha <e...@digifort.com.br>
>> *To:* ICS support mailing <twsocket@lists.elists.org>
>> *Date:* Mon, 21 Nov 2016 17:32:42 -0500
>>
>> Hello all
>>
>> Could someone take a more serious look at this issue I've reported in
>> the
>> past???
>>
>> This has become a huge issue for us because the TCustomSslWSocket is
>> CHANGING the behavior of base socket EVEN IF SSL IS NOT IN USE!!!
>>
>> Today we got another issue that increased the processing of our
>> application
>> to 100% of CPU usage because OnDataReceive is being called much more
>> frequently.
>>
>> The whole problem is that Do_FD_READ(var Msg: TMessage) from
>> TCustomSslWSocket is changing the behavior of base socket
>>
>>
>> procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
>> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
>> configurable }
>> SetLength(Buffer, BuffSize);
>>
>>  { V8.22 moved here from Do_SSL_FD_READ  }
>> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>>  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
>> FD_CONNECT);
>> try
>> ...
>> finally
>>{ V8.22 moved here from Do_SSL_FD_READ }
>> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>>   FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_READ or FD_WRITE or
>> FD_CLOSE or FD_CONNECT);
>> end;
>>
>>
>>
>> Before calling WSocket_Synchronized_WSAASyncSelect you MUST check if
>> SSL is
>> active. Currently you are checking if SSL is active ONLY after TRY:
>>
>> if (not FSslEnable) or (FSocksState <> socksData) or
>>(FHttpTunnelState <> htsData) then begin
>> inherited Do_FD_READ(msg);
>> Exit;
>> end;
>>
>> This chunk of code should be placed BEFORE
>> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

I have created a very simple demo application that shows how USE_SSL
changes the behavior of OnDataAvailable:

Please download at
http://www.digifort.com/files/temp/ShowModal.rar

I included the source code and 2 compiled projects in Delphi XE7 with ICS
from April 17, but I have tested with latest version and still the same
problem happens

This demo shows how OnDataAvailable affects calling Modal forms directly or
indirectly from OnDataAvailable event, blocking all the socket
communication while Modal form is open (You can see that MB sent keeps
increasing, but receiving socket never process new windows socket messages
until modal closes)


I'm still trying to simulate the other most critical issue (That increased
the CPU usage of our software H.264 video data parsing by 200%) but this is
much more complicated to simulate because it is triggered / affected by
peaks in CPU usage, but I will find a way to simulate and show how USE_SSL
enabled has affected my whole system performance.

Please remember that I'm not using SSL at this moment, the whole problem is
because by adding TCustomSslWSocket to the inheritance changes the behavior
of OnDataReceive even if I'm not using SSL and breaks our software (And I
bet I should not be the only user being affected that this...)

Thank you for taking a closer look at this issue, this is very critical to
us

Eric



On Tue, Nov 22, 2016 at 3:45 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> I have already previously investigated your problem and have been
> unable to reproduce it.
>
> I've explained previously why your fix is unacceptable.
>
> I have limited time for ICS, and need to spend it on development that
> benefits all ICS users, not just one.
>
> The only way I will look at this yet again is if you supply a very
> simple Delphi application that illustrates the problem and does not
> require any external libraries, ie copy all twsocket units needed to
> the application directory, create all components in code.
>
> It really would be so much easier if you rewrote your application to
> remove blocking modal dialogs from event handlers.
>
> Angus
>
>
>  Original Message 
>
> *Subject:* Re: [twsocket] Compiling with SSL active changes socket
> behavior
> *From:* Éric Fleming Bonilha <e...@digifort.com.br>
> *To:* ICS support mailing <twsocket@lists.elists.org>
> *Date:* Mon, 21 Nov 2016 17:32:42 -0500
>
> Hello all
>
> Could someone take a more serious look at this issue I've reported in
> the
> past???
>
> This has become a huge issue for us because the TCustomSslWSocket is
> CHANGING the behavior of base socket EVEN IF SSL IS NOT IN USE!!!
>
> Today we got another issue that increased the processing of our
> application
> to 100% of CPU usage because OnDataReceive is being called much more
> frequently.
>
> The whole problem is that Do_FD_READ(var Msg: TMessage) from
> TCustomSslWSocket is changing the behavior of base socket
>
>
> procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
> SetLength(Buffer, BuffSize);
>
>  { V8.22 moved here from Do_SSL_FD_READ  }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
> FD_CONNECT);
> try
> ...
> finally
>{ V8.22 moved here from Do_SSL_FD_READ }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>   FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_READ or FD_WRITE or
> FD_CLOSE or FD_CONNECT);
> end;
>
>
>
> Before calling WSocket_Synchronized_WSAASyncSelect you MUST check if
> SSL is
> active. Currently you are checking if SSL is active ONLY after TRY:
>
> if (not FSslEnable) or (FSocksState <> socksData) or
>(FHttpTunnelState <> htsData) then begin
> inherited Do_FD_READ(msg);
> Exit;
> end;
>
> This chunk of code should be placed BEFORE
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
>
> This way, if we are NOT using SSL socket, the base socket would behave
> correctly, as it would simply call the inherited Do_FD_READ.
>
> Right now, when Do_FD_READ is called it process
> WSocket_Synchronized_WSAASyncSelect TWICE (Before, in the beginning of
> the
> method and inside FINALLY) if we are NOT using SSL.
>
> I have identified two critical issues with this behavior
>
> First is the one I have already reported, that we cannot call a
> ShowModal
> indirectly from OnDataAvailable because it would block all further
> socket
> processing
>
> Second problem we just i

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

While I try to create an example software to show the different behavior,
would you care to explain me why are you changing the behavior of the base
class in the SSL socket?

I mean... if I don't want to use SSL (but SSL socket is forced on class
derivation), why we should have a different socket behavior that is
introduced on Do_FD_READ than just using the socket without USE_SSL active?

When you introduce the WSocket_Synchronized_WSAASyncSelect inside
Do_FD_READ for SSL without actually checking first if SSL is indeed
enabled, you are just changing how Do_FD_READ would work without USE_SSL,
effectively changing the behavior or the sockets since now every Do_FD_READ
will call WSocket_Synchronized_WSAASyncSelect then inherited DO_FD_READ
then WSocket_Synchronized_WSAASyncSelect again on Finally block, which it
would NOT happen if USE_SSL is deactivated.

I just want to understand why you have to change the base socket behavior
(making the socket behave differently if USE_SSL is enabled) for all
sockets (even if I don't want to use SSL).

If I want SSL activated it would be fine to have this behavior, but if I
don't need SSL why changing the behavior? All other messages
like Do_FD_WRITE, Do_FD_CONNECT, Do_FD_CLOSE and Do_FD_ACCEPT will check if
the socket is SSL enabled first, without adding any extra behavior to the
base class, only changing in case the socket is actually SSL

Thanks



On Tue, Nov 22, 2016 at 3:45 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> I have already previously investigated your problem and have been
> unable to reproduce it.
>
> I've explained previously why your fix is unacceptable.
>
> I have limited time for ICS, and need to spend it on development that
> benefits all ICS users, not just one.
>
> The only way I will look at this yet again is if you supply a very
> simple Delphi application that illustrates the problem and does not
> require any external libraries, ie copy all twsocket units needed to
> the application directory, create all components in code.
>
> It really would be so much easier if you rewrote your application to
> remove blocking modal dialogs from event handlers.
>
> Angus
>
>
>  Original Message 
>
> *Subject:* Re: [twsocket] Compiling with SSL active changes socket
> behavior
> *From:* Éric Fleming Bonilha <e...@digifort.com.br>
> *To:* ICS support mailing <twsocket@lists.elists.org>
> *Date:* Mon, 21 Nov 2016 17:32:42 -0500
>
> Hello all
>
> Could someone take a more serious look at this issue I've reported in
> the
> past???
>
> This has become a huge issue for us because the TCustomSslWSocket is
> CHANGING the behavior of base socket EVEN IF SSL IS NOT IN USE!!!
>
> Today we got another issue that increased the processing of our
> application
> to 100% of CPU usage because OnDataReceive is being called much more
> frequently.
>
> The whole problem is that Do_FD_READ(var Msg: TMessage) from
> TCustomSslWSocket is changing the behavior of base socket
>
>
> procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
> SetLength(Buffer, BuffSize);
>
>  { V8.22 moved here from Do_SSL_FD_READ  }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
> FD_CONNECT);
> try
> ...
> finally
>{ V8.22 moved here from Do_SSL_FD_READ }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>   FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_READ or FD_WRITE or
> FD_CLOSE or FD_CONNECT);
> end;
>
>
>
> Before calling WSocket_Synchronized_WSAASyncSelect you MUST check if
> SSL is
> active. Currently you are checking if SSL is active ONLY after TRY:
>
> if (not FSslEnable) or (FSocksState <> socksData) or
>(FHttpTunnelState <> htsData) then begin
> inherited Do_FD_READ(msg);
> Exit;
> end;
>
> This chunk of code should be placed BEFORE
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
>
> This way, if we are NOT using SSL socket, the base socket would behave
> correctly, as it would simply call the inherited Do_FD_READ.
>
> Right now, when Do_FD_READ is called it process
> WSocket_Synchronized_WSAASyncSelect TWICE (Before, in the beginning of
> the
> method and inside FINALLY) if we are NOT using SSL.
>
> I have identified two critical issues with this behavior
>
> First is the one I have already reported, that we cannot call a
> ShowModal
> indirectly from OnDataAvailable because it would block all further
> socket
> processing
>

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

I will provide the samples for you... It is NOT only the blocking of
showmodal dialogs (that I have already fixed by changing some behaviors on
my base communication classes), the main issue right now is the
unacceptable behavior of OnDataAvailable being called much (MUCH) more
frequently which triggers our data parsing much more often that has to
analyse all received data and increased processor usage by about 200%
bringing the performance of our application to its knees...

I will try to make a sample application that shows exactly both issues for
you

Eric



On Tue, Nov 22, 2016 at 3:45 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> I have already previously investigated your problem and have been
> unable to reproduce it.
>
> I've explained previously why your fix is unacceptable.
>
> I have limited time for ICS, and need to spend it on development that
> benefits all ICS users, not just one.
>
> The only way I will look at this yet again is if you supply a very
> simple Delphi application that illustrates the problem and does not
> require any external libraries, ie copy all twsocket units needed to
> the application directory, create all components in code.
>
> It really would be so much easier if you rewrote your application to
> remove blocking modal dialogs from event handlers.
>
> Angus
>
>
>  Original Message 
>
> *Subject:* Re: [twsocket] Compiling with SSL active changes socket
> behavior
> *From:* Éric Fleming Bonilha <e...@digifort.com.br>
> *To:* ICS support mailing <twsocket@lists.elists.org>
> *Date:* Mon, 21 Nov 2016 17:32:42 -0500
>
> Hello all
>
> Could someone take a more serious look at this issue I've reported in
> the
> past???
>
> This has become a huge issue for us because the TCustomSslWSocket is
> CHANGING the behavior of base socket EVEN IF SSL IS NOT IN USE!!!
>
> Today we got another issue that increased the processing of our
> application
> to 100% of CPU usage because OnDataReceive is being called much more
> frequently.
>
> The whole problem is that Do_FD_READ(var Msg: TMessage) from
> TCustomSslWSocket is changing the behavior of base socket
>
>
> procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
> SetLength(Buffer, BuffSize);
>
>  { V8.22 moved here from Do_SSL_FD_READ  }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
> FD_CONNECT);
> try
> ...
> finally
>{ V8.22 moved here from Do_SSL_FD_READ }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>   FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_READ or FD_WRITE or
> FD_CLOSE or FD_CONNECT);
> end;
>
>
>
> Before calling WSocket_Synchronized_WSAASyncSelect you MUST check if
> SSL is
> active. Currently you are checking if SSL is active ONLY after TRY:
>
> if (not FSslEnable) or (FSocksState <> socksData) or
>(FHttpTunnelState <> htsData) then begin
> inherited Do_FD_READ(msg);
> Exit;
> end;
>
> This chunk of code should be placed BEFORE
> BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now
> configurable }
>
> This way, if we are NOT using SSL socket, the base socket would behave
> correctly, as it would simply call the inherited Do_FD_READ.
>
> Right now, when Do_FD_READ is called it process
> WSocket_Synchronized_WSAASyncSelect TWICE (Before, in the beginning of
> the
> method and inside FINALLY) if we are NOT using SSL.
>
> I have identified two critical issues with this behavior
>
> First is the one I have already reported, that we cannot call a
> ShowModal
> indirectly from OnDataAvailable because it would block all further
> socket
> processing
>
> Second problem we just identified today and created a HUGE issue with
> many
> of our customers, because the application is now very unstable and
> using a
> lot of CPU. We have identified that OnDataAvailable is called MUCH MORE
> frequently than before
>
> To solve the issues we are compiling ICS without SSL, but we will need
> SSL
> soon, and if we activate SSL it will simply break our whole software
> because you are changing the receive behavior of all sockets
>
> Could someone please take this problem seriously and analyse the issue?
>
> I can show you over a TeamViewer session the effects of this on our
> software
>
>
> Thank you!
>
> Eric Fleming Bonilha
>
>
> On Wed, Aug 31, 2016 at 6:18 PM, Éric Fleming Bonilha
> <e...@digifort.com

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Hello all

Could someone take a more serious look at this issue I've reported in the
past???

This has become a huge issue for us because the TCustomSslWSocket is
CHANGING the behavior of base socket EVEN IF SSL IS NOT IN USE!!!

Today we got another issue that increased the processing of our application
to 100% of CPU usage because OnDataReceive is being called much more
frequently.

The whole problem is that Do_FD_READ(var Msg: TMessage) from
TCustomSslWSocket is changing the behavior of base socket


procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now configurable }
SetLength(Buffer, BuffSize);

 { V8.22 moved here from Do_SSL_FD_READ  }
WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
 FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
FD_CONNECT);
try
...
finally
   { V8.22 moved here from Do_SSL_FD_READ }
WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_READ or FD_WRITE or
FD_CLOSE or FD_CONNECT);
end;



Before calling WSocket_Synchronized_WSAASyncSelect you MUST check if SSL is
active. Currently you are checking if SSL is active ONLY after TRY:

if (not FSslEnable) or (FSocksState <> socksData) or
   (FHttpTunnelState <> htsData) then begin
inherited Do_FD_READ(msg);
Exit;
end;

This chunk of code should be placed BEFORE
BuffSize := (GSSL_BUFFER_SIZE * 2)-1;  { V8.27 size now configurable }

This way, if we are NOT using SSL socket, the base socket would behave
correctly, as it would simply call the inherited Do_FD_READ.

Right now, when Do_FD_READ is called it process
WSocket_Synchronized_WSAASyncSelect TWICE (Before, in the beginning of the
method and inside FINALLY) if we are NOT using SSL.

I have identified two critical issues with this behavior

First is the one I have already reported, that we cannot call a ShowModal
indirectly from OnDataAvailable because it would block all further socket
processing

Second problem we just identified today and created a HUGE issue with many
of our customers, because the application is now very unstable and using a
lot of CPU. We have identified that OnDataAvailable is called MUCH MORE
frequently than before

To solve the issues we are compiling ICS without SSL, but we will need SSL
soon, and if we activate SSL it will simply break our whole software
because you are changing the receive behavior of all sockets

Could someone please take this problem seriously and analyse the issue?

I can show you over a TeamViewer session the effects of this on our software


Thank you!

Eric Fleming Bonilha


On Wed, Aug 31, 2016 at 6:18 PM, Éric Fleming Bonilha <e...@digifort.com.br>
wrote:

> Angus
>
> Would you at least take a look at it again? Adding that line in SSL socket
> before actually checking if SSL is being used is changing the behavior of
> the base class and breaking compatibility
>
>
>> Not quite sure why the V8.22 change in wsocket made this change, but it
>> was an important bug fix that preventing ICS locking up on heavy
>> traffic, so very reluctant to change it.
>>
>> I think I saw this problem in one of my own applications a few days ago,
>> when a log window was opened, it's on my list to investigate.
>>
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Straw poll - Should ICS continue to support old OpenSSL versions?

[Éric Fleming Bonilha]

- Drop older version
- Remove obsolete properties





On Wed, Oct 5, 2016 at 5:46 AM, Angus Robertson - Magenta Systems Ltd <
an...@magsys.co.uk> wrote:

> The recent introduction of OpenSSL 1.1.0 required a lot of internal
> changes in ICS which were made backward compatible with older versions.
>
>
> ICS V8.34 currently supports three OpenSSL versions, but it would make
> future maintenance of ICS easier if only the latest OpenSSL 1.1.0 were
> supported.  Support for 1.0.1 ceases this year anyway, but 1.0.2 is a
> long term version for with support continuing for two or three years.
>
> Currently there is a lot of conditional code where different versions
> differ in some way which leads to bloat.  We are also continuing to
> support some obsolete properties that no longer work in 1.1.0, to make
> ICS fully backward compatible.
>
> Benefits of continued support for OpenSSL 1.0.2
> - 1.0.2 has longer history, proven reliable
> - 1.1.0 only released six weeks ago, perhaps buggy
>
> Benefits of dropping support for OpenSSL 1.0.2
> - easier to maintain ICS, less ICS code
> - could drop support for obsolete properties, like SslVersionMethod
> - no actual new bugs introduced in 1.1.0, yet
>
> So can I please ask users of ICS to comment to this message saying
> whether new versions of ICS should continue to support old versions of
> OpenSSL.
>
> Also, would you prefer obsolete properties to remain so old programs
> don't need any changes, but perhaps confusing new users who may not
> realise these properties are useless.
>
> Angus
>
>
>
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Angus

Would you at least take a look at it again? Adding that line in SSL socket
before actually checking if SSL is being used is changing the behavior of
the base class and breaking compatibility


> Not quite sure why the V8.22 change in wsocket made this change, but it
> was an important bug fix that preventing ICS locking up on heavy
> traffic, so very reluctant to change it.
>
> I think I saw this problem in one of my own applications a few days ago,
> when a log window was opened, it's on my list to investigate.
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] RES: Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Tecnically, a showmodal is not total blocking process, although it blocks
the current flow. Internally it will call a message loop that will keep
processing windows messages so the application can run. 

This have always worked for years without any issue and my whole software is
based on that, basically opening screens as a response to a network command.

Changing to postmessage it not an option right now because of all the work
involved.

Basically when I´m on a ShowModal from inside a ICS event handler (for data
available) (It is not called directly, but indirectly), and then it will
keep processing new windows messages and those new windows messages will be
socket messages that will keep processing.

With the new implementation on the SSL layer it breaks everything and it
does not allow windows to process any more socket messages until showmodal
is finished. 

That is a single line that was added to SSL socket. 

So, the question is why this method 
WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
 FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
FD_CONNECT);
Is being called from TCustomSslWSocket.Do_FD_READ(var Msg: TMessage) before
actually checking if SSL socket is in use?

When you do this you are basically altering the functioning of the base
class even if we don´t use SSL...




> We have always said any processing within ICS event handlers will stop
further data being processed.  Calling a modal dialog is a blocking event,
so I'm surprised your application ever worked, rather than it's stopped
working. 

> A better solution would be to post a message that opens the window, so the
event handler continues processing.  

> Not quite sure why the V8.22 change in wsocket made this change, but it
was an important bug fix that preventing ICS locking up on heavy traffic, so
very reluctant to change it.  

> I think I saw this problem in one of my own applications a few days ago,
when a log window was opened, it's on my list to investigate. 

-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] RES: Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Nobody is receiving these e-mails??

 

De: Éric Fleming Bonilha [mailto:e...@digifort.com.br] 
Enviada em: Monday, August 15, 2016 7:08 PM
Para: ICS support mailing
Assunto: Re: Compiling with SSL active changes socket behavior

 

Could anyone check this? Or at least acknowledge that it was received? 

 

Thanks




 

On Sat, Aug 13, 2016 at 3:13 PM, Éric Fleming Bonilha <e...@digifort.com.br> 
wrote:

I evaluated a little more on this behavior and realized that actually many many 
parts of our software will open a SHOWMODAL dialog to the response of a message 
from the server, that is parsed with our protocol parser from within the 
OnDataAvailable event of the socket, and this behavior completely broken our 
software because now everytime it opens a form from within the dataavailable 
event (indirectly), the socket will no longer process any more messages while 
the form is open, and I guess that this is wrong...

 

Since ICS is designed to be asynchronous it should not prevent me from opening 
a form using showmodal as a response of a socket message...

 

Just to make clear, I'm using a PLAIN TCP socket without SSL

 

Well, researching a little more on the issue I found this:

 

procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);

var

Len: Integer; // How much to receive

Buffer : array [0..(SSL_BUFFER_SIZE * 2) -1] of AnsiChar;

NumRead: Integer;

nError : Integer;

Res: Integer;

PBuf   : TWSocketData;

Dummy  : Byte;

begin

 { V8.22 moved here from Do_SSL_FD_READ  }

WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}

 FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or 
FD_CONNECT);

try

if (not FSslEnable) or (FSocksState <> socksData) or

   (FHttpTunnelState <> htsData) then begin

inherited Do_FD_READ(msg);

Exit;

end;

 

 

 

The problem seems to be with this first call from CustomSslWSocket.Do_FD_READ:

WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF} FHSocket, 
Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or FD_CONNECT);

 

If I just comment out this routine, then the problem is solved but I guess that 
SSL will not work (I'm not using SSL at this moment anyway)

 

In my point of view, if I'm using a plain socket without SSL, this 
TCustomSslWSocket class should NOT interfere with the operation of the socket 
and should just call inherited Do_FD_READ in case SSL is not being used instead 
of calling some socket routine...

 

Would someone please look into this issue? I can make some example application 
to demonstrate the behavior (I just don't know how to properly fix it)

 

Thanks




 

 

On Sat, Aug 13, 2016 at 2:50 PM, Éric Fleming Bonilha <e...@digifort.com.br> 
wrote:

Hi

 

I recently came across an issue with our software after updating to ICS 8.26

 

We previously used ICS 8.06 and its default compiling directives did not add 
SSL support, but 8.26 by default activates SSL and makes TWSocket class to be 
derived from SSL socket

 

I don't know if the problem is related to the SSL socket derivation but 
probably yes since deactivating SSL will make socket behave like before, which 
makes me think that could be a bug because socket behavior should not change 
because I activated SSL (even though I don't use it now, our project will 
require it later)

 

So, basically the issue is in a message loop processing I guess

 

In our CLIENT application there is a code that opens a form using SHOWMODAL 
when it receives a message from the server (Client is connected to server using 
our own protocol). As you probably know, SHOWMODAL will create a windows 
message loop so application can keep processing windows messsages. The problem 
is that apparently when compiling with SSL, when I call a showmodal from my 
protocol parser (that is called from OnDataAvailable) basically making 
showmodal being indirectly called from within the event handler of socket 
OnDataAvailable, it will not process any more socket messages while my form is 
open, although the application keeps processing all other windows messages it 
looks like any socket message (at least data rcv messages) are no longer being 
processed and my socket gets "hang" while form called with SHOWMODAL is open. 

 

This behavior occurs only when activating SSL, but if I deactivate SLL support 
it will work just fine, and while my form is open with showmodal, the socket 
still keeps processing new messages (from within the message loop created by 
showmodal calling).

 

I guess I know I should not indirectly block OnDataAvailable event handler by 
calling a showmodal because of show it works (creates a new message loop and 
things can get messy), but this works just fine without the USE_SSL define, so, 
I would like to understand why this happens.

 

I could just disable USE_SSL for now, but I know I 

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Could anyone check this? Or at least acknowledge that it was received?

Thanks



On Sat, Aug 13, 2016 at 3:13 PM, Éric Fleming Bonilha <e...@digifort.com.br>
wrote:

> I evaluated a little more on this behavior and realized that actually many
> many parts of our software will open a SHOWMODAL dialog to the response of
> a message from the server, that is parsed with our protocol parser from
> within the OnDataAvailable event of the socket, and this behavior
> completely broken our software because now everytime it opens a form from
> within the dataavailable event (indirectly), the socket will no longer
> process any more messages while the form is open, and I guess that this is
> wrong...
>
> Since ICS is designed to be asynchronous it should not prevent me from
> opening a form using showmodal as a response of a socket message...
>
> Just to make clear, I'm using a PLAIN TCP socket without SSL
>
> Well, researching a little more on the issue I found this:
>
> procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
> var
> Len: Integer; // How much to receive
> Buffer : array [0..(SSL_BUFFER_SIZE * 2) -1] of AnsiChar;
> NumRead: Integer;
> nError : Integer;
> Res: Integer;
> PBuf   : TWSocketData;
> Dummy  : Byte;
> begin
>  { V8.22 moved here from Do_SSL_FD_READ  }
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
>  FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
> FD_CONNECT);
> try
> if (not FSslEnable) or (FSocksState <> socksData) or
>(FHttpTunnelState <> htsData) then begin
> inherited Do_FD_READ(msg);
> Exit;
> end;
>
>
>
> The problem seems to be with this first call from
> CustomSslWSocket.Do_FD_READ:
> WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
> FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or FD_CONNECT);
>
> If I just comment out this routine, then the problem is solved but I guess
> that SSL will not work (I'm not using SSL at this moment anyway)
>
> In my point of view, if I'm using a plain socket without SSL,
> this TCustomSslWSocket class should NOT interfere with the operation of the
> socket and should just call inherited Do_FD_READ in case SSL is not being
> used instead of calling some socket routine...
>
> Would someone please look into this issue? I can make some example
> application to demonstrate the behavior (I just don't know how to properly
> fix it)
>
> Thanks
>
>
>
> On Sat, Aug 13, 2016 at 2:50 PM, Éric Fleming Bonilha <
> e...@digifort.com.br> wrote:
>
>> Hi
>>
>> I recently came across an issue with our software after updating to ICS
>> 8.26
>>
>> We previously used ICS 8.06 and its default compiling directives did not
>> add SSL support, but 8.26 by default activates SSL and makes TWSocket class
>> to be derived from SSL socket
>>
>> I don't know if the problem is related to the SSL socket derivation but
>> probably yes since deactivating SSL will make socket behave like before,
>> which makes me think that could be a bug because socket behavior should not
>> change because I activated SSL (even though I don't use it now, our project
>> will require it later)
>>
>> So, basically the issue is in a message loop processing I guess
>>
>> In our CLIENT application there is a code that opens a form using
>> SHOWMODAL when it receives a message from the server (Client is connected
>> to server using our own protocol). As you probably know, SHOWMODAL will
>> create a windows message loop so application can keep processing windows
>> messsages. The problem is that apparently when compiling with SSL, when I
>> call a showmodal from my protocol parser (that is called from
>> OnDataAvailable) basically making showmodal being indirectly called from
>> within the event handler of socket OnDataAvailable, it will not process any
>> more socket messages while my form is open, although the application keeps
>> processing all other windows messages it looks like any socket message (at
>> least data rcv messages) are no longer being processed and my socket gets
>> "hang" while form called with SHOWMODAL is open.
>>
>> This behavior occurs only when activating SSL, but if I deactivate SLL
>> support it will work just fine, and while my form is open with showmodal,
>> the socket still keeps processing new messages (from within the message
>> loop created by showmodal calling).
>>
>> I guess I know I should not indirectly block OnDataAvailable even

Re: [twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

I evaluated a little more on this behavior and realized that actually many
many parts of our software will open a SHOWMODAL dialog to the response of
a message from the server, that is parsed with our protocol parser from
within the OnDataAvailable event of the socket, and this behavior
completely broken our software because now everytime it opens a form from
within the dataavailable event (indirectly), the socket will no longer
process any more messages while the form is open, and I guess that this is
wrong...

Since ICS is designed to be asynchronous it should not prevent me from
opening a form using showmodal as a response of a socket message...

Just to make clear, I'm using a PLAIN TCP socket without SSL

Well, researching a little more on the issue I found this:

procedure TCustomSslWSocket.Do_FD_READ(var Msg: TMessage);
var
Len: Integer; // How much to receive
Buffer : array [0..(SSL_BUFFER_SIZE * 2) -1] of AnsiChar;
NumRead: Integer;
nError : Integer;
Res: Integer;
PBuf   : TWSocketData;
Dummy  : Byte;
begin
 { V8.22 moved here from Do_SSL_FD_READ  }
WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
 FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or
FD_CONNECT);
try
if (not FSslEnable) or (FSocksState <> socksData) or
   (FHttpTunnelState <> htsData) then begin
inherited Do_FD_READ(msg);
Exit;
end;



The problem seems to be with this first call from
CustomSslWSocket.Do_FD_READ:
WSocket_Synchronized_WSAASyncSelect({$IFDEF POSIX}Self,{$ENDIF}
FHSocket, Handle, FMsg_WM_ASYNCSELECT, FD_WRITE or FD_CLOSE or FD_CONNECT);

If I just comment out this routine, then the problem is solved but I guess
that SSL will not work (I'm not using SSL at this moment anyway)

In my point of view, if I'm using a plain socket without SSL,
this TCustomSslWSocket class should NOT interfere with the operation of the
socket and should just call inherited Do_FD_READ in case SSL is not being
used instead of calling some socket routine...

Would someone please look into this issue? I can make some example
application to demonstrate the behavior (I just don't know how to properly
fix it)

Thanks



On Sat, Aug 13, 2016 at 2:50 PM, Éric Fleming Bonilha <e...@digifort.com.br>
wrote:

> Hi
>
> I recently came across an issue with our software after updating to ICS
> 8.26
>
> We previously used ICS 8.06 and its default compiling directives did not
> add SSL support, but 8.26 by default activates SSL and makes TWSocket class
> to be derived from SSL socket
>
> I don't know if the problem is related to the SSL socket derivation but
> probably yes since deactivating SSL will make socket behave like before,
> which makes me think that could be a bug because socket behavior should not
> change because I activated SSL (even though I don't use it now, our project
> will require it later)
>
> So, basically the issue is in a message loop processing I guess
>
> In our CLIENT application there is a code that opens a form using
> SHOWMODAL when it receives a message from the server (Client is connected
> to server using our own protocol). As you probably know, SHOWMODAL will
> create a windows message loop so application can keep processing windows
> messsages. The problem is that apparently when compiling with SSL, when I
> call a showmodal from my protocol parser (that is called from
> OnDataAvailable) basically making showmodal being indirectly called from
> within the event handler of socket OnDataAvailable, it will not process any
> more socket messages while my form is open, although the application keeps
> processing all other windows messages it looks like any socket message (at
> least data rcv messages) are no longer being processed and my socket gets
> "hang" while form called with SHOWMODAL is open.
>
> This behavior occurs only when activating SSL, but if I deactivate SLL
> support it will work just fine, and while my form is open with showmodal,
> the socket still keeps processing new messages (from within the message
> loop created by showmodal calling).
>
> I guess I know I should not indirectly block OnDataAvailable event handler
> by calling a showmodal because of show it works (creates a new message loop
> and things can get messy), but this works just fine without the USE_SSL
> define, so, I would like to understand why this happens.
>
> I could just disable USE_SSL for now, but I know I will soon need this is
> our future release and I just want to understand why this socket behavior
> changed and what should be the "correct" way of working with it
>
> Thanks
>
> Eric
>
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] Compiling with SSL active changes socket behavior

[Éric Fleming Bonilha]

Hi

I recently came across an issue with our software after updating to ICS 8.26

We previously used ICS 8.06 and its default compiling directives did not
add SSL support, but 8.26 by default activates SSL and makes TWSocket class
to be derived from SSL socket

I don't know if the problem is related to the SSL socket derivation but
probably yes since deactivating SSL will make socket behave like before,
which makes me think that could be a bug because socket behavior should not
change because I activated SSL (even though I don't use it now, our project
will require it later)

So, basically the issue is in a message loop processing I guess

In our CLIENT application there is a code that opens a form using SHOWMODAL
when it receives a message from the server (Client is connected to server
using our own protocol). As you probably know, SHOWMODAL will create a
windows message loop so application can keep processing windows messsages.
The problem is that apparently when compiling with SSL, when I call a
showmodal from my protocol parser (that is called from OnDataAvailable)
basically making showmodal being indirectly called from within the event
handler of socket OnDataAvailable, it will not process any more socket
messages while my form is open, although the application keeps processing
all other windows messages it looks like any socket message (at least data
rcv messages) are no longer being processed and my socket gets "hang" while
form called with SHOWMODAL is open.

This behavior occurs only when activating SSL, but if I deactivate SLL
support it will work just fine, and while my form is open with showmodal,
the socket still keeps processing new messages (from within the message
loop created by showmodal calling).

I guess I know I should not indirectly block OnDataAvailable event handler
by calling a showmodal because of show it works (creates a new message loop
and things can get messy), but this works just fine without the USE_SSL
define, so, I would like to understand why this happens.

I could just disable USE_SSL for now, but I know I will soon need this is
our future release and I just want to understand why this socket behavior
changed and what should be the "correct" way of working with it

Thanks

Eric
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] ICS with FPC

[Éric Fleming Bonilha]

Dear ICS team

Does ICS supports FreePascal?

I've seen from the include file that some work has been done in the past to
make it compatible with FreePascal but maybe that was old work.

I would like to know if you have any plans to support Lazarus and
FreePascal?

We are now starting to worry about the future of Delphi since the sale to
Idera and departure of their main chief Cientist Allen Bauer.

We are considering our options for the future development of our products
and the clear path would be Free Pascal.

I'm in fact very impressed with the compiler itself, but a bit letdown by
Lazarus IDE. Hopefully in the future FPC/Lazarus would catch up with Delphi
should it end.

Thanks!
-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] Socket hanging at ntdll routine

[Éric Fleming Bonilha]

Hello all

 

We have stumbled at a very strange issue with our software that uses ICS as
core for sockets.

 

This is not the first time that it happens and has happened in different
customers but it is kind of rare.. What happens is that ASyncReceive routine
from TCustomWSocket hangs in Ics_ioctlsocket routine.

 

We have installed a debug in our customers and got the following:

 

|Running Thread: ID=4276; Parent=2992; Priority=0
|

|Class=TServiceThread; Name= (Vcl.SvcMgr.TServiceThread.Execute)
|

|DeadLock=0; Wait Chain=
|

|Comment=
|

|---

-|

|7FFE|03 ||76F1F911|ntdll.dll |0001F911|ntdll
|   |ZwDeviceIoControlFile
|  |

|0020|03 |05EFF7A4|767830A1|WS2_32.dll|30A1|WS2_32
|   |ioctlsocket
|  |

|0020|04 |05EFF7D0|00810CC7|Server.exe
|00410CC7|OverbyteIcsWinsock |
|Ics_ioctlsocket   |1177[3]   |

|0020|04 |05EFF7F0|008241B4|Server.exe
|004241B4|OverbyteIcsWSocket |TCustomWSocket
|ASyncReceive  |7071[30]  |

|0020|03 |05EFF84C|75E20810|KERNELBASE.dll|00010810|KERNELBASE
|   |WaitForSingleObjectEx
|  |

|0020|04 |05EFFC2C|00824292|Server.exe
|00424292|OverbyteIcsWSocket |TCustomWSocket
|Do_FD_READ|7104[5]   |

|0020|04 |05EFFC3C|0082442F|Server.exe
|0042442F|OverbyteIcsWSocket |TCustomWSocket
|WMASyncSelect |7254[20]  |

|0020|04 |05EFFC50|00822D1A|Server.exe
|00422D1A|OverbyteIcsWSocket |TCustomWSocket
|WndProc   |6181[4]   |

|0020|04 |05EFFC74|00832A00|Server.exe
|00432A00|OverbyteIcsWSocket |TCustomHttpTunnelWSocket
|WndProc   |19157[9]  |

|0020|04 |05EFFC8C|0082EF5C|Server.exe
|0042EF5C|OverbyteIcsWSocket |TCustomLineWSocket
|WndProc   |10944[12] |

|0020|04 |05EFFCA4|0081A9CA|Server.exe
|0041A9CA|OverbyteIcsWndControl  |TIcsWndHandler
|WndProc   |1133[6]   |

|0020|04 |05EFFCCC|00819C82|Server.exe
|00419C82|OverbyteIcsWndControl  |
|WndControlWindowsProc |466[30]   |

|0020|03 |05EFFD1C|76146D35|USER32.dll|00016D35|USER32
|   |(possible
GetThreadDesktop+210)   |  |

|0020|03 |05EFFD94|761477BF|USER32.dll|000177BF|USER32
|   |(possible
CharPrevW+307)  |  |

|0020|03 |05EFFDF4|76147885|USER32.dll|00017885|USER32
|   |DispatchMessageW
|  |

|0020|03 |05EFFE04|00641F04|Server.exe|00241F04|Vcl.SvcMgr
|TServiceThread |ProcessRequests
|  |

|0020|03 |05EFFE68|00641BB5|Server.exe|00241BB5|Vcl.SvcMgr
|TServiceThread |Execute
|  |

|0020|03 |05EFFEEC|004CDC9E|Server.exe|000CDC9E|System.Classes
|   |ThreadProc
|  |

|0020|03 |05EFFF1C|0040A008|Server.exe|A008|System
|   |ThreadWrapper
|  |

|0020|04 |05EFFF30|00650AA2|Server.exe
|00250AA2|EExceptionManager  |
|DefaultThreadHandleException  |2852[5]   |

|0020|04 |05EFFF78|00553423|Server.exe|00153423|EThreadsManager
|   |ThreadWrapper
|611[11]   |

|0020|03 |05EFFF8C|75FD33C8|kernel32.dll  |000133C8|kernel32
|   |BaseThreadInitThunk
|  |

|7FFE|03 ||00641AA8|Server.exe|00241AA8|Vcl.SvcMgr
|TServiceThread |Create
|  |

|---

-|

 

What I can see from here is that it is hang in ZwDeviceIoControlFile routine
from ntdll that was called from ioctlsocket which was called by ICS.

 

In this debug that was captured, the main thread is hang because that is
where this specific socket is running, but we have found this issue in
another part 

[twsocket] RES: Attach to a target thread?

[Éric Fleming Bonilha]

François

 I share this opinion.
 I would let the client sockets attached to a single the thread (no
necessarily the main thread) and handle all communication. Once a job has
been received, then the data is passed to a worker thread, thru a queue, for
processing. Probably a single queue serviced by a pool of thread is enough.
 
What are your thoughts in having all communication in a single thread? In my
application we are starting to face some bottlenecks because all processing
of incomming data is received by the main thread, what we did was to change
the handle of some connections by detaching from main thread and attaching
to another thread to process the messages. The connections that we did that
are the ones that were causing the bottleneck.

So, my question is, in order to ICS to scale better in a multi-core machine
we should process the incoming Windows messages in different threads (Of
course, in a VERY WELL designed architecture to avoid deadlocks and
conflics) right?

Eric

-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] RES: Multi-Threading in TWSocketServer

[Éric Fleming Bonilha]

 No, ICS is not thread-safe (out of the box)! Like the VCL is not
thread-safe as well. Use of ThreadDetach and ThreadAttach just ensures that
ICS-events trigger in the attached thread's context. There's more to
serialize like access to the client list, etc. 

Ok, so if I use ThreadAttach and ThreadDetach in clients from TWSocketServer
I may have trouble?

My code  inside the event triggers is fine for multi-thread

Eric 

-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

[twsocket] Multi-Threading in TWSocketServer

[Éric Fleming Bonilha]

Hello

 

I have an application that TWSocketServer is running in main thread,
consequently, all messages from clients are also processed by the main
thread loop

 

I have been able to change the processing thread of the clientes from main
thread to a pool of threads that I have (By using ThreadDetach and
ThreadAttach) and it has been working fine, but I would like to know if this
kind of operation is advised or if I can run into some erros... is ICS
totally thread-aware for this kind of thing?



Thanks

Eric

-- 
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be