Re: [squid-users] external helper development

[Eliezer Croitoru]

Hey David,

 

Since the handle_stdout runs in it’s own thread it’s sole purpose is to send 
results to stdout.

If I will run the next code in a simple software without the 0.5 sleep time:

 while RUNNING:

 if quit > 0:

   return

 while len(queue) > 0:

 item = queue.pop(0)

 sys.stdout.write(item)

 sys.stdout.flush()

 time.sleep(0.5)

 

 

what will happen is that the software will run with 100% CPU looping over and 
over on the size of the queue
while sometimes it will spit some data to stdout.

Adding a small delay with 0.5 secs will allow some “idle” time for the cpu in 
the loop preventing it from consuming
all the CPU time.

It’s a very old technique and there are others which are more efficient but 
it’s enough to demonstrate that a simple
threaded helper is much better then any PHP code that was not meant to be 
running as a STDIN/OUT daemon/helper software.

 

All The Bests,

Eliezer

 



Eliezer Croitoru

NgTech, Tech Support

Mobile: +972-5-28704261

Email: ngtech1...@gmail.com  

 

From: David Touzeau  
Sent: Monday, February 7, 2022 02:42
To: Eliezer Croitoru ; squid-users@lists.squid-cache.org
Subject: Re: [squid-users] external helper development

 

Sorry  Elizer

It was a mistake... No, your code is clean..
Impressive for the first shot
Many thanks for your example, we will run our stress tool to see the 
difference...

Just a question

Why did you send 500 milliseconds of sleep in the handle_stdoud ? Is it for let 
squid closing the pipe ?




Le 06/02/2022 à 11:46, Eliezer Croitoru a écrit :

Hey David,

 

Not a fully completed helper but it seems to works pretty nice and might be 
better then what exist already:

https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py

 

#!/usr/bin/env python

 

import sys

import time

import urllib.request

import signal

import threading

 

#set debug mode for True or False

debug = False

#debug = True

queue = []

threads = []

 

RUNNING = True

 

quit = 0

rand_api_url =   
"https://cloud1.ngtech.co.il/api/test.php";

 

def sig_handler(signum, frame):

sys.stderr.write("Signal is received:" + str(signum) + "\n")

global quit

quit = 1

global RUNNING

RUNNING=False

 

 

def handle_line(line):

 if not RUNNING:

 return

 if not line:

 return

 if quit > 0:

 return

 

 arr = line.split()

 response = urllib.request.urlopen( rand_api_url )

 response_text = response.read()

 

 queue.append(arr[0] + " " + response_text.decode("utf-8"))

 

def handle_stdout(n):

 while RUNNING:

 if quit > 0:

   return

 while len(queue) > 0:

 item = queue.pop(0)

 sys.stdout.write(item)

 sys.stdout.flush()

 time.sleep(0.5)

 

def handle_stdin(n):

while RUNNING:

 line = sys.stdin.readline()

 if not line:

 break

 if quit > 0:

 break

 line = line.strip()

 thread = threading.Thread(target=handle_line, args=(line,))

 thread.start()

 threads.append(thread)

 

signal.signal(signal.SIGUSR1, sig_handler)

signal.signal(signal.SIGUSR2, sig_handler)

signal.signal(signal.SIGALRM, sig_handler)

signal.signal(signal.SIGINT, sig_handler)

signal.signal(signal.SIGQUIT, sig_handler)

signal.signal(signal.SIGTERM, sig_handler)

 

stdout_thread = threading.Thread(target=handle_stdout, args=(1,))

stdout_thread.start()

 

threads.append(stdout_thread)

 

stdin_thread = threading.Thread(target=handle_stdin, args=(2,))

stdin_thread.start()

 

threads.append(stdin_thread)

 

while(RUNNING):

time.sleep(3)

 

print("Not RUNNING")

for thread in threads:

thread.join()

print("All threads stopped.")

## END

 

Eliezer

 



Eliezer Croitoru

NgTech, Tech Support

Mobile: +972-5-28704261

Email: ngtech1...@gmail.com  

 

From: squid-users   
 On Behalf Of David Touzeau
Sent: Friday, February 4, 2022 16:29
To: squid-users@lists.squid-cache.org 
 
Subject: Re: [squid-users] external helper development

 

Elizer,

Thanks for all this advice and indeed your arguments are valid between opening 
a socket, sending data, receiving data and closing the socket unlike direct 
access to a regex or a memory entry even if the calculation has already been 
done.

But what surprises me the most is that we have produced a python plugin in 
thread which I provide you a code below. 
The php code is like your mentioned example ( No thread, just a loop and output 
OK ) 

Results are after 6k requests, squid freeze and no surf can be made as with PHP 
code we can up to 10K requ

Re: [squid-users] external helper development

[David Touzeau]

You are the best,
We will launch a benchmark to see the diff

Le 07/02/2022 à 16:14, Eliezer Croitoru a écrit :


Hey David,

Since the handle_stdout runs in it’s own thread it’s sole purpose is 
to send results to stdout.


If I will run the next code in a simple software without the 0.5 sleep 
time:


 while RUNNING:

 if quit > 0:

   return

 while len(queue) > 0:

 item = queue.pop(0)

sys.stdout.write(item)

sys.stdout.flush()

 time.sleep(0.5)

what will happen is that the software will run with 100% CPU looping 
over and over on the size of the queue

while sometimes it will spit some data to stdout.

Adding a small delay with 0.5 secs will allow some “idle” time for the 
cpu in the loop preventing it from consuming

all the CPU time.

It’s a very old technique and there are others which are more 
efficient but it’s enough to demonstrate that a simple
threaded helper is much better then any PHP code that was not meant to 
be running as a STDIN/OUT daemon/helper software.


All The Bests,

Eliezer



Eliezer Croitoru

NgTech, Tech Support

Mobile: +972-5-28704261

Email: ngtech1...@gmail.com

*From:*David Touzeau 
*Sent:* Monday, February 7, 2022 02:42
*To:* Eliezer Croitoru ; 
squid-users@lists.squid-cache.org

*Subject:* Re: [squid-users] external helper development

Sorry Elizer

It was a mistake... No, your code is clean..
Impressive for the first shot
Many thanks for your example, we will run our stress tool to see the 
difference...


Just a question

Why did you send 500 milliseconds of sleep in the handle_stdoud ? Is 
it for let squid closing the pipe ?



Le 06/02/2022 à 11:46, Eliezer Croitoru a écrit :

Hey David,

Not a fully completed helper but it seems to works pretty nice and
might be better then what exist already:


https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py

#!/usr/bin/env python

import sys

import time

import urllib.request

import signal

import threading

#set debug mode for True or False

debug = False

#debug = True

queue = []

threads = []

RUNNING = True

quit = 0

rand_api_url = "https://cloud1.ngtech.co.il/api/test.php";


def sig_handler(signum, frame):

sys.stderr.write("Signal is received:" + str(signum) + "\n")

    global quit

    quit = 1

    global RUNNING

    RUNNING=False

def handle_line(line):

 if not RUNNING:

 return

 if not line:

 return

 if quit > 0:

 return

 arr = line.split()

 response = urllib.request.urlopen( rand_api_url )

 response_text = response.read()

 queue.append(arr[0] + " " + response_text.decode("utf-8"))

def handle_stdout(n):

 while RUNNING:

 if quit > 0:

   return

 while len(queue) > 0:

 item = queue.pop(0)

sys.stdout.write(item)

sys.stdout.flush()

 time.sleep(0.5)

def handle_stdin(n):

    while RUNNING:

 line = sys.stdin.readline()

 if not line:

 break

 if quit > 0:

 break

 line = line.strip()

 thread = threading.Thread(target=handle_line, args=(line,))

 thread.start()

threads.append(thread)

signal.signal(signal.SIGUSR1, sig_handler)

signal.signal(signal.SIGUSR2, sig_handler)

signal.signal(signal.SIGALRM, sig_handler)

signal.signal(signal.SIGINT, sig_handler)

signal.signal(signal.SIGQUIT, sig_handler)

signal.signal(signal.SIGTERM, sig_handler)

stdout_thread = threading.Thread(target=handle_stdout, args=(1,))

stdout_thread.start()

threads.append(stdout_thread)

stdin_thread = threading.Thread(target=handle_stdin, args=(2,))

stdin_thread.start()

threads.append(stdin_thread)

while(RUNNING):

    time.sleep(3)

print("Not RUNNING")

for thread in threads:

    thread.join()

print("All threads stopped.")

## END

Eliezer



Eliezer Croitoru

NgTech, Tech Support

Mobile: +972-5-28704261

Email: ngtech1...@gmail.com

*From:*squid-users 
 *On Behalf Of
*David Touzeau
*Sent:* Friday, February 4, 2022 16:29
*To:* squid-users@lists.squid-cache.org
*Subject:* Re: [squid-users] external helper development

Elizer,

Thanks for all this advice and indeed your arguments are valid
between opening a socket, sending data, receiving data and closing
the socket unlike direct access to a regex or a memory entry even
if the calculation has already been done.

But what surprises me the most is that we have produced a python
plugin