[squid-dev] [RFC] Wiki sections redesign

[Amos Jeffries]

I've been looking over the RoadMap pages again today and wondering about
cleaning it up (again).

One thing that is standing out is that placing wishlist designs under
Features/ was a bad idea. It makes it very hard to identify those
wishlist entries from actually available, in-use or has-been features.

With some of us (me included) one way or another keeping feature plans
semi-secret until the last moment the whole concept behind Priority
levels has fallen on its face. No way to help each other make progress
on high priority projects if all we see is rough mention of a project
then final audit patches at the end.

* I propose adding a section "Blueprints/" or "Wishlist/" to contain
what is now the Features pages for projects which are either just
describing wishlist outlines, or developer-only projects and not
actually user visible features of Squid.
 To be moved to Features/ when the page is re-written with final
end-user feature documentation.


I also re-propose an old suggestion:

* That we make the ProgrammerGuide/ (or "Developers/" ?) section
actually the how-to documents for developers instead of just an outdated
FAQ copy on how squid-2 ('ish) code works.


Amos
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] PayloadFormatter (was PackableStream)

[Alex Rousskov]

On 08/20/2015 07:13 AM, Amos Jeffries wrote:
>> This part of the email thread was discussing whether the existence of
>> admin scripts (rather than various imprecise syntax rules or
>> cachemgr.cgi code) should be the primary factor in our decision making.
>> How is the above information relevant to that discussion?

> If such admin scripts are based on anything beyond blind guesses it will
> be based on that public documentation and dicussion.

Thank you for explaining why you talked about Guide and cachemgr.cgi code.

I suspect most admin scripts are based on the output received from
Squid. You may call that "blind guesses" if you wish; the label is not
important in this case. Since there was no public documentation when
most of those scripts were hacked together, I do not think we should
expect much else, *even* if we fantasize about admins that read guide
books and study obscure source code just to parse simple (but
non-standard) output.


>>> +class PayloadFormatter
>> ...
>>> +Packable &outBuf;
>>> +};
>>
>> Since the majority of PayloadFormatter methods and callers are going to
>> assemble and format pieces of text and numbers, I think this should
>> become an ostream.

> You want to go back to PackableStream now?

 If possible, the new page/report/payload building interface should be
 ostream-enabled, not PackableStream-enabled. IIRC, that's what I
 sketched a few emails back.
>>>
>>> So how does the data get from std::ostreambuf to the client if its not
>>> going near a Packable (ie StoreEntry).
>>
>> I do not think ostream should use std::ostreambuf if that is what you
>> meant. It should use our own buffer, backed by Packable (or StoreEntry).
>> However, cache manager code would not know that and, hence, there should
>> be no linking problems associated with that knowledge.
>>
> 
> Okay. I think I undertand you there.
> 
> But the statement about cachemgr does not quite click. Formatter and all
> thus *is* cache manager (Mgr::) code. I dont see how Mgr:: code can not
> know about how its own internals work.
> 
> I assume by "cache manager code" you mean Action and the HTTP
> request/reply handling part of Mgr::.

... and Page/Formatter itself. The code that needs to know about
StoreEntry is the code that creates a specific ostream object to
configure the Page/Formatter object with. Whether that code is inside
the Mgr:: namespace is irrelevant to the linking problems you were
worried about.


>>> You ask to data-copy into the stream buffer, then data-copy a second
>>> time to the StoreEntry buffer.
>>
>> No, I do not. The stream buffer should use StoreEntry as storage [when
>> the output goes to Store].
>>
>>
>>> PackableStream is an ostream child providing StoreEntry as the buffer.
>>> So PackableStream data-copies to the StoreEntry buffer.
>>
>> Yes.
>>
>>
>>> Which would you expect provides better performance:
>>>  one or two data-copies ?
>>
>> One copy.
>>
>> (The design choices here are actually not driven by performance, but by
>> the requirement to avoid buffering of huge cache manager responses in
>> RAM. However, the same custom ostream buffer design happens to eliminate
>> the extra copy as a positive performance side effect).
> 
> 
> Then I am going back to the PackableStream patch and making a new
> iteration that just renames StoreEntryStream and fixes the syntax
> things. Patch for that in the other thread as soon as it works.

Re-introducing PackableStream sounds good to me. Please note that
Actions should not get PackableStream, StoreEntryStream, or ostream as
the new dump() argument though! They should get an object with
payload/page/report formatting methods. See my Page sketch for an example.


> This patch will then have Formatter creating one of those streams
> if/when necessary to drop values into.

That does not sound quite right:

* an ostream would be needed for virtually any Formatter/Page method
and, hence, should be created once;

* an ostream should be backed by StoreEntry or another complex
destination that Formatter/Page should not know about and, hence, it
should be created outside Formatter/Page.


> Agreed?

To avoid misunderstanding: I hesitate reviewing a future patch now,
based on a verbal description of changes, especially since "fixes the
syntax things" may mean very different things to me and you. Even the
specific red flags I noted above may not match your true intent and/or
the next patch iteration!



 Said that, ostream is the wrong primary interface for assembling
 payload/pages/reports. IMO, you should reintroduce ostream capabilities,
 but we should not be [going back to] assembling primary
 payload/pages/reports using ostream. More on that below.


>> Page provides a set of methods for high-level assembly, including:
>>
>> /// starts recording a list of values
>> virtual Page &beginList() = 0;
>> /// finishes recording a list of values
>> virtual Page &endLi

Re: [squid-dev] [PATCH] turn FtpServer::EarlyErrorKind into strongly-typed enum

[Alex Rousskov]

On 08/20/2015 06:10 AM, Kinkie wrote:

>   the attached patch turns FtpServer::EarlyErrorKind from a C enum into
> c++11 strongly-typed enum. The code did not require any change, just a
> bit of search-and-replace.

Consider adding begin/end markers (for your EnumIteration code) while
you rewriting/renaming this (or any other) enum.

Alex.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Jenkins build is back to normal : trunk-matrix » clang,d-ubuntu-trusty #309

[noc]

See 


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Build failed in Jenkins: trunk-polygraph #824

[noc]

See 

--
Started by an SCM change
Building remotely on polygraph (12.04 amd64-Ubuntu Ubuntu amd64-Ubuntu-12.04 
Ubuntu-12.04 amd64) in workspace 

$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14235 squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y
" stderr: ""
[trunk-polygraph] $ bzr update
 M  src/cf.data.pre
All changes applied successfully.
Updated to revision 14235 of branch http://bzr.squid-cache.org/bzr/squid3/trunk
[trunk-polygraph] $ bzr switch http://bzr.squid-cache.org/bzr/squid3/trunk/
Tree is up to date at revision 14235.
Switched to branch: http://bzr.squid-cache.org/bzr/squid3/trunk/
[trunk-polygraph] $ bzr revert
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14235 squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y
" stderr: ""
[trunk-polygraph] $ bzr log -v -r 
revid:squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y..revid:squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y
 --long --show-ids
Getting local revision...
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14235 squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y
" stderr: ""
RevisionState revno:14235 
revid:squ...@treenet.co.nz-20150820122833-5veyhcj4hrdxe42y
[trunk-polygraph] $ /bin/sh -xe /tmp/hudson2660139788619984972.sh
+ cd /home/jenkins/squidperf
+ python SquidBasicPerf.py --audited 
http://build.squid-cache.org/job/trunk-polygraph/355/artifact/logs/test.lx 
--jjid 824 --svnurl http://bzr.squid-cache.org/bzr/squid3/trunk/ --jobname 
trunk-polygraph
There is an error during Squid deployment:  


Traceback (most recent call last):
  File "SquidBasicPerf.py", line 82, in 
proxy.deployFromBzr(args.svnurl, squidRevision)
  File "/home/jenkins/squidperf/Side.py", line 230, in deployFromBzr
self.execute("cd /tmp/squidsource && make -j 5 && make install")
  File "/home/jenkins/squidperf/Side.py", line 158, in execute
output = instanceSSH.execute(command)
  File "/home/jenkins/squidperf/ssh.py", line 33, in execute
assert (exitCode == 0), "Error during execution command " + command + ":" + 
error
AssertionError: Error during execution command cd /tmp/squidsource && make -j 5 
&& make install:
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Description set: 
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] PayloadFormatter (was PackableStream)

[Amos Jeffries]

On 20/08/2015 9:18 a.m., Alex Rousskov wrote:
> On 08/19/2015 09:47 AM, Amos Jeffries wrote:
>> On 19/08/2015 4:50 p.m., Alex Rousskov wrote:
>>> On 08/15/2015 12:20 AM, Amos Jeffries wrote:
 For now this class is specifically and intentionally dumping out the
 (old) format for cachemgr.cgi. Other third-party tools are considered
 only so far as the cachemgr ambiguous syntax was published in various
 parts in various places. As long as the output still matches that they
 should be fine (or better off).
> 
>>> IMO, made-up after-the-fact syntax rules are irrelevant (because
>>> "nobody" knows about them, and they are imprecise). Cachemgr.cgi itself
>>> is irrelevant (because we control it). Admin scripts parsing old output
>>> are relevant. Thus, we should either keep the old output pretty much "as
>>> is" or replace it with something sufficiently better to justify the
>>> change pains for the admins. Adjusting output (while claiming
>>> compatibility with some syntax rules we made up) is a bad approach
>>> because it makes admins unhappy while not making us happy.
> 
> 
>> "The Definitive Guide" 14.2. All the way through its talking about
>> cachemgr.cgi display "columns".
>>
>> To find the detail of what a input "column" actually is refer to
>> cachemgr.cgi code itself. Specificaly the munge_other_line() function
>> for converting report text/plain to text/html. This code has not changed
>> in any significant way since the 1990's.
>>
>> * Any line excluding a \t is a free-form text line. Lets call that ...
>> comment, kv-pair, LF.
>>
>> * table is a series of \t delimited cells. Lets call that ... 'table-row'.
>>
>> * If the first cell of the first line of a table includes a ':' its a
>> header row. Lets call that ... 'table' for the whole set of rows, and we
>> will need a optional 'label' for the pre-colon bit in the first table-row.
> 
>> The Guide is also talking about how several reports including
>> 'redirectors' are *identical* in format to 'idns' ... not today, they
>> use tab-delimiting. idns got screwed up years later and now uses space
>> delimiting and looks like crap in CGI display as a result.
>>
>>
>> The kv-pairs is not part of the Guide + CGI documentation. Those parts I
>> took from your own emails and IRC chats educating me an kinkie about
>> what the report format was supposed to be.
> 
> 
> This part of the email thread was discussing whether the existence of
> admin scripts (rather than various imprecise syntax rules or
> cachemgr.cgi code) should be the primary factor in our decision making.
> How is the above information relevant to that discussion?
> 

If such admin scripts are based on anything beyond blind guesses it will
be based on that public documentation and dicussion.


> 
>> +class PayloadFormatter
> ...
>> +Packable &outBuf;
>> +};
>
> Since the majority of PayloadFormatter methods and callers are going to
> assemble and format pieces of text and numbers, I think this should
> become an ostream.
>>>
 You want to go back to PackableStream now?
>>>
>>> If possible, the new page/report/payload building interface should be
>>> ostream-enabled, not PackableStream-enabled. IIRC, that's what I
>>> sketched a few emails back.
>>
>> So how does the data get from std::ostreambuf to the client if its not
>> going near a Packable (ie StoreEntry).
> 
> I do not think ostream should use std::ostreambuf if that is what you
> meant. It should use our own buffer, backed by Packable (or StoreEntry).
> However, cache manager code would not know that and, hence, there should
> be no linking problems associated with that knowledge.
> 

Okay. I think I undertand you there.

But the statement about cachemgr does not quite click. Formatter and all
thus *is* cache manager (Mgr::) code. I dont see how Mgr:: code can not
know about how its own internals work.

I assume by "cache manager code" you mean Action and the HTTP
request/reply handling part of Mgr::.


> 
>> You ask to data-copy into the stream buffer, then data-copy a second
>> time to the StoreEntry buffer.
> 
> No, I do not. The stream buffer should use StoreEntry as storage [when
> the output goes to Store].
> 
> 
>> PackableStream is an ostream child providing StoreEntry as the buffer.
>> So PackableStream data-copies to the StoreEntry buffer.
> 
> Yes.
> 
> 
>> Which would you expect provides better performance:
>>  one or two data-copies ?
> 
> One copy.
> 
> (The design choices here are actually not driven by performance, but by
> the requirement to avoid buffering of huge cache manager responses in
> RAM. However, the same custom ostream buffer design happens to eliminate
> the extra copy as a positive performance side effect).


Then I am going back to the PackableStream patch and making a new
iteration that just renames StoreEntryStream and fixes the syntax
things. Patch for that in the other thread as soon as it works.

This patch will then have Formatter creating one 

[squid-dev] Jenkins build is back to normal : trunk-matrix » clang,d-centos-7 #309

[noc]

See 


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Jenkins build is back to normal : trunk-matrix » clang,rs-fbsd-10 #309

[noc]

See 


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Build failed in Jenkins: trunk-polygraph #823

[noc]

See 

--
Started by an SCM change
Building remotely on polygraph (12.04 amd64-Ubuntu Ubuntu amd64-Ubuntu-12.04 
Ubuntu-12.04 amd64) in workspace 

$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14234 squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0
" stderr: ""
[trunk-polygraph] $ bzr update
 M  src/esi/Libxml2Parser.h
All changes applied successfully.
Updated to revision 14234 of branch http://bzr.squid-cache.org/bzr/squid3/trunk
[trunk-polygraph] $ bzr switch http://bzr.squid-cache.org/bzr/squid3/trunk/
Tree is up to date at revision 14234.
Switched to branch: http://bzr.squid-cache.org/bzr/squid3/trunk/
[trunk-polygraph] $ bzr revert
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14234 squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0
" stderr: ""
[trunk-polygraph] $ bzr log -v -r 
revid:squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0..revid:squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0
 --long --show-ids
Getting local revision...
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14234 squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0
" stderr: ""
RevisionState revno:14234 
revid:squid...@squid-cache.org-20150820121220-ui3hmo3hkcpa5zb0
[trunk-polygraph] $ /bin/sh -xe /tmp/hudson5838556537891819703.sh
+ cd /home/jenkins/squidperf
+ python SquidBasicPerf.py --audited 
http://build.squid-cache.org/job/trunk-polygraph/355/artifact/logs/test.lx 
--jjid 823 --svnurl http://bzr.squid-cache.org/bzr/squid3/trunk/ --jobname 
trunk-polygraph
There is an error during Squid deployment:  


Traceback (most recent call last):
  File "SquidBasicPerf.py", line 82, in 
proxy.deployFromBzr(args.svnurl, squidRevision)
  File "/home/jenkins/squidperf/Side.py", line 230, in deployFromBzr
self.execute("cd /tmp/squidsource && make -j 5 && make install")
  File "/home/jenkins/squidperf/Side.py", line 158, in execute
output = instanceSSH.execute(command)
  File "/home/jenkins/squidperf/ssh.py", line 33, in execute
assert (exitCode == 0), "Error during execution command " + command + ":" + 
error
AssertionError: Error during execution command cd /tmp/squidsource && make -j 5 
&& make install:
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Description set: 
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] turn FtpServer::EarlyErrorKind into strongly-typed enum

[Kinkie]

Hi,
  the attached patch turns FtpServer::EarlyErrorKind from a C enum into
c++11 strongly-typed enum. The code did not require any change, just a bit
of search-and-replace.


-- 
Francesco
=== modified file 'src/servers/FtpServer.cc'
--- src/servers/FtpServer.cc	2015-08-19 10:18:02 +
+++ src/servers/FtpServer.cc	2015-08-20 12:04:36 +
@@ -557,43 +557,43 @@
 const char *errUri = "error:ftp-internal-early-error";
 
 switch (eek) {
-case eekHugeRequest:
+case EarlyErrorKind::HugeRequest:
 scode = 421;
 reason = "Huge request";
 errUri = "error:ftp-huge-request";
 break;
 
-case eekMissingLogin:
+case EarlyErrorKind::MissingLogin:
 scode = 530;
 reason = "Must login first";
 errUri = "error:ftp-must-login-first";
 break;
 
-case eekMissingUsername:
+case EarlyErrorKind::MissingUsername:
 scode = 501;
 reason = "Missing username";
 errUri = "error:ftp-missing-username";
 break;
 
-case eekMissingHost:
+case EarlyErrorKind::MissingHost:
 scode = 501;
 reason = "Missing host";
 errUri = "error:ftp-missing-host";
 break;
 
-case eekUnsupportedCommand:
+case EarlyErrorKind::UnsupportedCommand:
 scode = 502;
 reason = "Unknown or unsupported command";
 errUri = "error:ftp-unsupported-command";
 break;
 
-case eekInvalidUri:
+case EarlyErrorKind::InvalidUri:
 scode = 501;
 reason = "Invalid URI";
 errUri = "error:ftp-invalid-uri";
 break;
 
-case eekMalformedCommand:
+case EarlyErrorKind::MalformedCommand:
 scode = 421;
 reason = "Malformed command";
 errUri = "error:ftp-malformed-command";
@@ -661,7 +661,7 @@
 if (cmd.length() > tokenMax || params.length() > tokenMax) {
 changeState(fssError, "huge req token");
 quitAfterError(NULL);
-return earlyError(eekHugeRequest);
+return earlyError(EarlyErrorKind::HugeRequest);
 }
 
 // technically, we may skip multiple NLs below, but that is OK
@@ -670,7 +670,7 @@
 if (in.buf.length() >= Config.maxRequestHeaderSize) {
 changeState(fssError, "huge req");
 quitAfterError(NULL);
-return earlyError(eekHugeRequest);
+return earlyError(EarlyErrorKind::HugeRequest);
 } else {
 flags.readMore = true;
 debugs(33, 5, "Waiting for more, up to " <<
@@ -691,7 +691,7 @@
 if (!master->clientReadGreeting) {
 // the first command must be USER
 if (!pinning.pinned && cmd != cmdUser())
-return earlyError(eekMissingLogin);
+return earlyError(EarlyErrorKind::MissingLogin);
 }
 
 // process USER request now because it sets FTP peer host name
@@ -702,7 +702,7 @@
 }
 
 if (!Ftp::SupportedCommand(cmd))
-return earlyError(eekUnsupportedCommand);
+return earlyError(EarlyErrorKind::UnsupportedCommand);
 
 const HttpRequestMethod method =
 cmd == cmdAppe() || cmd == cmdStor() || cmd == cmdStou() ?
@@ -717,7 +717,7 @@
 debugs(33, 5, "Invalid FTP URL: " << uri);
 uri.clear();
 safe_free(newUri);
-return earlyError(eekInvalidUri);
+return earlyError(EarlyErrorKind::InvalidUri);
 }
 
 request->flags.ftpNative = true;
@@ -1342,12 +1342,12 @@
 Ftp::Server::handleUserRequest(const SBuf &, SBuf ¶ms)
 {
 if (params.isEmpty())
-return earlyError(eekMissingUsername);
+return earlyError(EarlyErrorKind::MissingUsername);
 
 // find the [end of] user name
 const SBuf::size_type eou = params.rfind('@');
 if (eou == SBuf::npos || eou + 1 >= params.length())
-return earlyError(eekMissingHost);
+return earlyError(EarlyErrorKind::MissingHost);
 
 // Determine the intended destination.
 host = params.substr(eou + 1, params.length());

=== modified file 'src/servers/FtpServer.h'
--- src/servers/FtpServer.h	2015-08-20 09:55:56 +
+++ src/servers/FtpServer.h	2015-08-20 12:04:44 +
@@ -70,15 +70,15 @@
 friend void StartListening();
 
 // errors detected before it is possible to create an HTTP request wrapper
-typedef enum {
-eekHugeRequest,
-eekMissingLogin,
-eekMissingUsername,
-eekMissingHost,
-eekUnsupportedCommand,
-eekInvalidUri,
-eekMalformedCommand
-} EarlyErrorKind;
+enum class EarlyErrorKind {
+HugeRequest,
+MissingLogin,
+MissingUsername,
+MissingHost,
+UnsupportedCommand,
+InvalidUri,
+MalformedCommand
+};
 
 /* ConnStateData API */
 virtual ClientSocketContext *parseOneRequest();

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Build failed in Jenkins: trunk-polygraph #822

[noc]

See 

--
Started by an SCM change
Building remotely on polygraph (12.04 amd64-Ubuntu Ubuntu amd64-Ubuntu-12.04 
Ubuntu-12.04 amd64) in workspace 

$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14232 kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5
" stderr: ""
[trunk-polygraph] $ bzr update
 M  src/servers/FtpServer.h
All changes applied successfully.
Updated to revision 14232 of branch http://bzr.squid-cache.org/bzr/squid3/trunk
[trunk-polygraph] $ bzr switch http://bzr.squid-cache.org/bzr/squid3/trunk/
Tree is up to date at revision 14232.
Switched to branch: http://bzr.squid-cache.org/bzr/squid3/trunk/
[trunk-polygraph] $ bzr revert
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14232 kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5
" stderr: ""
[trunk-polygraph] $ bzr log -v -r 
revid:kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5..revid:kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5
 --long --show-ids
Getting local revision...
$ bzr revision-info -d 
info result: bzr revision-info -d 
 returned 0. Command 
output: "14232 kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5
" stderr: ""
RevisionState revno:14232 
revid:kin...@squid-cache.org-20150820095556-zi4gfexuzo8nimy5
[trunk-polygraph] $ /bin/sh -xe /tmp/hudson8477938266971405734.sh
+ cd /home/jenkins/squidperf
+ python SquidBasicPerf.py --audited 
http://build.squid-cache.org/job/trunk-polygraph/355/artifact/logs/test.lx 
--jjid 822 --svnurl http://bzr.squid-cache.org/bzr/squid3/trunk/ --jobname 
trunk-polygraph
There is an error during Squid deployment:  


Traceback (most recent call last):
  File "SquidBasicPerf.py", line 82, in 
proxy.deployFromBzr(args.svnurl, squidRevision)
  File "/home/jenkins/squidperf/Side.py", line 230, in deployFromBzr
self.execute("cd /tmp/squidsource && make -j 5 && make install")
  File "/home/jenkins/squidperf/Side.py", line 158, in execute
output = instanceSSH.execute(command)
  File "/home/jenkins/squidperf/ssh.py", line 33, in execute
assert (exitCode == 0), "Error during execution command " + command + ":" + 
error
AssertionError: Error during execution command cd /tmp/squidsource && make -j 5 
&& make install:
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Description set: 
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Developing customized Cache Selection algorithm from Round Robin, Least Load

[Du, Hongfei]

Hi Alex
Thanks for the comments, really helpful, actually by the time I read this email 
in the morning, I have managed to find some things exactly using search 
methodology as you mentioned, please find my findings inline.

Best Regards,

Humphrey

/*

Hongfei Du
Staff Engineer (UK Software)
InterDigital UK, Inc.
Shoreditch Business Center
64 Great Eastern Street
London,  EC2A 3QR
T: +44 207.749.9140
hongfei...@interdigital.com
www.InterDigital.com

[cid:image5363d4.BMP@8e96b82c.42b8be99]

This e-mail is intended only for the use of the individual or entity to which 
it is addressed, and may contain information that is privileged, confidential 
and/or otherwise protected from disclosure to anyone other than its intended 
recipient. Unintended transmission shall not constitute waiver of any privilege 
or confidentiality obligation. If you received this communication in error, 
please do not review, copy or distribute it, notify me immediately by email, 
and delete the original message and any attachments. Unless expressly stated in 
this e-mail, nothing in this message or any attachment should be construed as a 
digital or electronic signature.


-Original Message-
/*From: Alex Rousskov [mailto:rouss...@measurement-factory.com]
/*Sent: Thursday, August 20, 2015 5:16 AM
/*To: squid-dev@lists.squid-cache.org
/*Cc: Du, Hongfei 
/*Subject: Re: [squid-dev] Developing customized Cache Selection algorithm from
/*Round Robin, Least Load
/*
/*On 08/19/2015 05:03 AM, Du, Hongfei wrote:
/*
/*> We are in an attempt to extend Squid Cache selection algorithm to
/*> develop a solution for controlled storage of subscriber’s content in
/*> Squid Caches, and thereafter apply access rule on these contents, a
/*> few questions to start with:
/*>
/*> ·As we probably has to rewrite new algorithm and recompile it,
/*> so does anyone know where(or which file) is the existing Round Robin
/*> or Least Load algorithm defined in source codes?
/*
/*Search for the storeDirSelectSwapDir variable in src/store_dir.cc. To learn 
more
/*about customizing cache dir selection code, run:
/*
/*  $ bzr grep Config.store_dir_select_algorithm
/*
/*or an equivalent search command.
/*
[Du, Hongfei(Humphrey)] Using search commands, I found the following points:
- grep store_dir_select_algorithm, first hit at src/SquidConfig.h
- LL and RR is declared at store_dir.cc using static STDIRSELECT
- storeDirSelectSwapDir is really the one playing the trick(i.e. returns where 
the content is to be put into ), and it uses dirn as index with 
Config.cacheSwap.n_configured(I guess this is the total number of caches 
predefined in the disk?) I attached RR algorithm definition, really short yet 
helpful to learn the logics:

static int
storeDirSelectSwapDirRoundRobin(const StoreEntry * e)
{
   // e->objectLen() is negative at this point when we are still STORE_PENDING
   ssize_t objsize = e->mem_obj->expectedReplySize();
   if (objsize != -1)
   objsize += e->mem_obj->swap_hdr_sz;

   // Increment the first candidate once per selection (not once per
   // iteration) to reduce bias when some disk(s) attract more entries.
   static int firstCandidate = 0;
   if (++firstCandidate >= Config.cacheSwap.n_configured)
   firstCandidate = 0;

   for (int i = 0; i < Config.cacheSwap.n_configured; ++i) {
   const int dirn = (firstCandidate + i) % Config.cacheSwap.n_configured;
   const SwapDir *sd = dynamic_cast(INDEXSD(dirn));

   int load = 0;
   if (!sd->canStore(*e, objsize, load))
   continue;

   if (load < 0 || load > 1000) {
   continue;
   }

   return dirn;
   }

   return -1;
}


/*
/*> ·Is there straight forward method to tell/instruct squid to
/*> store content(e.g. an URL)  from network in a predefined specific disk
/*> folder rather than using the selection algorithm itself?
/*
/*You can use cache_dir min-size and max-size options to control cache_dir
/*selection to some extent.
/*
/*You can use the "cache" directive in squid.conf to control what gets stored 
and
/*served from the cache. This directive accepts Squid ACLs, so it can be 
applied to
/*specific URLs, but it cannot allow or prohibit individual cache_dir usage.
/*
/*I am not aware of any other configurable interface at the individual URL 
level.
/*There is no configurable mapping of URLs/StoreIDs to cache_dirs (and beyond).
/*The storeDirSelectSwapDir variable mentioned above points to one of the two
/*functions that do the mapping (selected at configuration time based on
/*store_dir_select_algorithm in squid.conf).
/*
/*
/*The details of the cache_dir selection code are rather ugly, with several 
half-
/*broken interfaces augmented by weird storage module-dependent
/*implementations. Do not expect a smooth sailing.
/*
[Du, Hongfei(Humphrey)] We tested many access control functions with 
squid.conf, and managed to response correctly from squidclient URL requests, 
works perfectly via monitoring t