Compile error on 3.2.0.15 on Solaris 10 / SunStudio 12.3
Hello, - I'm trying to test squid 3.2.0.15 on Solaris 10, but compiling 3.2.0.15 with SunStudio 12.3 produces the following error at link fase: /bin/bash ../libtool --tag=CXX --mode=link /opt/solarisstudio12.3/bin/CC -xlang=c99 -errwarn=%all,no%badargtype2w,no%wbadinit,no%wbadasg -errtags -D_REENTRANT -lpthread -norunpath -m32 -erroff=wvarhidemem,nullref -features=%all -export-dynamic -dlopen force -R/usr/sfw/lib -L/usr/sfw/lib -lssl -lcrypto -lsasl -o squid AclRegs.o AuthReg.o AccessLogEntry.o AsyncEngine.o cache_cf.o ProtoPort.o CacheDigest.o cache_manager.o carp.o cbdata.o ChunkedCodingParser.o client_db.o client_side.o client_side_reply.o client_side_request.o BodyPipe.o clientStream.o CompletionDispatcher.o ConfigOption.o ConfigParser.o CpuAffinity.o CpuAffinityMap.o CpuAffinitySet.o debug.o delay_pools.o DelayId.o DelayBucket.o DelayConfig.o DelayPool.o DelaySpec.o DelayTagged.o DelayUser.o DelayVector.o NullDelayId.o ClientDelayConfig.o disk.o DiskIO/DiskIOModule.o DiskIO/ReadRequest.o DiskIO/WriteRequest.o dlink.o dns_internal.o DnsLookupDetails.o errorpage.o ETag.o event.o EventLoop.o external_acl.o ExternalACLEntry.o FadingCounter.o fd.o fde.o filemap.o forward.o fqdncache.o ftp.o gopher.o helper.o HelperChildConfig.o htcp.o http.o HttpStatusLine.o HttpHdrCc.o HttpHdrRange.o HttpHdrSc.o HttpHdrScTarget.o HttpHdrContRange.o HttpHeader.o HttpHeaderTools.o HttpBody.o HttpMsg.o HttpParser.o HttpReply.o HttpRequest.o HttpRequestMethod.o icp_v2.o icp_v3.o int.o internal.o ipc.o ipcache.o list.o main.o mem.o mem_node.o MemBuf.o MemObject.o mime.o mime_header.o multicast.o neighbors.o Packer.o Parsing.o pconn.o peer_digest.o peer_proxy_negotiate_auth.o peer_select.o peer_sourcehash.o peer_userhash.o redirect.o refresh.o RemovalPolicy.o send-announce.o MemBlob.o SquidMath.o SquidNew.o stat.o StatHist.o String.o stmem.o store.o StoreFileSystem.o store_io.o StoreIOState.o store_client.o store_digest.o store_dir.o store_key_md5.o store_log.o store_rebuild.o store_swapin.o store_swapmeta.o store_swapout.o StoreMeta.o StoreMetaMD5.o StoreMetaSTD.o StoreMetaSTDLFS.o StoreMetaUnpacker.o StoreMetaURL.o StoreMetaVary.o StoreStats.o StoreSwapLogData.o Server.o SwapDir.o MemStore.o time.o tools.o tunnel.o unlinkd.o url.o URLScheme.o urn.o wccp.o wccp2.o whois.o wordlist.o LoadableModule.o LoadableModules.o DiskIO/DiskIOModules_gen.o err_type.o err_detail_type.o globals.o hier_code.o icp_opcode.o lookup_t.o repl_modules.o swap_log_op.o auth/libacls.la ident/libident.la acl/libacls.la eui/libeui.la acl/libstate.la auth/libauth.la libAIO.a libBlocking.a libDiskDaemon.a libDiskThreads.a libIpcIo.a libMmapped.a acl/libapi.la base/libbase.la libsquid.la ip/libip.la fs/libfs.la ipc/libipc.la mgr/libmgr.la anyp/libanyp.la comm/libcomm.la eui/libeui.la icmp/libicmp.la icmp/libicmp-core.la log/liblog.la format/libformat.la DiskIO/AIO/AIODiskIOModule.o DiskIO/Blocking/BlockingDiskIOModule.o DiskIO/DiskDaemon/DiskDaemonDiskIOModule.o DiskIO/DiskThreads/DiskThreadsDiskIOModule.o DiskIO/IpcIo/IpcIoDiskIOModule.o DiskIO/Mmapped/MmappedDiskIOModule.o repl/libheap.a repl/liblru.a -lrt -lcrypt -lmd5ssl/libsslsquid.la ssl/libsslutil.la ../lib/libmisccontainers.la ../lib/libmiscencoding.la ../lib/libmiscutil.la -L/usr/sfw/lib -lssl -lcrypto -L/usr/lib -R/usr/lib -lgss -lresolv -lsocket -lnsl -L/usr/lib -R/usr/lib -m32 -xO4 -xchip=pentium -mt -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -lkrb5 -L../compat -lcompat-squid -lm -lsocket -lresolv -lnsl -lrt -L.. ../libltdl/libltdlc.la libtool: link: rm -f .libs/squid.nm .libs/squid.nmS .libs/squid.nmT libtool: link: (cd .libs /opt/solarisstudio12.3/bin/cc -xc99=all -m32 -xO4 -xchip=pentium -mt -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -c squidS.c) libtool: link: rm -f .libs/squidS.c .libs/squid.nm .libs/squid.nmS .libs/squid.nmT libtool: link: /opt/solarisstudio12.3/bin/CC -xlang=c99 -errwarn=%all,no%badargtype2w,no%wbadinit,no%wbadasg -errtags -D_REENTRANT -norunpath -m32 -erroff=wvarhidemem,nullref -features=%all .libs/squidS.o -o squid AclRegs.o AuthReg.o AccessLogEntry.o AsyncEngine.o cache_cf.o ProtoPort.o CacheDigest.o cache_manager.o carp.o cbdata.o ChunkedCodingParser.o client_db.o client_side.o client_side_reply.o client_side_request.o BodyPipe.o clientStream.o CompletionDispatcher.o ConfigOption.o ConfigParser.o CpuAffinity.o CpuAffinityMap.o CpuAffinitySet.o debug.o delay_pools.o DelayId.o DelayBucket.o DelayConfig.o DelayPool.o DelaySpec.o DelayTagged.o DelayUser.o DelayVector.o NullDelayId.o ClientDelayConfig.o disk.o DiskIO/DiskIOModule.o DiskIO/ReadRequest.o DiskIO/WriteRequest.o dlink.o dns_internal.o DnsLookupDetails.o errorpage.o ETag.o event.o EventLoop.o external_acl.o ExternalACLEntry.o FadingCounter.o fd.o fde.o filemap.o forward.o fqdncache.o ftp.o gopher.o helper.o HelperChildConfig.o htcp.o http.o HttpStatusLine.o HttpHdrCc.o HttpHdrRange.o HttpHdrSc.o HttpHdrScTarget.o HttpHdrContRange.o HttpHeader.o HttpHeaderTools.o HttpBody.o HttpMsg.o
Re: Fixing trunk build on OpenBSD
On 15/02/2012 4:53 a.m., Alex Rousskov wrote: On 02/13/2012 11:31 PM, Amos Jeffries wrote: On 23/01/2012 1:44 a.m., Amos Jeffries wrote: On 22/01/2012 11:24 p.m., Kinkie wrote: On Sun, Jan 22, 2012 at 4:01 AM, Amos Jeffriessqu...@treenet.co.nz wrote: On 22/01/2012 12:12 p.m., Henrik Nordström wrote: lör 2012-01-21 klockan 22:27 +0100 skrev Kinkie: Hi all, the patch below seems to fix the build on OpenBSD. Does it make sense? What is the error? But yes makes sense.. mgr depends on ipc I think. Yes it does. And libipc depends on libmgr right back. libmgr defines Mgr Request/Reply types with methods that use IPC, and libipc defines the Coordinator functions that create and use the MgrRequest/Reply types. I was arguing this out with Alex a while back but never got the detailed function level analysis done. Which library gets detected as broken depends on which source files have been touched recently. So I understand that there really is no simple way now to fix things? Maybe slicing off libmgr the Coordinator functions? I'm sure there is. I just did not finish the analysis that was needed to find it. Amos Okay, I have finally got my fingers into action again on that dependency loop between libipc and libmgr. Here is the symbol table scan results: libipc depends on these types (the constructors/destructors): Mgr::Request Mgr::Inquirer Mgr::Response Mgr::StoreToCommWriter::close also makes an appearance but seems to be a loose link that appears only when scanning MGR for provided symbols used by IPC. But disappears when scanning MGR for required symbols needed by IPC. libmgr depends on: Ipc::StoreMap Ipc::SendMessage Ipc::TypedMsgHdr Ipc::ImportFdIntoComm Ipc::Inquirer Ipc::Forwarder The list of these is so long its clear that this is the lower level library and libmgr should be depending on it. required by libipc: U _ZN3Mgr7RequestC1ERKN3Ipc11TypedMsgHdrE U _ZN3Mgr8InquirerC1E8RefCountINS_6ActionEERKNS_7RequestERKSt6vectorIN3Ipc11StrandCoordESaIS9_EE U _ZN3Mgr8ResponseC1Ej8RefCountINS_6ActionEE U _ZN3Mgr8ResponseC1ERKN3Ipc11TypedMsgHdrE U _ZNK3Mgr8Response4packERN3Ipc11TypedMsgHdrE So, yes Kinkie your patch was right for a first step. But scanning through Makefile.am I find a LOT of similar reversed lists like that testRock one which is the only thing failing at present. Kind of strange why they do not also fail just about everywhere. Makes me nervous about making that change alone. FWIW, the solution is probably to split libipc and/or libmgr into two libraries so that the resulting low- and high-level libraries can be properly ordered when linked. Changing the linking order of current libmgr and libipc cannot create a proper order since the libraries are mutually dependent. Because some simple objects are constructed in the wrong library? I dont think so. IMO there is no reason for libipc to be running those constructors. It could be pulling the packet data out into a buffer and passing it to a handler registered by libmgr. Same way UDP packet handling works through comm_read handlers from the upper level ICP/HTCP/DNS/WCCP components. On the sending side, the Mgr:: objects inherit from Ipc::TypedMsgHdr, so it could (does already?) provide a virtual pack() function which they implement to pack themselves into a buffer that Ipc::Send...() controls. I got very close to that with the Subscription attempt earlier. Just went way too far on the handler split. Working like UDP will also make it very easy to do some magic foo under the hood on Windows etc and actually use localhost UDP for the transport, instead of raw Unix sockets on POSIX systems. Platform-specific failures just obscure the fact that the current order is wrong and cannot be fixed without changing the libraries. I cannot object to changing the current order if it seems to help, but we should keep in mind that any perceived improvement is probably illusory. Amos
Re: Fixing trunk build on OpenBSD
On 02/15/2012 06:48 AM, Amos Jeffries wrote: Okay, I have finally got my fingers into action again on that dependency loop between libipc and libmgr. Here is the symbol table scan results: libipc depends on these types (the constructors/destructors): Mgr::Request Mgr::Inquirer Mgr::Response Mgr::StoreToCommWriter::close also makes an appearance but seems to be a loose link that appears only when scanning MGR for provided symbols used by IPC. But disappears when scanning MGR for required symbols needed by IPC. libmgr depends on: Ipc::StoreMap Ipc::SendMessage Ipc::TypedMsgHdr Ipc::ImportFdIntoComm Ipc::Inquirer Ipc::Forwarder The list of these is so long its clear that this is the lower level library and libmgr should be depending on it. required by libipc: U _ZN3Mgr7RequestC1ERKN3Ipc11TypedMsgHdrE U _ZN3Mgr8InquirerC1E8RefCountINS_6ActionEERKNS_7RequestERKSt6vectorIN3Ipc11StrandCoordESaIS9_EE U _ZN3Mgr8ResponseC1Ej8RefCountINS_6ActionEE U _ZN3Mgr8ResponseC1ERKN3Ipc11TypedMsgHdrE U _ZNK3Mgr8Response4packERN3Ipc11TypedMsgHdrE So, yes Kinkie your patch was right for a first step. But scanning through Makefile.am I find a LOT of similar reversed lists like that testRock one which is the only thing failing at present. Kind of strange why they do not also fail just about everywhere. Makes me nervous about making that change alone. FWIW, the solution is probably to split libipc and/or libmgr into two libraries so that the resulting low- and high-level libraries can be properly ordered when linked. Changing the linking order of current libmgr and libipc cannot create a proper order since the libraries are mutually dependent. Because some simple objects are constructed in the wrong library? I dont think so. because at least one of these two libraries both * provides code needed by the other library and * uses code provided by the other library You have already identified at least some of that offending code as Mgr::Request,etc constructors/destructors that are called in libipc. I suspect there is consensus that libipc should be limited to low-level IPC operations. We just have to agree on the best way to move the higher-level IPC stuff (such as type-specific parsing of IPC messages) elsewhere. IMO there is no reason for libipc to be running those constructors. It could be pulling the packet data out into a buffer and passing it to a handler registered by libmgr. Same way UDP packet handling works through comm_read handlers from the upper level ICP/HTCP/DNS/WCCP components. On the sending side, the Mgr:: objects inherit from Ipc::TypedMsgHdr, so it could (does already?) provide a virtual pack() function which they implement to pack themselves into a buffer that Ipc::Send...() controls. I hope the parsing problem can be solved by introducing a libipc message_type_id:message_object_creator registry map that will be used to create the right objects which can correctly parse received [UDS] messages without exposing libipc to those parsing details. I do not recommend discarding transport-specific information by copying the received message data into a buffer before parsing begins, especially if you want to use the same registration mechanism for handling IPC messages on Windows. A message is more than a buffer and there should be no reason to hide the remaining message information from the parsing code (which can have platform/transport-specific components). Message packing should be already handled by the pack() API. Object creation is a little more difficult because, without an object, we cannot use virtual parsing methods. HTH, Alex. I got very close to that with the Subscription attempt earlier. Just went way too far on the handler split. Working like UDP will also make it very easy to do some magic foo under the hood on Windows etc and actually use localhost UDP for the transport, instead of raw Unix sockets on POSIX systems. Platform-specific failures just obscure the fact that the current order is wrong and cannot be fixed without changing the libraries. I cannot object to changing the current order if it seems to help, but we should keep in mind that any perceived improvement is probably illusory.
Which projects are most important?
Hello, I am aware of several important problems that I can help with: 1. Removing of store_table global and moving non-shared memory cache into its own class. It would be nice to do that before we add support for shared caching of large objects (#2) so that the new code can be isolated to shared caching areas only. However, other than new bugs, this project alone will not result in changes immediately visible to users. 2. Support for shared caching of large objects in memory and Rock store. Currently, the shared memory caching limit is hard-coded to 32KB and Rock Store maximum is configurable. 3. Triage bug reports related to slow loading of Rock store at startup and fix the problem, if any. Rock store loads cache index in disker, not worker processes, so the performance impact should be minimal, but perhaps something still needs fixing or optimizing. 4. Research and possibly optimize IPC exchanges between Squid kids. I suspect busy (but not overloaded) Squids suffer from overheads related to creating new UDS sockets. If that suspicion is confirmed, we may be able to noticeably improve performance by keeping UDS sockets persistent. This project may also have a positive impact on how Squid kids locate each other during startup and kids restarts. 5. Work with Kinkie to complete the performance regression investigation and finalize performance regression testing procedures going forward. Kinkie made great progress, but several critical (and hardest to get) data points are still missing. Doing general optimizations such as StringNG or IPv6 without a good performance testing framework would be rather foolish. 6. Fix libipc/libmgr linking problems. 7. StringNG. 8. IP::Address optimization/polishing to address a known performance regression added by IPv6 support. In your opinion, what are the two or three projects I should focus on first? Please feel free to add new items if I missed something important. I will try to pick one from your prioritized list. Thank you, Alex.
Re: Which projects are most important?
Hi Alex, I'd go for the quickest wins, to remove blockers for other work. In that way, I'd work on 5. (non-coding task, time consuming) and 6, then 1. Thanks for doing this (sorry for bottom-quoting but it seems the most appropriate way to quote in this case) Kinkie On Wed, Feb 15, 2012 at 6:42 PM, Alex Rousskov rouss...@measurement-factory.com wrote: Hello, I am aware of several important problems that I can help with: 1. Removing of store_table global and moving non-shared memory cache into its own class. It would be nice to do that before we add support for shared caching of large objects (#2) so that the new code can be isolated to shared caching areas only. However, other than new bugs, this project alone will not result in changes immediately visible to users. 2. Support for shared caching of large objects in memory and Rock store. Currently, the shared memory caching limit is hard-coded to 32KB and Rock Store maximum is configurable. 3. Triage bug reports related to slow loading of Rock store at startup and fix the problem, if any. Rock store loads cache index in disker, not worker processes, so the performance impact should be minimal, but perhaps something still needs fixing or optimizing. 4. Research and possibly optimize IPC exchanges between Squid kids. I suspect busy (but not overloaded) Squids suffer from overheads related to creating new UDS sockets. If that suspicion is confirmed, we may be able to noticeably improve performance by keeping UDS sockets persistent. This project may also have a positive impact on how Squid kids locate each other during startup and kids restarts. 5. Work with Kinkie to complete the performance regression investigation and finalize performance regression testing procedures going forward. Kinkie made great progress, but several critical (and hardest to get) data points are still missing. Doing general optimizations such as StringNG or IPv6 without a good performance testing framework would be rather foolish. 6. Fix libipc/libmgr linking problems. 7. StringNG. 8. IP::Address optimization/polishing to address a known performance regression added by IPv6 support. In your opinion, what are the two or three projects I should focus on first? Please feel free to add new items if I missed something important. I will try to pick one from your prioritized list. Thank you, Alex.
Re: Which projects are most important?
Performance is a hot topic at the moment; I would love to see more time going into that through any of the performance related items you listed (or other ones you may come up with). -Rob
Re: Which projects are most important?
On 16.02.2012 06:42, Alex Rousskov wrote: Hello, I am aware of several important problems that I can help with: 1. Removing of store_table global and moving non-shared memory cache into its own class. It would be nice to do that before we add support for shared caching of large objects (#2) so that the new code can be isolated to shared caching areas only. However, other than new bugs, this project alone will not result in changes immediately visible to users. I realize its probably a big work though. So don't let it threaten or suck time away from 3.2 stability. 2. Support for shared caching of large objects in memory and Rock store. Currently, the shared memory caching limit is hard-coded to 32KB and Rock Store maximum is configurable. 3. Triage bug reports related to slow loading of Rock store at startup and fix the problem, if any. Rock store loads cache index in disker, not worker processes, so the performance impact should be minimal, but perhaps something still needs fixing or optimizing. Please take a look at the bugzilla list with rating major+. Everything, including 2.x bugs are on the chopping block now. There may be some low-hanging bugs you can add to this list. ++ You could do is followup on our earlier discussion about converting the shutdown procedure into a async call series. We ended up with a good plan IMHO, but I have not had time to implement it and you can probably implement as a async faster anyway. This will have a user visible effect on faster restart/shutdown and less crash bugs. 4. Research and possibly optimize IPC exchanges between Squid kids. I suspect busy (but not overloaded) Squids suffer from overheads related to creating new UDS sockets. If that suspicion is confirmed, we may be able to noticeably improve performance by keeping UDS sockets persistent. This project may also have a positive impact on how Squid kids locate each other during startup and kids restarts. 5. Work with Kinkie to complete the performance regression investigation and finalize performance regression testing procedures going forward. Kinkie made great progress, but several critical (and hardest to get) data points are still missing. Doing general optimizations such as StringNG or IPv6 without a good performance testing framework would be rather foolish. 6. Fix libipc/libmgr linking problems. 7. StringNG. 8. IP::Address optimization/polishing to address a known performance regression added by IPv6 support. With the 3.2 focus on release, this all kind of falls under improvement of new features. Good, but really can be done after stable 1 goes out. The new parallel-DNS slow bug need properly isolating before it can be determined whether its worth being on the blockers list (looks like a config related bug right now). In your opinion, what are the two or three projects I should focus on first? Please feel free to add new items if I missed something important. I will try to pick one from your prioritized list. Bug fixing aimed at stability top priority please. The next round of major distro LTE releases start in April. So if at all possible to be properly stable by then it would be great for Squid. This seems like a reasonably ordered by priority already. I would just place (3) above (1) as it seems to be more of a low-hanging fruit. Amos
