Re: AD and enforced password policies
Subject: Re: AD and enforced password policies Date: Tue, Jan 03, 2012 at 02:16:38PM - Quoting Tim Franklin (t...@pelican.org): There is indeed a difference between Europe (or is it only .SE?) and USA here; no bank in Sweden lets you login without at least a client certificate and password/pin code. Most banks have a hardware token, either challenge-response or HOTP/TOTP; some use the chip in chip-and-pin cards as certificate carrier, and combine it with a reader device to manage pin code entry. Can't speak for Europe as a whole, but certainly in the UK it's not common - and I wish it was. I do have different passwords for my banking and other finance-type sites (pensions etc), both for each site and distinct from my fuzzykittens passwords (which do re-use a handful of variations on a couple of themes). A hardware token would be very nice though. If it only was one token for all. Public services usually use most of the several national ID card standards that we have so for things like doing tax returns, applying for public health insurance payments, etc, one solution works -- but all the others have one each. Identity federations are probably the way to go. Client cert worries me a bit - while it *should* be standards-based, I'm sure there's some way to implement it such that it only works on Windows. Given how long it took for banks to stop with the Safari! Evil! Access denied! routine, I don't hold much faith in their willingness or ability to build cross-platform solutions. It sometimes works. Sometimes not. I have chip-and-pin with cert on and reader. If I use it as a standalone authenticator I can even use elinks, but to use it as national ID card I need to run a bunch of apps, and must stay on Firefox3. This is for OSX. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 UH-OH!! I think KEN is OVER-DUE on his R.V. PAYMENTS and HE'S having a NERVOUS BREAKDOWN too!! Ha ha. signature.asc Description: Digital signature
Re: AD and enforced password policies
Subject: Re: AD and enforced password policies Date: Tue, Jan 03, 2012 at 10:58:35PM -0600 Quoting Jimmy Hess (mysi...@gmail.com): Manual forced immediate password expiration should be in the security admin's toolbox as a possible response to observation of questionable or potentially remotely suspicious activity on a system that user had been logged into recently. Indeed. If doubt arises, just change. Have been on the fringe of a kdc compromise. 1 students and faculty were required to show up in person and change on approved terminals. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 Wow! Look!! A stray meatball!! Let's interview it! signature.asc Description: Digital signature
incoming smtp from v6 addresses
for incoming mail that is *accepted*, i.e. not stuff like 2012-01-04 00:37:28 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 2012-01-04 00:37:28 H=(nexo.es) [118.39.80.118] F=ped...@nexo.es rejected RCPT owner-radius...@ops.ietf.org: blocked because 118.39.80.118 is in blacklist at rbl-plus.mail-abuse.org: Mail from 118.39.80.118 blocked using Trend Micro Email Reputation database. Please see http://www.mail-abuse.com/cgi-bin/lookup?118.39.80.118 2012-01-04 00:37:28 no host name found for IP address 118.39.80.118 2012-01-04 00:37:29 REJECT 118.39.80.118 too many bad recip 2012-01-04 00:37:29 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. what do other folk see? randy
Re: incoming smtp from v6 addresses
Randy Bush (randy) writes: 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. what do other folk see? What's your primary configuration ? Hub, end user system ? Care to share the methodology ? I can run some stats, but want to be sure we're comparing the same thing :) Cheers, Phil
Re: subnet prefix length 64 breaks IPv6?
On 12/28/11 07:30 , Ryan Malayter wrote: Except nowhere in there is the prefix length for the test indicated, and the exact halving of forwarding rate for IPv6 leads one to believe that there are two TCAM lookups for IPv6 (hence 64-bit prefix lookups) versus one for IPv4. A cam (assuming your router uses one) can easily be parititioned to support 144 bit words, and you can look up the whole address in one go. A router designer might well choose to fold the lookup and partion a cam table in a different fashion, to reduce memory consumption, save power etc. if they choose to split lookups (for example with the 72 most significant bits in the first lookup and the last 56 in a second) it's because they believe the tradeoff associated with two constant time lookups is acceptable. remember the cam table lookup is competing against a prefix trie lookup with a variable stride pattern done in really fast dram for mind/market share. For example, what is the forwarding rate for IPv6 when the tables are filled with /124 IPv6 routes that differ only in the last 60 bits? Even then EANTC test results you reference make no mention of the prefix length for IPv4 or IPv6, or even the number of routes in the lookup table during the testing: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd800c958a.pdf
Re: incoming smtp from v6 addresses
7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. What's your primary configuration ? Hub, end user system ? the main smtp receiver and sender for maybe 100 users and a few dozen mailing list of small to lower middle class size. Care to share the methodology ? I can run some stats, but want to be sure we're comparing the same thing :) hold your nose zgrep '=.*\[:' /var/spool/exim/log/main* | wc zgrep '=' /var/spool/exim/log/main* | wc and the ever failthful bc :) randy
Re: incoming smtp from v6 addresses
Am 04.01.2012 11:10, schrieb Randy Bush: for incoming mail that is *accepted*, i.e. not stuff like 2012-01-04 00:37:28 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 2012-01-04 00:37:28 H=(nexo.es) [118.39.80.118] F=ped...@nexo.es rejected RCPT owner-radius...@ops.ietf.org: blocked because 118.39.80.118 is in blacklist at rbl-plus.mail-abuse.org: Mail from 118.39.80.118 blocked using Trend Micro Email Reputation database. Please see http://www.mail-abuse.com/cgi-bin/lookup?118.39.80.118 2012-01-04 00:37:28 no host name found for IP address 118.39.80.118 2012-01-04 00:37:29 REJECT 118.39.80.118 too many bad recip 2012-01-04 00:37:29 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. what do other folk see? randy Received $ grep 'amavis' mail.log | grep Passed | wc -l 448 $ grep 'amavis' mail.log | grep Passed | grep IPv6 | wc -l 91 $ grep 'amavis' mail.log | grep Passed | grep IPv6 | grep -v '2001:1838::cc5d:d48a' | wc -l 18 Sent $ grep 'postfix/smtp' mail.log | grep 'status=sent' | grep -v '127.0.0.1' |wc -l 253 enceladus:/var/log# grep 'postfix/smtp' mail.log | grep 'status=sent' | egrep '\[([a-f0-9]{0,4}:)+[a-f0-9]{0,4}\]' | wc -l 19 with most of them going to mailin.v6.t-online.de[2003:2:2:10:fee::32]:25 ~40 silent users Sebastian
anycast load balancing issue
Hi, I'm in the process of deploying an anycast DNS service internally. We're on a pretty provider-like network, where we run MPLS to provide several network overlays for different services. iBGP is used to distribute routing information, and ISIS is used as IGP. In one of the VRFen we would like to place name servers using a common IP address. To get speedy network updates when outages occur we'll be using OSPF on the name servers to inject the routes into the IGP. The P/E router then redistributes the route into the right VRF. (the name server OSPF process is not aware of MPLS; it just talks to a router.) So far so good. This works. Trouble is, we find that (untweaked) cost and metric are such that all nodes are equal. The last resort (peer router ID) gets invoked and all traffic goes to one single instance. Of course, when that instance falls off the net recalculation takes place and another node steps in, but I'd like true path lengths (IGP hop count) to influence more than iBGP (route-reflector-style) selection. Any clues? Oh, all-cisco, all ASR1000 series. All links GE. ~90 routers in IGP. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 ... this must be what it's like to be a COLLEGE GRADUATE!! signature.asc Description: Digital signature
Re: incoming smtp from v6 addresses
On Jan 4, 2012, at 5:26 AM, Randy Bush wrote: 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. What's your primary configuration ? Hub, end user system ? the main smtp receiver and sender for maybe 100 users and a few dozen mailing list of small to lower middle class size. Care to share the methodology ? I can run some stats, but want to be sure we're comparing the same thing :) hold your nose zgrep '=.*\[:' /var/spool/exim/log/main* | wc zgrep '=' /var/spool/exim/log/main* | wc and the ever failthful bc :) Similar footprint, and I have something like the following on puck: puck:~$ grep IPv6: /var/log/maillog | grep stat=Sent | wc -l 9043 puck:~$ grep stat=Sent /var/log/maillog | wc -l 110343 If gmail were to host for their MX I would see a lot more mail delivered over there. - Jared -- stats -- unique list delivery [mailman@puck jared]$ /home/mailman/bin/find_member @ | grep -v 'found in' | wc -l 26442 [mailman@puck jared]$ /home/mailman/bin/find_member @gmail | grep -v 'found in' | wc -l 7098 unique addresses [mailman@puck jared]$ /home/mailman/bin/find_member @ | grep 'found in' | wc -l 16044 [mailman@puck jared]$ /home/mailman/bin/find_member @gmail | grep 'found in' | wc -l 4076
Re: incoming smtp from v6 addresses
On Wed, Jan 4, 2012 at 3:56 PM, Randy Bush ra...@psg.com wrote: zgrep '=.*\[:' /var/spool/exim/log/main* | wc zgrep '=' /var/spool/exim/log/main* | wc frodo:/home/suresh# zgrep '=.*\[:' /var/log/exim4/mainlog* | wc 16673 385620 7023087 frodo:/home/suresh# zgrep '=' /var/log/exim4/mainlog* | wc 24277 559746 10110840 -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: incoming smtp from v6 addresses
Received # grep 'amavis' mail.log | grep Passed | wc -l 1411 (1189 if only counting CLEAN, post amavisd) #grep 'amavis' mail.log | grep Passed | grep IPv6 | grep -v '::1' | wc -l 255 (253 if only counting CLEAN - so less spam in IPv6 :) Sent # grep 'postfix/smtp' mail.log | grep 'status=sent' | grep -v '127.0.0.1' | wc -l 1422 # grep 'postfix/smtp' mail.log | grep 'status=sent' | egrep '\[([a-f0-9]{0,4}:)+[a-f0-9]{0,4}\]' | wc -l 13 (filtered out a v6 IP that gets a copy of every mail) 18% incoming, .9% outgoing...
Re: anycast load balancing issue
Subject: anycast load balancing issue Date: Wed, Jan 04, 2012 at 01:02:55PM +0100 Quoting Måns Nilsson (mansa...@besserwisser.org): Trouble is, we find that (untweaked) cost and metric are such that all nodes are equal. s/all nodes/all nodes in my pathetically small test case/ Was no issue. I just was unlucky in selecting test cases. Sorry. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 Do you have exactly what I want in a plaid poindexter bar bat?? signature.asc Description: Digital signature
Re: incoming smtp from v6 addresses
In a message written on Wed, Jan 04, 2012 at 07:18:11AM -0500, Jared Mauch wrote: Similar footprint, and I have something like the following on puck: puck:~$ grep IPv6: /var/log/maillog | grep stat=Sent | wc -l 9043 puck:~$ grep stat=Sent /var/log/maillog | wc -l 110343 I have a mail system that has almost 0 technical users on it. % grep IPv6: /var/log/maillog | grep stat=Sent | wc -l 4 % grep stat=Sent /var/log/maillog | wc -l 1298 :( If gmail were to host for their MX I would see a lot more mail delivered over there. Agreed, gmail, yahoo, hotmail and AOL are probably 80% of the total mail on that box, so those four could make a huge swing, individually or collectively. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ pgpuHbwOarGf9.pgp Description: PGP signature
Re: anycast load balancing issue
On Jan 4, 2012 4:52 AM, Måns Nilsson mansa...@besserwisser.org wrote: Subject: anycast load balancing issue Date: Wed, Jan 04, 2012 at 01:02:55PM +0100 Quoting Måns Nilsson (mansa...@besserwisser.org): Trouble is, we find that (untweaked) cost and metric are such that all nodes are equal. s/all nodes/all nodes in my pathetically small test case/ Was no issue. I just was unlucky in selecting test cases. Sorry. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 Do you have exactly what I want in a plaid poindexter bar bat?? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAk8ES2oACgkQ02/pMZDM1cUXpgCfQtLkFUBsbO5Z3wDPiWV1djQB SukAnA7hBBWC83iTzjjogsxPIfI5GxmK =L5pI -END PGP SIGNATURE- I use: Anycast = server loop back Protocol to server = bgp / bfd This allows for ecmp horizontal scaling for n number of dns servers (where n is less than Max ecmp paths) You may need to turn the bgp ecmp multipath knob.
Re: incoming smtp from v6 addresses
Randy Bush wrote, on 01/04/2012 05:10 AM: 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. A consequence of whitelisting? Simon -- DTN made easy, lean, and smart -- http://postellation.viagenie.ca NAT64/DNS64 open-source-- http://ecdysis.viagenie.ca STUN/TURN server -- http://numb.viagenie.ca
Re: incoming smtp from v6 addresses
On 1/4/2012 5:10 AM, Randy Bush wrote: for incoming mail that is *accepted*, i.e. not stuff like 2012-01-04 00:37:28 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 2012-01-04 00:37:28 H=(nexo.es) [118.39.80.118] F=ped...@nexo.es rejected RCPT owner-radius...@ops.ietf.org: blocked because 118.39.80.118 is in blacklist at rbl-plus.mail-abuse.org: Mail from 118.39.80.118 blocked using Trend Micro Email Reputation database. Please see http://www.mail-abuse.com/cgi-bin/lookup?118.39.80.118 2012-01-04 00:37:28 no host name found for IP address 118.39.80.118 2012-01-04 00:37:29 REJECT 118.39.80.118 too many bad recip 2012-01-04 00:37:29 REJECT because 118.39.80.118 listed in rbl-plus.mail-abuse.org 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. For accepted mail today, 2% is v6 for outbound, 4% for v6 is inbound. I suspect the higher inbound values might be due to tech mailling lists which tend to come from IPv6 enabled hosts ? ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Re: incoming smtp from v6 addresses
On 1/4/2012 10:46 AM, Mike Tancsa wrote: I suspect the higher inbound values might be due to tech mailling lists which tend to come from IPv6 enabled hosts ? Yeah, all of my (non-internal) ipv6 mail is from such mailing lists. -Dave
Re: Does anybody out there use Authentication Header (AH)?
Tom, It seems NIST recommends ESP over AH. You can look at the following 2 emails from Manav and Sriram on the IPsecME WG: http://www.ietf.org/mail-archive/web/ipsec/current/msg07403.html http://www.ietf.org/mail-archive/web/ipsec/current/msg07407.html Jack On Mon, Jan 2, 2012 at 5:57 AM, TR Shaw ts...@oitc.com wrote: On Jan 1, 2012, at 7:12 PM, John Smith wrote: Hi, I am trying to see if there are people who use AH specially since RFC 4301 has a MAY for AH and a MUST for ESP-NULL. While operators may not care about a MAY or a MUST in an RFC, but the IETF protocols and vendors do. So all protocols that require IPsec for authentication implicitly have a MAY for AH and a MUST for ESP-NULL. Given that there is hardly a difference between the two, I am trying to understand the scenarios where people might want to use AH? OR is it that people dont care and just use what their vendors provide them? Regards, John AH provides for connectionless integrity and data origin authentication and provides protection against replay attacks. Many US Gov departments that have to follow NIST and do not understand what this means require it between internal point-to-point routers between one portion of their organization and another adding more expense for no increase in operational security. If you are following NIST or DCID-63, this is required to meet certain integrity requirements ESP provides confidentiality, data origin authentication, connectionless integrity, an anti-replay service, and limited traffic flow confidentiality. EG AH portion provides for the integrity requirement and the ESP encryption provides for the confidentiality requirement of NIST. Think of AH that it is like just signing a PGPMail and ESP as signing and encrypting a PGPMail. There are reasons for both. Tom
Trouble accessing www.nanog.org
Is anyone else having trouble accessing www.nanog.org. I can ping the site but don't get any response from HTTP requests. -- Ron Bonica vcard: www.bonica.org/ron/ronbonica.vcf
Re: Trouble accessing www.nanog.org
works for me
Re: Trouble accessing www.nanog.org
I was seeing the same problem, but it seems to be working now. On Jan 4, 2012, at 11:09 AM, Andrew D Kirch wrote: works for me
Re: Trouble accessing www.nanog.org
Works for me as well : I will check to see if there was some interruption in service and report as warranted. Betty On Wed, Jan 4, 2012 at 11:09 AM, Andrew D Kirch trel...@trelane.net wrote: works for me -- Betty Burke NewNOG/NANOG Executive Director Office (810) 214-1218 Direct (510) 492-4030
Re: Trouble accessing www.nanog.org
The brief problem in accessing www.nanog.org was due to numerous parallel downloads of a large video file by a single source IP address. We have no reason to believe it was malicious in intent, but the offender has been blocked anyway. Anyone from AS37986 around? Duane W.
Re: subnet prefix length 64 breaks IPv6?
Le 03/01/2012 23:36, Owen DeLong a écrit : On Dec 24, 2011, at 6:48 AM, Glen Kent wrote: SLAAC only works with /64 - yes - but only if it runs on Ethernet-like Interface ID's of 64bit length (RFC2464). Ok, the last 64 bits of the 128 bit address identifies an Interface ID which is uniquely derived from the 48bit MAC address (which exists only in ethernet). Not exactly. Most media have some form of link-layer addressing. For Firewire, it's native EUI-64. For Ethernet, it's EUI-48 MAC addresses. For token ring, I believe there are also EUI-48 addresses. For FDDI (Remember FDDI?) I believe it was EUI-48 addresses. ATM and Frame Relay also have EUI addresses built in to their interfaces (though I don't remember the exact format and am too lazy to look it up at the moment). SLAAC could work ok with /65 on non-Ethernet media, like a point-to-point link whose Interface ID's length be negotiated during the setup phase. If we can do this for a p2p link, then why cant the same be done for an ethernet link? I'm not so sure the statement above is actually true. I think that's right, sorry. I mean - a reread of the PPPv6 RFC tells that the Interface ID negotiated by PPP is stricly 64bit length. (although it does refer to rfc4941 which specifically acks that note that an IPv6 identifier does not necessarily have to be 64 bits in length). It's a mess :-) Alex Owen Glen Other non-64 Interface IDs could be constructed for 802.15.4 links, for example a 16bit MAC address could be converted into a 32bit Interface ID. SLAAC would thus use a /96 prefix in the RA and a 32bit IID. IP-over-USB misses an Interface ID altogether, so one is free to define its length. Alex Regards, K.
2012-Big-Data-Big-Traffic
New issues for massive data movement http://www.infineta.com/sites/default/files/pdf/IRG-2012-Big-Data-Big-Traffic-and-the-WAN.pdf Henry
IPv6 resolvers
Hi Nanog, Owen, I was wondering if many people are seeing horrendous latency on the free Hurricane Electric resolvers? Both accessing the v4 or v6 resolvers have horrendous latency. This could well be coupled to their free nature and popularity. So far when contacting Hurricane Electric they restart the resolver on their end and all is well again, but now other pfSense users in the US were noticing these latency issues as well, leading me to believe it is a larger issue. But I was wondering if a more permanent solution for these resolvers exist. 74.82.42.42 2373 msec 2001:470:20::2 2592 msec The google DNS server I'm using is doing swimmingly so far, OpenDNS seems ok too. 2001:4860:4860::884416 msec Kind regards, Seth Mos
RE: Trouble accessing www.nanog.org
From: Wessels, Duane [mailto:dwess...@verisign.com] Sent: Wednesday, January 04, 2012 1:41 PM Subject: Re: Trouble accessing www.nanog.org The brief problem in accessing www.nanog.org was due to numerous parallel downloads of a large video file by a single source IP address. We have no reason to believe it was malicious in intent, but the offender has been blocked anyway. [WEG] In the lovely CGN future, not only will you see this type of behavior (multiple pulls from the same IP) all of the time, your response to block it would have taken tens or hundreds of users out of service simultaneously. /troll Not meant to fault your response, merely to point out yet one more way that CGN is likely to break things where an assumption of 1 IP = 1 user/host/network exists. Wes George This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
Re: Trouble accessing www.nanog.org
On Wed, Jan 04, 2012 at 03:10:13PM -0500, George, Wes wrote: From: Wessels, Duane [mailto:dwess...@verisign.com] Sent: Wednesday, January 04, 2012 1:41 PM Subject: Re: Trouble accessing www.nanog.org The brief problem in accessing www.nanog.org was due to numerous parallel downloads of a large video file by a single source IP address. We have no reason to believe it was malicious in intent, but the offender has been blocked anyway. [WEG] In the lovely CGN future, not only will you see this type of behavior (multiple pulls from the same IP) all of the time, your response to block it would have taken tens or hundreds of users out of service simultaneously. /troll Not meant to fault your response, merely to point out yet one more way that CGN is likely to break things where an assumption of 1 IP = 1 user/host/network exists. Wes George Hum... thats not how I read Duanes response at all.. I thought they blocked the (excessively) large video file from download... :) /bill
Re: IPv6 resolvers
Hi! But I was wondering if a more permanent solution for these resolvers exist. 74.82.42.42 2373 msec 2001:470:20::2 2592 msec The google DNS server I'm using is doing swimmingly so far, OpenDNS seems ok too. 2001:4860:4860::8844 16 msec [root@ipv6proxy ~]# ping 74.82.42.42 PING 74.82.42.42 (74.82.42.42) 56(84) bytes of data. 64 bytes from 74.82.42.42: icmp_seq=1 ttl=61 time=0.664 ms 64 bytes from 74.82.42.42: icmp_seq=2 ttl=61 time=0.640 ms 64 bytes from 74.82.42.42: icmp_seq=3 ttl=61 time=0.551 ms 64 bytes from 74.82.42.42: icmp_seq=4 ttl=61 time=0.614 ms [root@ipv6proxy ~]# ping6 2001:470:20::2 PING 2001:470:20::2(2001:470:20::2) 56 data bytes 64 bytes from 2001:470:20::2: icmp_seq=1 ttl=61 time=0.488 ms 64 bytes from 2001:470:20::2: icmp_seq=2 ttl=61 time=0.478 ms 64 bytes from 2001:470:20::2: icmp_seq=3 ttl=61 time=0.739 ms 64 bytes from 2001:470:20::2: icmp_seq=4 ttl=61 time=0.515 ms Looks pretty normal here. Bye, Raymond.
Re: IPv6 resolvers
On Wed, Jan 4, 2012 at 3:00 PM, Seth Mos seth@dds.nl wrote: Hi Nanog, Owen, I was wondering if many people are seeing horrendous latency on the free Hurricane Electric resolvers? Both accessing the v4 or v6 resolvers have horrendous latency. This could well be coupled to their free nature and popularity. So far when contacting Hurricane Electric they restart the resolver on their end and all is well again, but now other pfSense users in the US were noticing these latency issues as well, leading me to believe it is a larger issue. err, are all pfsense people automatically configured to use he's servers? that seems sorta rude if so... But I was wondering if a more permanent solution for these resolvers exist. 74.82.42.42 2373 msec 2001:470:20::2 2592 msec The google DNS server I'm using is doing swimmingly so far, OpenDNS seems ok too. 2001:4860:4860::8844 16 msec Kind regards, Seth Mos
Re: IPv6 resolvers
On Wed, Jan 04, 2012 at 09:00:26PM +0100, Seth Mos wrote: I was wondering if many people are seeing horrendous latency on the free Hurricane Electric resolvers? Looks fine to me: (neodymium:15:27)% dig @74.82.42.42 cnn.com. A ; DiG 9.7.3 @74.82.42.42 cnn.com. A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 53277 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;cnn.com. IN A ;; ANSWER SECTION: cnn.com.299 IN A 157.166.226.26 cnn.com.299 IN A 157.166.255.19 cnn.com.299 IN A 157.166.255.18 cnn.com.299 IN A 157.166.226.25 ;; Query time: 38 msec ;; SERVER: 74.82.42.42#53(74.82.42.42) ;; WHEN: Wed Jan 4 15:27:17 2012 ;; MSG SIZE rcvd: 89 (neodymium:15:32)% dig @2001:470:20::2 cnn.com. A ; DiG 9.7.3 @2001:470:20::2 cnn.com. A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 41382 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;cnn.com. IN A ;; ANSWER SECTION: cnn.com.295 IN A 157.166.226.25 cnn.com.295 IN A 157.166.255.18 cnn.com.295 IN A 157.166.255.19 cnn.com.295 IN A 157.166.226.26 ;; Query time: 20 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:32:27 2012 ;; MSG SIZE rcvd: 89 That being said, keep in mind these are anycasted. I'm using 216.66.22.2 [tserv13.ash1.ipv6.he.net] for IPv4 and 209.51.161.14 [tserv4.nyc4.ipv6.he.net] according to the A record returned by whoami.akamai.net. I might not be hitting the same server you are. - Mark -- Mark Kamichoff p...@prolixium.com http://www.prolixium.com/ signature.asc Description: Digital signature
Re: IPv6 resolvers
Hi, Just pointing out to other responding to this thread that I was referring to the *query* response times, I said nothing about ICMP which is perfectly fine. So please stop responding with ping response times already :-) No, pfSense does not set these per default, they are in wide use because these are part of the Google DNS whitelist for V6 records. Op 4 jan 2012, om 21:33 heeft Mark Kamichoff het volgende geschreven: ;; ANSWER SECTION: cnn.com. 299 IN A 157.166.226.26 cnn.com. 299 IN A 157.166.255.19 cnn.com. 299 IN A 157.166.255.18 cnn.com. 299 IN A 157.166.226.25 And a similar mistake I see others respond too as well, this is another domain with just a IPv4 record. That was not really what I was complaining about but I was not specific enough in my email When requesting the DNS for the hostname with a Quad A the story is entirely different! Try www.pfsense.com or www.didi.nl Those will definitely hit the issue, otherwise one can always use Nanog.org like below. 74.82.42.42 2204 msec 2001:4860:4860::884417 msec 2001:470:20::2 2890 msec Best regards, Seth ;; Query time: 38 msec ;; SERVER: 74.82.42.42#53(74.82.42.42) ;; WHEN: Wed Jan 4 15:27:17 2012 ;; MSG SIZE rcvd: 89 (neodymium:15:32)% dig @2001:470:20::2 cnn.com. A ; DiG 9.7.3 @2001:470:20::2 cnn.com. A ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 41382 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;cnn.com. IN A ;; ANSWER SECTION: cnn.com. 295 IN A 157.166.226.25 cnn.com. 295 IN A 157.166.255.18 cnn.com. 295 IN A 157.166.255.19 cnn.com. 295 IN A 157.166.226.26 ;; Query time: 20 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:32:27 2012 ;; MSG SIZE rcvd: 89 That being said, keep in mind these are anycasted. I'm using 216.66.22.2 [tserv13.ash1.ipv6.he.net] for IPv4 and 209.51.161.14 [tserv4.nyc4.ipv6.he.net] according to the A record returned by whoami.akamai.net. I might not be hitting the same server you are. - Mark -- Mark Kamichoff p...@prolixium.com http://www.prolixium.com/
Re: IPv6 resolvers
On Wed, Jan 04, 2012 at 09:39:39PM +0100, Seth Mos wrote: And a similar mistake I see others respond too as well, this is another domain with just a IPv4 record. That was not really what I was complaining about but I was not specific enough in my email When requesting the DNS for the hostname with a Quad A the story is entirely different! Try www.pfsense.com or www.didi.nl Still not seeing additional latency from here: (neodymium:15:44)% dig @2001:470:20::2 www.didi.nl. ; DiG 9.7.3 @2001:470:20::2 www.didi.nl. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 33979 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.didi.nl. IN ;; ANSWER SECTION: www.didi.nl.3520IN 2001:888:2087:33::132 ;; Query time: 20 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:44:06 2012 ;; MSG SIZE rcvd: 57 And if that is already cached, let's try something that should require a fresh lookup: (neodymium:15:44)% dig @2001:470:20::2 tengigabitethernet.com. ; DiG 9.7.3 @2001:470:20::2 tengigabitethernet.com. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 41662 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;tengigabitethernet.com.IN ;; ANSWER SECTION: tengigabitethernet.com. 3600IN 2001:48c8:1:104::e ;; Query time: 84 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:44:41 2012 ;; MSG SIZE rcvd: 68 Again, not too bad.. - Mark -- Mark Kamichoff p...@prolixium.com http://www.prolixium.com/ signature.asc Description: Digital signature
Re: Trouble accessing www.nanog.org
Le mercredi 04 janvier 2012 à 20:18 +, bmann...@vacation.karoshi.com a écrit : On Wed, Jan 04, 2012 at 03:10:13PM -0500, George, Wes wrote: From: Wessels, Duane [mailto:dwess...@verisign.com] Sent: Wednesday, January 04, 2012 1:41 PM Subject: Re: Trouble accessing www.nanog.org The brief problem in accessing www.nanog.org was due to numerous parallel downloads of a large video file by a single source IP address. We have no reason to believe it was malicious in intent, but the offender has been blocked anyway. [WEG] In the lovely CGN future, not only will you see this type of behavior (multiple pulls from the same IP) all of the time, your response to block it would have taken tens or hundreds of users out of service simultaneously. /troll Not meant to fault your response, merely to point out yet one more way that CGN is likely to break things where an assumption of 1 IP = 1 user/host/network exists. Wes George Hum... thats not how I read Duanes response at all.. I thought they blocked the (excessively) large video file from download... :) Depends of how we (are supposed to) interpret ``the offender has been blocked anyway'' :) Cheers, mh /bill
Re: Looking for a Tier 1 ISP Mentor for career advice.
randal k wrote: This is a huge point. We've had a LOT of trouble finding good network engineers who have all of the previously mentioned soft attributes - anything, can't setup a syslog server, doesn't understand AD much less LDAP, etc. Imagine, an employee who can help themselves 90% of the time ... Finding the diamond that has strong niche skill, networking, with a broad just-deep-enough sysadmin background has been very, very hard. I cannot Raking up an older thread, but I have to comment on this. I understand it is hard to find the right person for the job. And even harder to find someone who has a wide range of knowledge and deep specialised knowledge to boot. When I was even more naive I always thought that in the world of IT most people knew a lot about many things, because it's not just a job but their hobby and passion (it is for me). So a sysadmin knows how to code and a coder knows how to set up a network and server etc. Yet what I noticed is that it is very rare to find such people. In fact I found people in one niche being almost ignorant of other fields. Say a coder gets confused when /tmp fills up and being unaware of this thing called a search engine and instead will virtually cry help my puter b0rked, I stuck! and vice versa. It looks to me it's just the nature of most people to be good at only one or a couple of things and be mostly ignorant about the rest. It's not going to change much, and we just have to accept that's how it is for the most part. However it can be mitigated to some extent: emphasize enough the importance of cross-training. Immensely valuable. This indeed will help a lot and is very important. Sadly though in the USA this kind of thing is not found to be important at all. Besides that, it is actually quite hard to find the right job. Or, actually, to be even acknowledged or heard by the employer of such a job. As always this thing goes both ways. Employers in the USA need to invest more in training their employees and learning should be an important and constant part of one's job and be actively encouraged. I think in this they're quite behind their Western European counterparts. Regards, Jeroen -- Earthquake Magnitude: 3.2 Date: Wednesday, January 4, 2012 17:24:31 UTC Location: Southern Alaska Latitude: 59.8964; Longitude: -153.3298 Depth: 135.00 km
RE: Looking for a Tier 1 ISP Mentor for career advice.
Say a coder gets confused when /tmp fills up and being unaware of this thing called a search engine and instead will virtually cry help my puter b0rked, I stuck! and vice versa. Hah! In my experience, this phenomenon is not unique to coders, sysadmins, or any other specialization. People prefer to look to other people for their answers. This one has bugged me for a long time, as I'm not sure what to attribute it to - is it a desire to be social, or to have the answer personalized? Is it a compliment indicative of respect of ones peer, or is it an indication of laziness? Employers in the USA need to invest more in training their employees and learning should be an important and constant part of one's job and be actively encouraged. I think in this they're quite behind their Western European counterparts. This is likely true in many larger corporations. I have found the startup and SMB sectors to be highly amenable to investing in their people. Cash-strapped businesses are most likely to consider the ROI of buying their employees skillsets (ie, training) vs hiring in new employees just to acquire those skillsets, whereas larger companies either already have a guy who knows how to do X, or doesn't really mind hiring an X specialist (or the all-too-common X consultant). Nathan
RE: Trouble accessing www.nanog.org
-Original Message- From: Michael Hallgren [mailto:m.hallg...@free.fr] Sent: Wednesday, January 04, 2012 1:11 PM To: bmann...@vacation.karoshi.com Cc: Wessels, Duane; nanog@nanog.org Subject: Re: Trouble accessing www.nanog.org Le mercredi 04 janvier 2012 à 20:18 +, bmann...@vacation.karoshi.com a écrit : On Wed, Jan 04, 2012 at 03:10:13PM -0500, George, Wes wrote: From: Wessels, Duane [mailto:dwess...@verisign.com] Sent: Wednesday, January 04, 2012 1:41 PM Subject: Re: Trouble accessing www.nanog.org The brief problem in accessing www.nanog.org was due to numerous parallel downloads of a large video file by a single source IP address. We have no reason to believe it was malicious in intent, but the offender has been blocked anyway. [WEG] In the lovely CGN future, not only will you see this type of behavior (multiple pulls from the same IP) all of the time, your response to block it would have taken tens or hundreds of users out of service simultaneously. /troll Not meant to fault your response, merely to point out yet one more way that CGN is likely to break things where an assumption of 1 IP = 1 user/host/network exists. Wes George Hum... thats not how I read Duanes response at all.. I thought they blocked the (excessively) large video file from download... :) Depends of how we (are supposed to) interpret ``the offender has been blocked anyway'' :) Cheers, mh /bill There was a single source IP with 200+ open, active http connections to a single large media file. The single IP address was blocked. The file itself is still available on the site. Mike
Re: IPv6 resolvers
On Jan 4, 2012, at 3:46 PM, Mark Kamichoff wrote: On Wed, Jan 04, 2012 at 09:39:39PM +0100, Seth Mos wrote: And a similar mistake I see others respond too as well, this is another domain with just a IPv4 record. That was not really what I was complaining about but I was not specific enough in my email When requesting the DNS for the hostname with a Quad A the story is entirely different! Try www.pfsense.com or www.didi.nl Still not seeing additional latency from here: Try random string.pfsense.org (see below) to avoid caching, since the problem in question does not rely on the name existing. I am able to reproduce it roughly every 3rd random string I try, definitely not every time. I am unable to reproduce it with other domains so far, only pfsense.org and when it does occur I see a 1500-2200ms query time: nova-dhcp-host111:~ ryan$ dig @ordns.he.net awegawregwaefg.pfsense.org ; DiG 9.6.0-APPLE-P2 @ordns.he.net awegawregwaefg.pfsense.org ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NXDOMAIN, id: 24807 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;awegawregwaefg.pfsense.org.IN A ;; AUTHORITY SECTION: pfsense.org.3600IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 2012010200 10001 1801 604801 3601 ;; Query time: 1695 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 18:34:17 2012 ;; MSG SIZE rcvd: 117 nova-dhcp-host111:~ ryan$ (neodymium:15:44)% dig @2001:470:20::2 www.didi.nl. ; DiG 9.7.3 @2001:470:20::2 www.didi.nl. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 33979 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.didi.nl. IN ;; ANSWER SECTION: www.didi.nl. 3520IN 2001:888:2087:33::132 ;; Query time: 20 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:44:06 2012 ;; MSG SIZE rcvd: 57 And if that is already cached, let's try something that should require a fresh lookup: (neodymium:15:44)% dig @2001:470:20::2 tengigabitethernet.com. ; DiG 9.7.3 @2001:470:20::2 tengigabitethernet.com. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 41662 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;tengigabitethernet.com. IN ;; ANSWER SECTION: tengigabitethernet.com. 3600IN 2001:48c8:1:104::e ;; Query time: 84 msec ;; SERVER: 2001:470:20::2#53(2001:470:20::2) ;; WHEN: Wed Jan 4 15:44:41 2012 ;; MSG SIZE rcvd: 68 Again, not too bad.. - Mark -- Mark Kamichoff p...@prolixium.com http://www.prolixium.com/
Re: IPv6 resolvers
Once upon a time, Ryan Rawdon r...@u13.net said: Try random string.pfsense.org (see below) to avoid caching, since the problem in question does not rely on the name existing. I am able to reproduce it roughly every 3rd random string I try, definitely not every time. I am unable to reproduce it with other domains so far, only pfsense.org and when it does occur I see a 1500-2200ms query time: This appears to be a problem with the authoritative servers for pfsense.org. They are dns[1-5].registrar-servers.com (which each have multiple IP addresses). If I try each IP, I get no response from 38.101.213.194 and 2+ second response time from 69.16.244.25. Both of those IPs are listed for dns1.registrar-servers.com. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: incoming smtp from v6 addresses
RB == Randy Bush ra...@psg.com writes: 7.8% is over ipv6 transport but only 2% of outgoing deliveries are over ipv6. This is incoming only, mostly mailing lists (including a few *busy* ones): :; zgrep -Ec 'client=[^[]+\[[^]]+:' /var/log/mail.info* |awk -F: '{i+=$NF} END {print i}' 33966 :; zgrep -Ec 'client=[^[]+\[[0-9]+\.' /var/log/mail.info* |awk -F: '{i+=$NF} END {print i}' 176978 so 19.19% ipv6. That is somewhat biased by the fact that debian and, IIRC, gnome lists are sent from ipv6-capable hosts and their bugs lists are among the busiest lists. For outgoing, s/client/relay/ which results in about 4.75% ipv6. -JimC -- James Cloos cl...@jhcloos.com OpenPGP: 1024D/ED7DAEA6 grep --color=yes -Ec 'client=[^[]+\[[^]]+:' /var/log/mail.info
Re: IPv6 resolvers
does pfsense need real dns hosting maybe? I hear: http://puck.nether.net/dns ... works. On Wed, Jan 4, 2012 at 6:48 PM, Chris Adams cmad...@hiwaay.net wrote: registrar-servers.com.
Re: Trouble accessing www.nanog.org
On Wed, Jan 4, 2012 at 6:10 PM, Michael K. Smith - Adhost mksm...@adhost.com wrote: There was a single source IP with 200+ open, active http connections to a single large media file. The single IP address was blocked. The file itself is still available on the site. oh! so the 200 or so users on tulip.net that were downloading nanog content were blocked, bummer :( /troll-mode=on Err, while we're talking about video files and nanog, why is the video content still served off (stored content I mean) nanog.org servers? Why not use one of the many video serving services? some of which are free even :) (that part's not a troll, a real question, even!) -chris
Re: Trouble accessing www.nanog.org
On Jan 4, 2012, at 7:36 PM, Christopher Morrow wrote: On Wed, Jan 4, 2012 at 6:10 PM, Michael K. Smith - Adhost mksm...@adhost.com wrote: There was a single source IP with 200+ open, active http connections to a single large media file. The single IP address was blocked. The file itself is still available on the site. oh! so the 200 or so users on tulip.net that were downloading nanog content were blocked, bummer :( /troll-mode=on And now if everyone would open their laptop and go to the following address… Err, while we're talking about video files and nanog, why is the video content still served off (stored content I mean) nanog.org servers? Why not use one of the many video serving services? some of which are free even :) (that part's not a troll, a real question, even!) -chris The website work hasn't yet begun, so that is certainly still on the table. If you would like to volunteer some of your time… Mike
Re: incoming smtp from v6 addresses
On Wed, Jan 4, 2012 at 5:26 AM, Randy Bush ra...@psg.com wrote: hold your nose zgrep '=.*\[:' /var/spool/exim/log/main* | wc zgrep '=' /var/spool/exim/log/main* | wc and the ever failthful bc :) err... one of 4 MX's for home email... (I'll catch the others later on) v6 inbound: $ egrep '\[2...:' /tmp/today.from |wc -l 244 v4 inbound: $ egrep -v '\[2...:' /tmp/today.from |wc -l 135591 percent v4: 135591/(244+135591) * 100 99.82 v6 outbound: $ egrep '\[2...:' /tmp/today.to |wc -l 198 v4 outbound: $ egrep -v '\[2...:' /tmp/today.to |wc -l 196 a note about the OUT numbers... I was apparently bouncing/connection-refusing to a relay over v6 :( so 2 REAL connections out, 196 failures, w00t! (this mailserver does little 'out' email apparently)
Re: Trouble accessing www.nanog.org
On Wed, Jan 4, 2012 at 10:41 PM, Michael K. Smith - Adhost mksm...@adhost.com wrote: Err, while we're talking about video files and nanog, why is the video content still served off (stored content I mean) nanog.org servers? Why not use one of the many video serving services? some of which are free even :) (that part's not a troll, a real question, even!) -chris The website work hasn't yet begun, so that is certainly still on the table. If you would like to volunteer some of your time… I'm sure we could arrange some process to ingest videos to some form of video-hosting-website... a videotubes site let's say. who should I chat with?
Re: Trouble accessing www.nanog.org
going offlist Mike On Jan 4, 2012, at 7:47 PM, Christopher Morrow wrote: On Wed, Jan 4, 2012 at 10:41 PM, Michael K. Smith - Adhost mksm...@adhost.com wrote: Err, while we're talking about video files and nanog, why is the video content still served off (stored content I mean) nanog.org servers? Why not use one of the many video serving services? some of which are free even :) (that part's not a troll, a real question, even!) -chris The website work hasn't yet begun, so that is certainly still on the table. If you would like to volunteer some of your time… I'm sure we could arrange some process to ingest videos to some form of video-hosting-website... a videotubes site let's say. who should I chat with? -- Michael K. Smith - CISSP, GSEC, GISP Chief Technical Officer - Adhost Internet LLC mksm...@adhost.com w: +1 (206) 404-9500 f: +1 (206) 404-9050 PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0x9A96777D)
RE: Looking for a Tier 1 ISP Mentor for career advice.
Nathan Eisenberg wrote: To: Jeroen van Aart jer...@mompl.net, NANOG list nanog@nanog.org Subject: RE: Looking for a Tier 1 ISP Mentor for career advice. Date: Wed, 4 Jan 2012 22:25:40 + Say a coder gets confused when /tmp fills up and being unaware of this thing called a search engine and instead will virtually cry help my puter b0rked, I stuck! and vice versa. Hah! In my experience, this phenomenon is not unique to coders, sysadmins, or any other specialization. People prefer to look to other people for their answers. This one has bugged me for a long time, as I'm not sure what to attribute it to - is it a desire to be social, or to have the answer personalized? Is it a compliment indicative of respect of ones peer, or is it an indication of laziness? This phenomona has been recognized for, well, forever. The 'reasons' are codified in 'traditional wisdom' like two heads are better than one, or the modern The solution to the most intractable problem is immediately obvious to the first unqualified observer. When ones own way of lookinng at a problem isn't working, it is necessary to find a different way of looking at the problem. The most efficient way to do that is talk to some who thinks differently than you do. Search engines are good for finding facts; 'less good' for finding abstract/concept info -- It's much harder to formulate a search query to find something to 'fill in the blanks' in an _incomplete_ conceptualization. If yu can foumulate the search for what you're missing the search probably contains the answers you're looking for. Also, the act of 'organizing ones thoughts' to explain the problem to someone who is *NOT* familiar with the background of the problem can lead to _self-recognition_ of the solution. I have phoned a collegue, many times, and/or had a collegue phone me, where the _one-sided_ conversation has gone; -- Hello? -- Hi! I've got a problem. like _this_ {launches into description}... OH!! never mind, the light just dawned! -- chuckle Glad I could help. Troubleshooting, however, _is_ a special case situation. I can pontificate on this at some length. You have been warned. grin Troubleshooting problems is an 'art', not a 'science'. Either you know how to do it, or you don't. And, like any other art, you can't teach it; you _can_ teach 'mechanics' that help people who have an 'instinctive' (for lack of a better word) grasp of the subject do it better. But the _ability_ has to be there in the first place. It's similaar to integral calculus -- you have a result, and are looking for the question. (Remember how _hard_ integration was -- until the 'AHA!' moment when, all of a sudden, it all made sense. And you were shaking your head wondering *why* you had so much trouble 'getting it'.) Troubleshooting is much the same. If you've seen that problem before, you have an idea of what -may- be causing it. And can start checking for the existing of each possible 'what' that you know about. With experience, you know _which_ what is most likely and to start there. Also, what _additional_ things to check, to narrow down the list of 'possibles'. 'Search engines' are good when you have a 'question' and are looking for looking for an 'answer' (like 'differential calculus', to use the math metaphor). But they're medium lousy, at best, at finding the 'question' that fits the 'answer'. There are some major attempts being made to build computers that _can_ reverse engineer the 'question' from an 'answer'. See 'Watson' -- the IBM research computer project that plays as a contestant on Jeopardy! The latest incarnation 'does good' a lot of the time, but when it's wrong it is *very* wrong. I don't think I've ever seen it be 'close, but incorrect'.