RE: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Don't call me Shirley Dohn N8EWY -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Simon Brown (HB9DRV) Sent: Monday, October 01, 2007 7:59 AM To: elecraft Subject: Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!) I expect he will be interested - with any luck he'll be at the RSGB HF convention, I'll ask him about this. Shirley someone can arrange for him to get his hands on a K3. Simon Brown, HB9DRV - Original Message - From: "Stewart Baker" <[EMAIL PROTECTED]> When I spoke with Peter a year or so back about a possible K2 review he didn't seem very interested. Who knows, the K3 may evoke a different response. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
-Original Message- From: Goody K3NG <[EMAIL PROTECTED]> Another thing to note is that ARRL buys all the gear they review, right off the shelf just like any other ham would. And it's bought in such a way that the seller doesn't know it's going to be an ARRL test unit. -- As for "fluff" in ARRL Product Reviews, I attribute that to the wide 'dynamic range' of the readership's technical knowledge. On the one end are hams who can recite testing methodologies for all sorts of performance metrics, including pitfalls and claimed-vs-observed numbers. At the other end are hams who don't understand why you'd want passband tuning, narrow filters, or what the attenuator/preamp switch does. And everything in between. On top of that are complaints that QST is "too technical" and/or "aimed only at the contest/DX/big station hams", etc., etc. So we get Product Reviews that are part lab test, part feelings, and part an attempt to Elmer. All jammed into a limited space, and on a limited time, to be done by a limited staff. The K2 got an expanded report that only appeared online. Went into more detail than the mag report. K3 should get the same. One ham's fluff is another ham's main interest. 73 de Jim, N2EY Email and AIM finally together. You've gotta check out free AOL Mail! - http://mail.aol.com ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: Re: [Elecraft] Sherwood on ARRL Testing Methodology
At these incredibly low signal levels a very good screened room is required. Even in the 70's I was using a copper enclosure to measure sensitivity levels within a double screened room. Extraneous radiation from the test equipment itself disturbed our measurements. We had to fit blanking plugs to unused outlets on various bits of gear. This is not for the home lab. David G3UNA > > From: Don Wilhelm <[EMAIL PROTECTED]> > Date: 2007/10/01 Mon PM 03:53:59 BST > To: elecraft@mailman.qth.net > Subject: Re: [Elecraft] Sherwood on ARRL Testing Methodology > > As measurements become more precise, it becomes more and more important > to recognize what Bill is pointing out. A slight difference in test > setup can result in a different result. These measurements are done at > the sub-microvolt level and it does not take much to create a difference. > > Test equipment must be calibrated, and the calibration tolerance should > be known. Traceable calibration is one thing, but the tolerance limits > of that calibration are also important - not all calibration labs are equal. > > Even with calibrated equipment and the same test setup, two different > equipment operators may yield two different results. As an example, > consider an instrument having a display for readout (like an > oscilloscope), the trace has a finite width, and one operator may place > the cursor on the midpoint of a trace width while another may place it > at one edge yielding two different values - how much they differ depends > on the resolution used, brightness of the trace, scale illumination, how > well the display was focused, etc. > > One good step in the right direction would be to report the region of > uncertainty for all measurements. For me, that is a piece of information > that becomes more critical as the measured values become smaller. The > ARRL lab may do that calculation in-house (I haven't asked), but they do > not state it in their published reports. > > So for now, when I see comparison data between two receiver that vary > only by a dB or so, I usually figure that is close enough to ignore the > difference (I usually do consider 3 dB or more difference to be > significant). > > 73, > Don W3FPR > > Bill Tippett wrote: > > snip... > > It's also dangerous to assume Elecraft's measurements > > will be identical to ARRL/Sherwood. There are often differences > > due to different test methodologies, people and equipment. > > > ___ > Elecraft mailing list > Post to: Elecraft@mailman.qth.net > You must be a subscriber to post to the list. > Subscriber Info (Addr. Change, sub, unsub etc.): > http://mailman.qth.net/mailman/listinfo/elecraft > > Help: http://mailman.qth.net/subscribers.htm > Elecraft web page: http://www.elecraft.com > - Email sent from www.virginmedia.com/email Virus-checked using McAfee(R) Software and scanned for spam ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
As measurements become more precise, it becomes more and more important to recognize what Bill is pointing out. A slight difference in test setup can result in a different result. These measurements are done at the sub-microvolt level and it does not take much to create a difference. Test equipment must be calibrated, and the calibration tolerance should be known. Traceable calibration is one thing, but the tolerance limits of that calibration are also important - not all calibration labs are equal. Even with calibrated equipment and the same test setup, two different equipment operators may yield two different results. As an example, consider an instrument having a display for readout (like an oscilloscope), the trace has a finite width, and one operator may place the cursor on the midpoint of a trace width while another may place it at one edge yielding two different values - how much they differ depends on the resolution used, brightness of the trace, scale illumination, how well the display was focused, etc. One good step in the right direction would be to report the region of uncertainty for all measurements. For me, that is a piece of information that becomes more critical as the measured values become smaller. The ARRL lab may do that calculation in-house (I haven't asked), but they do not state it in their published reports. So for now, when I see comparison data between two receiver that vary only by a dB or so, I usually figure that is close enough to ignore the difference (I usually do consider 3 dB or more difference to be significant). 73, Don W3FPR Bill Tippett wrote: snip... It's also dangerous to assume Elecraft's measurements will be identical to ARRL/Sherwood. There are often differences due to different test methodologies, people and equipment. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Good idea, I tried 2 years ago with the K2. Stewart G3RXQ Carefully steering clear of "Airplane" trap. On Mon, 1 Oct 2007 13:59:01 +0200, Simon Brown (HB9DRV) wrote: > I expect he will be interested - with any luck he'll be at the RSGB HF > convention, I'll ask him about this. > > Shirley someone can arrange for him to get his hands on a K3. > > Simon Brown, HB9DRV > > - Original Message - > From: "Stewart Baker" <[EMAIL PROTECTED]> > > When I spoke with Peter a year or so back about a possible K2 review he > didn't > seem very interested. Who knows, the K3 may evoke a different response. > > ___ > Elecraft mailing list > Post to: Elecraft@mailman.qth.net > You must be a subscriber to post to the list. > Subscriber Info (Addr. Change, sub, unsub etc.): > http://mailman.qth.net/mailman/listinfo/elecraft > > Help: http://mailman.qth.net/subscribers.htm > Elecraft web page: http://www.elecraft.com ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
[Elecraft] Sherwood on ARRL Testing Methodology
W5WVO: >I don't know what the test configuration of the SDR-5000 was -- I don't think it uses any discrete hardware roofing filters, does it? -- but Elecraft is already claiming that the IMDDR3 at 5 kHz spacing (400 Hz CW roofing filter) will be greater than 100 dB. (See K3 Specs page) I don't believe Eric would have let that spec be on a public web page unless he was pretty certain it would still be true when the final numbers come out. :-) The QSD design of the SDR-1000/5000 is completely different than that used by the Orion/K3, so it does not need roofing filters. Specs like IMD/BDR are determined by the resolution and linearity of the ADC in the sound card. Although Sherwood did not publish 2 kHz measurements, he did say the IMD/BDR performance is basically independent of signal spacing, so 2 kHz IMD performance is likely 96 dB also. BDR at 100 kHz was measured at 123 dB, which will also likely apply at 2 kHz. Phase noise was reported as: Phase noise (normalized) at 10 kHz spacing:123 (flat) dBc * * Phase noise does not fall off at 6 dB per octave as expected. Flex believes the present phase noise limitations are caused by A/D clock jitter. By contrast the K3's phase noise is: Rig 1kHz2 10 20 50 100 1M K3 -110 -119 -136 -140 -143 -144 -150 Bottom line is that QSD-based SDRs are very different beasts, so our assumptions based on classical designs are probably wrong. It's also dangerous to assume Elecraft's measurements will be identical to ARRL/Sherwood. There are often differences due to different test methodologies, people and equipment. For example Ten-Tec maintains Orion's 5 kHz IMD spec is "101 dB typical", but nobody else has ever measured more than about 95 dB, so I would wait for independent measurements on the K3. 73, Bill W4ZV ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
Another thing to note is that ARRL buys all the gear they review, right off the shelf just like any other ham would. So no vendor, advertiser or otherwise, has an opportunity to submit an individually tuned or "souped up" unit to get a better review. ARRL periodically auctions off review units, presumably at a loss. I think this further increases the validity of ARRL tests and just about dismisses any "sinister plot" theories. Tom Hammond wrote: Here's some comments on that article from Eric Swartz, WA6HHQ, of Elecraft. Blog: http://thek3ngreport.blogspot.com/ ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Pah! He will be there - he has a presentation on Saturday morning. Simon Brown, HB9DRV - Original Message - From: "Simon Brown (HB9DRV)" <[EMAIL PROTECTED]> I expect he will be interested - with any luck he'll be at the RSGB HF convention, I'll ask him about this. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
I expect he will be interested - with any luck he'll be at the RSGB HF convention, I'll ask him about this. Shirley someone can arrange for him to get his hands on a K3. Simon Brown, HB9DRV - Original Message - From: "Stewart Baker" <[EMAIL PROTECTED]> When I spoke with Peter a year or so back about a possible K2 review he didn't seem very interested. Who knows, the K3 may evoke a different response. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
Bill: I'm assuming you saw Rob Sherwood's recent posting to the FT-2000 reflector. It was recently re-printed to the elecraft reflector but Bill Tippett, W4ZV. Here's some comments on that article from Eric Swartz, WA6HHQ, of Elecraft. 73, Tom __ Hi Bill, I discussed the issue of the missing IMDDR3 numbers (Third Order Dynamic Range) with Michael Tracy at the ARRL a couple of weeks ago. I expressed my strong concern that they had dropped these from the review data and were only posting a footnote showing how to calculate them from the IMD level and MDS. I pointed out that we, and many others, use the IMDDR3 numbers as the primary IMD Dynamic range comparison between rigs and that the IP3 numbers were not as useful for receiver comparisons, since they can artificially be inflated by turning on the attenuator, or making a receiver have low sensitivity (deaf). They agreed that they could add the IMDDR3 numbers back in explicitly in the data table for future reviews. There is no sinister plot here. :-) They absolutely did NOT remove them at the request of any manufacturer. Quite the opposite. They were almost unintentionally removed in the post lab test formatting at the layout level for the review to save space as the review was rushed to print. After discussing this with Michael and Joel Hallas, the reviews manager, they said that they would add them back in for future reviews. Also, the multi-level IP3 numbers were in direct response to requests to the ARRL from several members of the advisory group that Rob mentions below. The primary reason for doing so, as mentioned in the sidebar of the FT-2000 review, was to show how some radios depart from the ideal IP3 curve at different levels. Personally, as I mentioned above, I feel that IP3, while useful for testing stand alone amplifier stages, is not useful for receiver comparisons. (Rob also alludes to this in his comments.) The same IMD level test is run for IMDDR3 and IP3. The IP3 number is just calculated differently from the exact same IMD data. The best comparison still is the full IMD Dynamic range, IMDDR3, which can not be as easily manipulated by artificially changing a receiver's sensitivity, as it can for IP3. I can make the K2 have a huge IP3 by reducing its front end gain by another 10 dB. Would anyone want that? No. The better test is to just take the signal generator level necessary to create an IMD product equal to the MDS, and compare that generator level to the MDS for the receiver. The net difference is the IMDDR3. Michael Tracey and the ARRL test lab are above reproach in my opinion. They take great pains to try to make accurate and useful measurements of each rig, and they are ethical to an extreme. Michael spends an incredible amount of time and care making each set of test measurements. They do not change any data for big advertisers and they buy the rigs they test on the open market. All of the recent changes were made as a result of input from the test advisory group that Rob is a member of. I'm sure there is still a lot of room for discussion and improvement of the tests they run, but the ARRL does a great service for the amateur community with their tests (as does Rob). Their ONLY goal with these latest changes is to make the data even more useful and repeatable between rigs. 73, Eric WA6HHQ - Bill Tippett wrote: Rob Sherwood gave his permission to post this on the FT-2000 list. Since that is public information I am posting it here also. BTW I agree with his comments on the "new" methodology. 73, Bill W4ZV What has gone wrong with the ARRL's new Product Reviews in QST? For several years I participated as part of a group of hams who were trying to improve the testing of radios reviewed in QST. Several reviews in the past had included questionable data on receiver performance. A sincere effort was initiated to correct testing problems within the lab, and hopefully to also improve the "hands on" portion of the report. What has changed in the ARRL reports? Some of the changes are of minor interest, like measuring the noise figure of an HF radio. Noise figure is generally used by VHF and UHF enthusiasts, but adding these data points certain hurts nothing. Is noise figure, or noise floor, or sensitivity a significant issue in today's receivers? The 75A-4 has an excellent noise floor, as reported in the January 2006 QST Annual Vintage Issue. Few of us have such a quiet location that atmospheric and galactic noise don't overshadow the noise floor of a modern receiver. What we did get was additional confusion in the ethereal world of third-order intercept (IP3), in place of real dynamic-range data. The League used to measure it one way, then a second way, and now three ways. Is this supposed to be helpful? The old way (measured at the noise floor) was acceptable. The
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
When I spoke with Peter a year or so back about a possible K2 review he didn't seem very interested. Who knows, the K3 may evoke a different response. 73 Stewart G3RXQ On Mon, 1 Oct 2007 11:55:43 +0100, Julian G4ILO wrote: > RSGB, unlike ARRL, does not buy products for review. Review products > are loaned free of charge by dealers or manufacturers (this practise > in common with all UK magazines I am aware of - and I worked for > several years as an editor and then freelance reviewer for computer > publications.) > > I think the only reviews of Elecraft products that have appeared to > date have been written by people known to the RSGB committee who > bought them for their own use. (I wrote and submitted a review of the > T1 ATU a few years ago but despite a favourable initial reaction from > the then editor it was never published.) > > If Elecraft wants a RadCom review of a K3 someone needs to arrange for > Peter Hart to receive the loan of one. > > -- > Julian, G4ILO K2 s/n: 392 K3 s/n: ??? > G4ILO's Shack: www.g4ilo.com > Ham-Directory: www.ham-directory.com > > > On 10/1/07, Stewart Baker <[EMAIL PROTECTED]> wrote: >> It will be interesting to see if the K3 gets a full spec review in RadCom. >> Probably not as there is no advertising money from Disti's involved. >> >> 73 >> Stewart G3RXQ >> >> On Mon, 1 Oct 2007 09:12:50 +0100, Julian G4ILO wrote: >>> That is exactly right. And it is not a fault only of ARRL. You never >>> read a harsh review in RSGB RadCom or Practical Wireless over here >>> either. PW reviews are a total waste of time as they are completely >>> subjective. >> >> > ___ > Elecraft mailing list > Post to: Elecraft@mailman.qth.net > You must be a subscriber to post to the list. > Subscriber Info (Addr. Change, sub, unsub etc.): > http://mailman.qth.net/mailman/listinfo/elecraft > > Help: http://mailman.qth.net/subscribers.htm > Elecraft web page: http://www.elecraft.com ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
RSGB, unlike ARRL, does not buy products for review. Review products are loaned free of charge by dealers or manufacturers (this practise in common with all UK magazines I am aware of - and I worked for several years as an editor and then freelance reviewer for computer publications.) I think the only reviews of Elecraft products that have appeared to date have been written by people known to the RSGB committee who bought them for their own use. (I wrote and submitted a review of the T1 ATU a few years ago but despite a favourable initial reaction from the then editor it was never published.) If Elecraft wants a RadCom review of a K3 someone needs to arrange for Peter Hart to receive the loan of one. -- Julian, G4ILO K2 s/n: 392 K3 s/n: ??? G4ILO's Shack: www.g4ilo.com Ham-Directory: www.ham-directory.com On 10/1/07, Stewart Baker <[EMAIL PROTECTED]> wrote: > It will be interesting to see if the K3 gets a full spec review in RadCom. > Probably not as there is no advertising money from Disti's involved. > > 73 > Stewart G3RXQ > > On Mon, 1 Oct 2007 09:12:50 +0100, Julian G4ILO wrote: > > That is exactly right. And it is not a fault only of ARRL. You never > > read a harsh review in RSGB RadCom or Practical Wireless over here > > either. PW reviews are a total waste of time as they are completely > > subjective. > > > ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
It will be interesting to see if the K3 gets a full spec review in RadCom. Probably not as there is no advertising money from Disti's involved. 73 Stewart G3RXQ On Mon, 1 Oct 2007 09:12:50 +0100, Julian G4ILO wrote: > That is exactly right. And it is not a fault only of ARRL. You never > read a harsh review in RSGB RadCom or Practical Wireless over here > either. PW reviews are a total waste of time as they are completely > subjective. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
That is exactly right. And it is not a fault only of ARRL. You never read a harsh review in RSGB RadCom or Practical Wireless over here either. PW reviews are a total waste of time as they are completely subjective. -- Julian, G4ILO K2 s/n: 392 K3 s/n: ??? G4ILO's Shack: www.g4ilo.com Ham-Directory: www.ham-directory.com On 10/1/07, Mike Fatchett W0MU <[EMAIL PROTECTED]> wrote: > > > > When will the day come when the information in QST is more than a > > fluff review, and a free multi-page advertisement for the manufacturer? > > When the magazine publishing costs are not paid for by the advertisers? ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
Bill Tippett wrote: (keyclicks and phase noise). They're getting better but IMHO they need to tone down the fluffiness in the QST articles. I agree. I find the articles themselves to be mostly non-critical rah-rah stuff that the Marketing Department at the radio company could just as well have written. BTW Sherwood measured the SDR-5000's IMDDR3 at 5 kHz as 96 dB. It's going to be a real horse race to see if the K3 may top that but if not it should be very close. I don't know what the test configuration of the SDR-5000 was -- I don't think it uses any discrete hardware roofing filters, does it? -- but Elecraft is already claiming that the IMDDR3 at 5 kHz spacing (400 Hz CW roofing filter) will be greater than 100 dB. (See K3 Specs page) I don't believe Eric would have let that spec be on a public web page unless he was pretty certain it would still be true when the final numbers come out. :-) Bill / W5WVO ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
[Elecraft] Sherwood on ARRL Testing Methodology
WA6HHQ: >I discussed the issue of the missing IMDDR3 numbers (Third Order Dynamic Range) with Michael Tracy at the ARRL a couple of weeks ago. I expressed my strong concern that they had dropped these from the review data and were only posting a footnote showing how to calculate them from the IMD level and MDS. I pointed out that we, and many others, use the IMDDR3 numbers as the primary IMD Dynamic range comparison between rigs and that the IP3 numbers were not as useful for receiver comparisons, since they can artificially be inflated by turning on the attenuator, or making a receiver have low sensitivity (deaf). I agree completely. It was very surprising they omitted the IMD numbers from the tables yet included them in the bar graphs at the very front of the article. >Also, the multi-level IP3 numbers were in direct response to requests to the ARRL from several members of the advisory group that Rob mentions below. I know W8JI who is on the group believes IP3 is simply a subject of much confusion and abuse. My guess is that one of the European VHF guys in the group lobbied for this. But I agree it's mostly a meaningless theoretical number better represented by measuring MDS and IMDDR3 separately. >Michael Tracey and the ARRL test lab are above reproach in my opinion. I agree completely but I wish more of his objectivity were shared by some of the folks writing the QST reviews. I feel some of Sherwood's criticisms are completely justified, but this is of course not Michael's responsibility. Rob also has his own axe to grind at times so you have to take some of what he says with a grain of salt. I had an Orion for 4 years and *never* heard the AGC hang artifact even though I spend 90% of my time on 160 meters where there are plenty of BIG QRN crashes. Rob can also go a bit overboard into conspiracy theories about QST advertisers. I don't buy that. It sounds like ARRL responded appropriately. I also found it humorous that W1ZR in the very same issue was describing IMDDR3 and its importance yet it was omitted in the FT-2000 tabes! :-) None of us is perfect. On the plus side, I commend ARRL for gradually moving toward more realistic testing...such as posting 2 kHz spaced measurements in the QST articles rather than simply burying the data in an obscure graph in the Expanded Test Reports. Same for TX signal purity (keyclicks and phase noise). They're getting better but IMHO they need to tone down the fluffiness in the QST articles. My personal favorite is G3SJX in RSGB's Radcom. It would almost be worth the cost of joining RSGB just to get his reviews. BTW Sherwood measured the SDR-5000's IMDDR3 at 5 kHz as 96 dB. It's going to be a real horse race to see if the K3 may top that but if not it should be very close. 73, Bill W4ZV ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology
Hi Bill, I discussed the issue of the missing IMDDR3 numbers (Third Order Dynamic Range) with Michael Tracy at the ARRL a couple of weeks ago. I expressed my strong concern that they had dropped these from the review data and were only posting a footnote showing how to calculate them from the IMD level and MDS. I pointed out that we, and many others, use the IMDDR3 numbers as the primary IMD Dynamic range comparison between rigs and that the IP3 numbers were not as useful for receiver comparisons, since they can artificially be inflated by turning on the attenuator, or making a receiver have low sensitivity (deaf). They agreed that they could add the IMDDR3 numbers back in explicitly in the data table for future reviews. There is no sinister plot here. :-) They absolutely did NOT remove them at the request of any manufacturer. Quite the opposite. They were almost unintentionally removed in the post lab test formatting at the layout level for the review to save space as the review was rushed to print. After discussing this with Michael and Joel Hallas, the reviews manager, they said that they would add them back in for future reviews. Also, the multi-level IP3 numbers were in direct response to requests to the ARRL from several members of the advisory group that Rob mentions below. The primary reason for doing so, as mentioned in the sidebar of the FT-2000 review, was to show how some radios depart from the ideal IP3 curve at different levels. Personally, as I mentioned above, I feel that IP3, while useful for testing stand alone amplifier stages, is not useful for receiver comparisons. (Rob also alludes to this in his comments.) The same IMD level test is run for IMDDR3 and IP3. The IP3 number is just calculated differently from the exact same IMD data. The best comparison still is the full IMD Dynamic range, IMDDR3, which can not be as easily manipulated by artificially changing a receiver's sensitivity, as it can for IP3. I can make the K2 have a huge IP3 by reducing its front end gain by another 10 dB. Would anyone want that? No. The better test is to just take the signal generator level necessary to create an IMD product equal to the MDS, and compare that generator level to the MDS for the receiver. The net difference is the IMDDR3. Michael Tracey and the ARRL test lab are above reproach in my opinion. They take great pains to try to make accurate and useful measurements of each rig, and they are ethical to an extreme. Michael spends an incredible amount of time and care making each set of test measurements. They do not change any data for big advertisers and they buy the rigs they test on the open market. All of the recent changes were made as a result of input from the test advisory group that Rob is a member of. I'm sure there is still a lot of room for discussion and improvement of the tests they run, but the ARRL does a great service for the amateur community with their tests (as does Rob). Their ONLY goal with these latest changes is to make the data even more useful and repeatable between rigs. 73, Eric WA6HHQ - Bill Tippett wrote: Rob Sherwood gave his permission to post this on the FT-2000 list. Since that is public information I am posting it here also. BTW I agree with his comments on the "new" methodology. 73, Bill W4ZV What has gone wrong with the ARRL's new Product Reviews in QST? For several years I participated as part of a group of hams who were trying to improve the testing of radios reviewed in QST. Several reviews in the past had included questionable data on receiver performance. A sincere effort was initiated to correct testing problems within the lab, and hopefully to also improve the "hands on" portion of the report. What has changed in the ARRL reports? Some of the changes are of minor interest, like measuring the noise figure of an HF radio. Noise figure is generally used by VHF and UHF enthusiasts, but adding these data points certain hurts nothing. Is noise figure, or noise floor, or sensitivity a significant issue in today's receivers? The 75A-4 has an excellent noise floor, as reported in the January 2006 QST Annual Vintage Issue. Few of us have such a quiet location that atmospheric and galactic noise don't overshadow the noise floor of a modern receiver. What we did get was additional confusion in the ethereal world of third-order intercept (IP3), in place of real dynamic-range data. The League used to measure it one way, then a second way, and now three ways. Is this supposed to be helpful? The old way (measured at the noise floor) was acceptable. The second way referenced an imprecise S5, now defined as -97 dBm, and a third new way at 0 dBm. Zero dBm is really strong, something we don't likely ever see, unless we are working Field Day or Multi-Multi contests from near-by transmitters. (I am assuming we are not living i
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Ouch - I would say he nailed it! I have serious issues with the AGC as handled by the Orion's DSP. What will the K3 offer? k4ia Craig "Buck" Fredericksburg, Virginia USA ** See what's new at http://www.aol.com ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
> > When will the day come when the information in QST is more than a > fluff review, and a free multi-page advertisement for the manufacturer? When the magazine publishing costs are not paid for by the advertisers? I can't wait to compare the K3 with my 2000 and 1000MP. Mike W0MU Sent using the Microsoft Entourage 2004 for Mac Test Drive. ___ Elecraft mailing list Post to: Elecraft@mailman.qth.net You must be a subscriber to post to the list. Subscriber Info (Addr. Change, sub, unsub etc.): http://mailman.qth.net/mailman/listinfo/elecraft Help: http://mailman.qth.net/subscribers.htm Elecraft web page: http://www.elecraft.com
Re: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Thats is why new is not always the best, I keep my ft 736r for 6 meters and my knwd ts 950sdx, loaded with all the options for Hf.. and I hope the K3 will out preform them all. as I expected with the FT 2000 which was a flop for me.. Keeping my fingers crossed, on the K3 , I am on the third waveBill thanks for your input..really appreciated... de Wp4o, Ed - Original Message - From: "Bill Tippett" <[EMAIL PROTECTED]> To: Sent: Sunday, September 30, 2007 7:08 PM Subject: [Elecraft] Sherwood on ARRL Testing Methodology (LONG!) Rob Sherwood gave his permission to post this on the FT-2000 list. Since that is public information I am posting it here also. BTW I agree with his comments on the "new" methodology. 73, Bill W4ZV What has gone wrong with the ARRL's new Product Reviews in QST? For several years I participated as part of a group of hams who were trying to improve the testing of radios reviewed in QST. Several reviews in the past had included questionable data on receiver performance. A sincere effort was initiated to correct testing problems within the lab, and hopefully to also improve the "hands on" portion of the report. As time went on, however, it appeared to me that the group had become fixated on minutia, and at the same time the League was unwilling to look at real problems in new radios being offered to the amateur radio operator. I have not contributed lately in the steering committee, as I felt I was banging my head against the wall. Some of the nonsense coming out of the League has been around for a long time, like the following quote from the 2004 review of the Icom IC-7800. "I was able to hear calling US stations on back scatter that I don't believe I would have heard on the '930." Did the reviewer bother to turn on his TS-930? No, he just assumed he was hearing something unusual on the receiver being reviewed (or hyped) in QST. Has anything improved in 2007? The latest October review of the FT-2000D (200 watt version of the FT-2000 that was earlier reviewed in February) states the following: "Why would I need a 200 W transceiver? After using it for a while, I was quite impressed with the extra punch the '2000D offered during routine CW and SSB contacts compared to the 100 W version." How could anyone tell a difference of 3 dB, especially compared to operation of the FT-2000 eight months ago? This kind of reporting is drivel. What has changed in the ARRL reports? Some of the changes are of minor interest, like measuring the noise figure of an HF radio. Noise figure is generally used by VHF and UHF enthusiasts, but adding these data points certain hurts nothing. Is noise figure, or noise floor, or sensitivity a significant issue in today's receivers? The 75A-4 has an excellent noise floor, as reported in the January 2006 QST Annual Vintage Issue. Few of us have such a quiet location that atmospheric and galactic noise don't overshadow the noise floor of a modern receiver. What we did get was additional confusion in the ethereal world of third-order intercept (IP3), in place of real dynamic-range data. The League used to measure it one way, then a second way, and now three ways. Is this supposed to be helpful? The old way (measured at the noise floor) was acceptable. The second way referenced an imprecise S5, now defined as -97 dBm, and a third new way at 0 dBm. Zero dBm is really strong, something we don't likely ever see, unless we are working Field Day or Multi-Multi contests from near-by transmitters. (I am assuming we are not living in Europe with their 5 megawatt AM broadcast transmitters.) 0 dBm is S9 + 73 dB, assuming any S meter reads that level accurately. (The Flex 5000A would actually do that.) On my IC-781, 0 dBm reads S9 + 50 with 30 dB of internal attenuation, or something like S9 + 80 dB with the attenuators off, if the S meter would read that high, which is does not. What happens when you put two 0 dBm signals into an IC-781 at 20 kHz spacing? The IMD reads S9 + 18 dB. At 2 kHz spacing the IMD reads S9 + 60 dB! The 781 is not a radio with performance problems, so what do these new and improved measurements really mean? If you look at the FT-2000 chart for IP3 at 2 kHz with the preamp off, you see the IP3, measured at the noise floor, is -19 dBm. This is not a good number, particularly since a Yaesu radio with "IPO" enabled (no preamp) is similar to most other radios with the 10 dB attenuator enabled. Yet if you measure the FT-2000 at 0 dBm, the IP3 calculates out to +15 dBm, which sounds good. This new information is meaningless at best, or misleading at its worst. Why is the IP3 so high at 0 dBM? Because the inter-modulation is so strong (S9 + 60 dB) the AGC has basically turned the gain of the radio off.
[Elecraft] Sherwood on ARRL Testing Methodology (LONG!)
Rob Sherwood gave his permission to post this on the FT-2000 list. Since that is public information I am posting it here also. BTW I agree with his comments on the "new" methodology. 73, Bill W4ZV What has gone wrong with the ARRL's new Product Reviews in QST? For several years I participated as part of a group of hams who were trying to improve the testing of radios reviewed in QST. Several reviews in the past had included questionable data on receiver performance. A sincere effort was initiated to correct testing problems within the lab, and hopefully to also improve the "hands on" portion of the report. As time went on, however, it appeared to me that the group had become fixated on minutia, and at the same time the League was unwilling to look at real problems in new radios being offered to the amateur radio operator. I have not contributed lately in the steering committee, as I felt I was banging my head against the wall. Some of the nonsense coming out of the League has been around for a long time, like the following quote from the 2004 review of the Icom IC-7800. "I was able to hear calling US stations on back scatter that I don't believe I would have heard on the '930." Did the reviewer bother to turn on his TS-930? No, he just assumed he was hearing something unusual on the receiver being reviewed (or hyped) in QST. Has anything improved in 2007? The latest October review of the FT-2000D (200 watt version of the FT-2000 that was earlier reviewed in February) states the following: "Why would I need a 200 W transceiver? After using it for a while, I was quite impressed with the extra punch the '2000D offered during routine CW and SSB contacts compared to the 100 W version." How could anyone tell a difference of 3 dB, especially compared to operation of the FT-2000 eight months ago? This kind of reporting is drivel. What has changed in the ARRL reports? Some of the changes are of minor interest, like measuring the noise figure of an HF radio. Noise figure is generally used by VHF and UHF enthusiasts, but adding these data points certain hurts nothing. Is noise figure, or noise floor, or sensitivity a significant issue in today's receivers? The 75A-4 has an excellent noise floor, as reported in the January 2006 QST Annual Vintage Issue. Few of us have such a quiet location that atmospheric and galactic noise don't overshadow the noise floor of a modern receiver. What we did get was additional confusion in the ethereal world of third-order intercept (IP3), in place of real dynamic-range data. The League used to measure it one way, then a second way, and now three ways. Is this supposed to be helpful? The old way (measured at the noise floor) was acceptable. The second way referenced an imprecise S5, now defined as -97 dBm, and a third new way at 0 dBm. Zero dBm is really strong, something we don't likely ever see, unless we are working Field Day or Multi-Multi contests from near-by transmitters. (I am assuming we are not living in Europe with their 5 megawatt AM broadcast transmitters.) 0 dBm is S9 + 73 dB, assuming any S meter reads that level accurately. (The Flex 5000A would actually do that.) On my IC-781, 0 dBm reads S9 + 50 with 30 dB of internal attenuation, or something like S9 + 80 dB with the attenuators off, if the S meter would read that high, which is does not. What happens when you put two 0 dBm signals into an IC-781 at 20 kHz spacing? The IMD reads S9 + 18 dB. At 2 kHz spacing the IMD reads S9 + 60 dB! The 781 is not a radio with performance problems, so what do these new and improved measurements really mean? If you look at the FT-2000 chart for IP3 at 2 kHz with the preamp off, you see the IP3, measured at the noise floor, is -19 dBm. This is not a good number, particularly since a Yaesu radio with "IPO" enabled (no preamp) is similar to most other radios with the 10 dB attenuator enabled. Yet if you measure the FT-2000 at 0 dBm, the IP3 calculates out to +15 dBm, which sounds good. This new information is meaningless at best, or misleading at its worst. Why is the IP3 so high at 0 dBM? Because the inter-modulation is so strong (S9 + 60 dB) the AGC has basically turned the gain of the radio off. Most operators will run an FT-2000 with preamp 1 enabled, since it gives a reasonable noise floor, sensitivity and AGC threshold. Yet no information is available with this typical setting for the newly touted IP3 reporting method, which at 2 kHz would be about -30 dBm for the League's sample. (The FT-2000 I measured was considerably worse.) To get a meaningful dynamic-range number, the reader now has to subtract two numbers. Why is this important data now missing, or at least obfuscated? Could it be the big advertisers in QST didn't like seeing 2 kHz dynamic-range numbers that are typically around 70 dB? Only the League could take
