This question has initiated an interesting exchange of ideas. Many good suggestions have been provided.
The fact that the desensitization exists when operating duplex on either of two antennas, or when using split antennas, combined with the results of the other tests [minimal desensitization on a dummy load, measured duplexer isolation, etc.] suggests that the initial concern may be true - that the issue is related to an external mix between the repeater transmitter and the DTV signal. Issues related to IM between narrow band and wide band systems will be a growing concern as more services convert to digital formats. There is not a great deal of published guidance regarding the identification and resolution of such issues. The use of a spectrum analyzer at the receiver port of the duplexer is a good place to continue the investigation. The problem with this test is that the mix between the repeater transmitter and the DTV will produce a wide band product. Wide band digital transmissions are noise like in character and must be observed using techniques that one uses to analyze noise. [see Agilent AN150, AN150-4, AN150-7, and AN1303] You did not indicate the instrument used to make the tests or the instrument settings and configuration. The issue is that a spectrum analyzer may not be able to "see" the problem. The noise figure of the spectrum analyzer could be on the order of 30 dB or more. On the other hand, the repeater receiver's noise figure will be less than 10 dB. The noise figure of your analyzer can be easily determined. Modern analyzers have a noise marker function. If you activate this function with the analyzer's input terminated with a 50 ohm load, you will get a result based on the analyzer's internal noise. The readout will be in the form of dBm per Hz. This metric is used because it is independent of bandwidth. Regardless of the currently selected analyzer RBW, the analyzer's processor will compute the noise marker to yield the same result. The difference between this number and the thermal noise floor [kTB] of -174 dBm/Hz is the analyzer's noise figure. Based on the receiver's sensitivity [and hence its noise figure], it will have an inferred noise floor. Noise which enters the receiver through the antenna port will add to this noise floor resulting in degradation of your effective receiver sensitivity. This external noise will be comprised of site noise, sideband noise from your own transmitter and any IM between your transmitter and the DTV signal. This new noise level can be determined based on the amount of desensitization you have measured. Based on your measurements, the new noise level will still be perhaps 10 dB below the ability of the spectrum analyzer to observe due to its much higher noise figure. The ability of the spectrum analyzer to see noise can be enhanced through the use of a high gain low noise amplifier such as the Agilent 11909A. The LNA and the spectrum analyzer combine to comprise a receiving system with a much lower noise figure than the analyzer alone - lower even than your repeater receiver. This will allow you to see the noise that is causing the issue. [Unfortunately - you still have to figure out where the noise originates] Some analyzers have a built in preamp. Generally these are optimized for display flatness and have a lower gain than an external amplifier. The use of an internal amplifier will improve the analyzer's noise figure but not to the extent that an external high gain amp can. This improvement may be sufficient or not depending on the specific instrument and the noise level you are attempting to measure. The use of the LNA has tradeoffs in the form of reduced dynamic range and reduced IM performance of the analyzer. Additional selectivity may be required ahead of the LNA.
