Hi Joe,
Yup, I was just looking at what all you've done.
For Windows, it looks like, all we need to do is update the CX_Freeze
files in Makefile.jtsdk2; remove kvasd.exe kvasd.dat
For Linux, the Makefile does not include KVASD so should be no trouble
there.
I looked through / verified the Inn
Thanks, Greg. I think I've already made the changes necessary to make
JTSDK build using sfrsd2. I have *not* removed the kvasd stuff, however.
-- Joe
On 10/20/2015 4:36 PM, Greg Beam wrote:
> Hi Joe,
>
> When you all are ready, I'll look at what needs doing to update the
> JTSDK
Hi Joe,
When you all are ready, I'll look at what needs doing to update the
JTSDK builds and InnoSetup files to ommit the KVASD binary inclusions
for WSJT. Both changes should be fairly simple.
73's
Greg, KI7MT
On 10/20/2015 14:13, Joe Taylor wrote:
> Hi all,
>
> I should have mentioned tha
Hi all,
I should have mentioned that (as of r5987) the code in .../trunk builds
WSJT10 so that JT65 decoding is done in sfrsd2, and kvasd[.exe] is no
longer required. If we no problems arise, we can probably leave it this
way and do away with all non-free code.
-- Joe
On 10/20/2015 3
Hi Steve and all,
I will be traveling and mostly out of touch for the next three days, so
I want to bring you up-to-date on what I've been doing.
Directory .../trunk/rsdtest now has code for three programs that do
their Reed-Solomon decoding in sfrsd2.
rsdtest - reads s3() data from file s
Joe,
Sounds good. Regarding tuning the sfrsd2 erasure probabilities for the HF use
case, I played around with that some and I reached the conclusion that we
should tune the algorithm using statistics derived from vectors that required
soft-symbol decoding (as opposed to using the bulk of our da
Hi STeve,
I think I understand what's going on with the less-than-perfect
selection of candidate frequencies at which to attempt JT65 decoding. I
hope to spend some time on it in the next couple of days. If I don't
get it sorted out then, it may be delayed for about a week. I'll be
away bet
> - r5955/r5922 ratio of BM-only decodes is 0.87, suggesting that our latest
> version is sending only 87% as many vectors to the decoder.
I should refine this to say “… our latest version is sending only 87% as many
*decodable* vectors to the decoder … .” Almost certainly, this is because the
Hi Joe,
> Correct False
> Program Decodes Decodes Decoder
> --
> JT65-HF2329 24BM + kvasd
> WSJT-X r5912 2249 0BM + kvasd
> WSJT-X r5955 2114 0BM + sfrsd
> WSJT-X r5955 1816 0BM only
Hi Joe,
> JT65-HF2329 24BM + kvasd
> WSJT-X r5912 2249 0BM + kvasd
> WSJT-X r5955 2114 0BM + sfrsd
> WSJT-X r5955 1816 0BM only
For what it’s worth - on a batch of my HF files, r5922, using either kvasd or
sfrsd and with the ncount threshold
Steve,
Sorry, I should have answered before. See lines 73-74 in jt65a.f90. I
turned these on when the Sync setting (on main window) is negative.
Setting sync=0 (or any positive number) should suppress output when
there was no decode.
-- Joe
On 10/2/2015 11:02 PM, Steven Franke wrote
Joe - in WSJT-X, where do I turn off the printing of blank lines for no-decodes?
> On Oct 2, 2015, at 7:14 PM, Joe Taylor wrote:
>
> Hi Steve,
>
> On 10/2/2015 7:26 PM, Steven Franke wrote:
>> Very interesting Joe. Is there any reason to think that JT-65 and
>> WSJT-X use the same symbol metric
Hi Steve,
On 10/2/2015 7:26 PM, Steven Franke wrote:
> Very interesting Joe. Is there any reason to think that JT-65 and
> WSJT-X use the same symbol metrics?
Most of the signal processing in JT65-HF is Fortran code copied directly
from WSJT.
I haven't checked carefully to see what changes ther
Very interesting Joe. Is there any reason to think that JT-65 and WSJT-X use
the same symbol metrics?
Here, I am working on writing out an s3.bin file for my 20m .wav files so that
I can use rsdtest to generate the erasure probabilities.
Steve k9an
> On Oct 2, 2015, at 3:25 PM, Joe Taylor wr
Hi Steve and all,
On 10/2/2015 9:50 AM, Steven Franke wrote:
Just re-read what I sent last night - I should have said “… 8 or more
no-decodes and only 2 decodes”. It is likely that many of those
no-decodes will eventually turn into decodes when the algorithm is
properly tuned.
Thanks for the
Joe,
> On 9/30/2015 2:57 PM, Steven Franke wrote:
>> As Joe said, for EME we probably want to run very large numbers
>> of trials. If one were to start multiple calls to sfrsd2, each
>> with its own random seed, all at once, and ensure that each
>> call runs in its own thread, would this accomplis
Hi Steve,
On 9/30/2015 2:57 PM, Steven Franke wrote:
> As Joe said, for EME we probably want to run very large numbers
> of trials. If one were to start multiple calls to sfrsd2, each
> with its own random seed, all at once, and ensure that each
> call runs in its own thread, would this accomplish
As Joe said, for EME we probably want to run very large numbers of trials. If
one were to start multiple calls to sfrsd2, each with its own random seed, all
at once, and ensure that each call runs in its own thread, would this
accomplish much of what we are after?
Steve
> On Sep 30, 2015, at 12
Joe -
Just a quick note - I played a bit over lunch and can verify that things tune
right up if the erasure matrix is matched to the metric. So far, no apparent
advantage for the p1/psum form, but I haven’t tried the mr2 probabilities yet.
Will look at it some more tonight.
Steve k9an
> On Se
Mike --
I was fearful that as soon as I committed code using sfrsd that anyone
could compile, there would be reports that "it did this" or "it didn't
do that" ...
It is *far* too early for any casual on-the-air comparisons to be
useful. The new sfrsd code has simply been "dropped in" after te
If r5949 is indeed using sfrsd (is it just calling it kvasd to make the
code happy?)
Saw the 1st instance of different decodes at 17347running WSJT-X 1.6.0
r5881 along side.
Got two decodes on r5949 at -15 and -20 that 1.6.0 did not see.
Follow at 1738 by a -1 decode on 1.6.0 that r5949 did not
Hi Bill,
Thanks for the information on Apple compilers.
Yes, there are several possible ways to slice the JT65 decoding task
that could benefit from parallel processing. Divide-by-frequency is one
possibility.
A disadvantage is that it would provide little or no help in most EME
situations,
Hi Joe,
Yeah, I've been working S.Pac / IO ( or trying too ) on 20m in the (AM)
my time. Been after VR2, FR5 but every (W6) in CA is calling me, I I
really don't understand it as I can work W6 almost 24x7 on 20m, but when
FR5, VR2, YB8, 3B9 are on and coming in well, I really don't want to
wor
Hi Steve,
I'm scratching my head...
At SVN revision r5949, WSJT-X v1.6.1 uses sfrsd2 in place of kvasd. It
seems to work well, but so far the only tuning I've done is to try both
the original (kvasd-inspired) metrics and the simple ones based on
p1/psum and p2/psum. In this instance the simp
Hi Greg,
Amusing tidbit.
I just finished building WSJT-X v1.6.1 r5949 on my in-shack computer.
The attached screen shot shows its first decodes of on-the-air JT65
signals, including KI7MT.
-- Joe
--
_
On 30/09/2015 16:57, Joe Taylor wrote:
> Hi Bill,
Hi Joe,
>
> A week or so ago you wrote "On the concurrency side the option to use
> OpenMP is not available as there is no C/C++ OpenMP available on Mac".
>
> Aren't we using OpenMP now in jt9[.exe], on all platforms? Are you
> saying that on the M
I'm not a Mac guy...but this claims a solution...
http://stackoverflow.com/questions/29057437/compile-openmp-programs-with-gcc-compiler-on-os-x-yosemite
On Wed, Sep 30, 2015 at 10:57 AM, Joe Taylor wrote:
> Hi Bill,
>
> A week or so ago you wrote "On the concurrency side the option to use
> Ope
Hi Bill,
A week or so ago you wrote "On the concurrency side the option to use
OpenMP is not available as there is no C/C++ OpenMP available on Mac".
Aren't we using OpenMP now in jt9[.exe], on all platforms? Are you
saying that on the Mac we can use OpenMP from Fortran, but not from C/C++ ??
Hi Claude,
I have added the file int.h to the rsdtest directory.
Please note that successfully executing this Makefile is unlikely to
produce anything that you will find useful.
-- Joe, K1JT
On 9/30/2015 10:59 AM, Claude Frantz wrote:
> On 09/30/2015 01:14 PM, Steven Franke wrote:
>
>
On 09/30/2015 01:14 PM, Steven Franke wrote:
Hi Steven,
> Hi Claude, I have done all of my recent development work on sfrsd2.c
> on a linux virtual machine, and the Makefile in rsdtest works for
> me. Steve k9an
At my site, it ends in the following manner:
gcc -I. -DWIN32 -DWin32 -DBIGSYM -DHAV
Hi all,
For the record: I have built a version of WSJT that calls sfrsd (with
ntrials=1) rather than kvasd. Results in line 20, below, show that
for the test data the new decoder is slightly better than kvasd (line 10).
# Test Decodes False Time
Hi Steve,
On 9/30/2015 8:52 AM, Steven Franke wrote:
>> A surprise discovery: it seems that all of our recent tests have been
>> using the original code in .../trunk/demod64a.f90, with its exp(x)
>> symbol metrics. I tried switching back to your simple p1/psum, p2/psum
>> mnetrics computed in ...
> A surprise discovery: it seems that all of our recent tests have been
> using the original code in .../trunk/demod64a.f90, with its exp(x)
> symbol metrics. I tried switching back to your simple p1/psum, p2/psum
> mnetrics computed in .../trunk/rsdtest/demod64b.f90, with much degraded
> resu
Hi Steve,
Agreed on all counts.
A surprise discovery: it seems that all of our recent tests have been
using the original code in .../trunk/demod64a.f90, with its exp(x)
symbol metrics. I tried switching back to your simple p1/psum, p2/psum
mnetrics computed in .../trunk/rsdtest/demod64b.f90,
Thanks Joe. Interesting. It could be, I suppose, that with large ntrials the
mr2syms are helping us find more decodes, but they are rejected by the soft
distance check.
In any case, I agree that we should just stick with the current erasures-only
scheme. but let's keep sfrsd3 on the side and s
Hi Claude,
I have done all of my recent development work on sfrsd2.c on a linux virtual
machine, and the Makefile in rsdtest works for me.
Steve k9an
> On Sep 30, 2015, at 1:43 AM, Claude Frantz
> wrote:
>
> Hi,
>
> Please modify the Makefiles in the rsdtest directory, so that they can
> be
Hi Steve
I played with your mr2 insertion code a bit. I agree that the
second-best symbols don't seem to help us much.
I posted two new plots of decodes vs. ntrials here:
http://physics.princeton.edu/pulsar/K1JT/decodes_vs_ntrials.pdf
http://physics.princeton.edu/pulsar/K1JT/decodes_vs_ntrials
Hi,
Please modify the Makefiles in the rsdtest directory, so that they can
be used in the Linux environment too. Thank you.
Best 88 de Claude
--
___
wsjt-devel mailing list
w
Joe -
If you get a chance, it might be interesting to plot decodes vs ntrials for the
sfrsd3.c that I committed just now. Bottom line is that I eventually found a
setting that does a significant number of mr2sym insertions and “does no harm”
at ntrials=1, i.e. the number of decodes is not
Mike --
On 9/29/2015 5:29 PM, Michael Black wrote:
> So the failures are a separate section?
No. Have you looked at what the decoder is doing?
If 25 or fewer of 63 received symbols are in error, the deterministic
Berlekamp_Massey (BM) algorithm is guaranteed to succeed. With more
than 25 err
So the failures are a separate section? So a quad core should get
something approaching 3X barring cache contention.
Question would be how often in real life do failures occur?
And would you make the # of trials dependent on # of cores?
On Tue, Sep 29, 2015 at 3:50 PM, Joe Taylor wrote:
> On 9/
On 9/29/2015 4:28 PM, Michael Black wrote:
> What kind of time is it taking now? The overhead of splitting the work can
> kill any advantage as I'm sure you know from the previous multi-threading
> of JT9.
Increasing ntrials from 10^4 to 10^5 increased running time by x 6.4:
about 22 minutes for
What kind of time is it taking now? The overhead of splitting the work can
kill any advantage as I'm sure you know from the previous multi-threading
of JT9.
Mike W9MDB
On Tue, Sep 29, 2015 at 2:16 PM, Joe Taylor wrote:
> Hi all,
>
> Some further information on the new Reed Solomon decoder.
>
>
Hi all,
Some further information on the new Reed Solomon decoder.
A simple timing measurement shows that for test program "rsdtest" 97% of
the execution time is spent in the Berlekamp-Massey
errors-and-erasures decoder.
This is good. It means that if/when we decide more speed is desirable,
it
Joe,
Interesting results! I agree with your interpretation of the differences
between the histograms for the “all attempts” and “successful attempts” cases,
and I also agree that your results seem to point to a needed adjustment in the
erasure probabilities. I guess that this process of iterati
Hi Steve and all,
A few more thoughts about my idea on how we might select erasure vectors
most likely to lead to successful decodes. This is potentially
important because the "right" erasure vector leads to a nearly
instantaneous decode; nearly all of the processing time in sfrsd2 is now
dev
Excellent news!
This will make both programs MUCH easier to get into distributions such as
Fedora and Debian without the need for hokey workarounds.
Thanks,
Richard
KF5OIM
--
__
GM all,
We have made excellent progress on the quest Steve started toward an
open-source Reed Solomon decoder that's as good or better than the
closed-source Koetter-Vardy algorithm. This message aims to be a
summary of results up to now (for our own future reference), followed by
some though
Joe,
> On Sep 26, 2015, at 8:54 PM, Joe Taylor wrote:
>
> Steve --
>
> On 9/26/2015 6:31 PM, Steven Franke wrote:
>> Joe -
>> A correction - it turns out I had ntrials set to 2:
>>
>> current sfrsd ntrials 2: 709/1000
>> current sfrsd ntrials 1: 665/1000
>>
>> Still, a worthwhile
Steve --
On 9/26/2015 6:31 PM, Steven Franke wrote:
> Joe -
> A correction - it turns out I had ntrials set to 2:
>
> current sfrsd ntrials 2: 709/1000
> current sfrsd ntrials 1: 665/1000
>
> Still, a worthwhile improvement.
Agreed. How did you define the erasure thresholds to get th
Joe -
A correction - it turns out I had ntrials set to 2:
current sfrsd ntrials 2: 709/1000
current sfrsd ntrials 1: 665/1000
Still, a worthwhile improvement.
Steve
> On Sep 26, 2015, at 5:18 PM, Joe Taylor wrote:
>
> Hi Steve,
>
> That's great! I've got almost the same results
Hi Steve,
That's great! I've got almost the same results here, obtained in a
slightly different way. I carved the p1-rank, p2/p1 plane into 8 equal
ranges for each coordinate, and accumulated 8x8 histograms for three
cases: sym=mrsym, sym=mr2sym, and sym=neither. The first and last
histogra
Hi Mike,
Yes, I can, and have, done such a plot. As you noted, there is strong
correlation between the rank of p1 and p2/p1 for each case (error and
no-error). There is also correlation between p1 and p2. But the extent of
correlation between p1 and p2 (or other quantities derived from them) i
Can you do a plot of p1 vs p2 and show one plot with errors and one with
correct?
Since you have p1 in both axes you automatically get a correlation between
them.
And could you post a link to download the data you have with
p1/p2/error/correct?
Thanks
Mike W9MDB
On Sat, Sep 26, 2015 at 3:49 PM,
Joe -
Just FYI - two plots attached. These are 2D histograms of the symbol rank and
p2/p1 ratio for all symbols associated with successfully decoded files from my
batch of 1000 JTSim files with SNR=-24dB. Note that each of these quantities
(rank, p2/p1) is invariant under any multiplicative sc
Hi Steve,
On 9/26/2015 10:40 AM, Steven Franke wrote:
> Don’t worry about it Joe - I tried scaling up my metrics by a factor
> of 4 and then kvasd gave me 644 decodes. So it’s clear that kvasd
> wants bigger numbers. For now, I’ll just focus on trying optimize sfrsd…
>
> Your rsdtest looks like a
Don’t worry about it Joe - I tried scaling up my metrics by a factor of 4 and
then kvasd gave me 644 decodes. So it’s clear that kvasd wants bigger numbers.
For now, I’ll just focus on trying optimize sfrsd…
Your rsdtest looks like a good way to do quick runs to test sfrsd. At your
leisure, can
Steve --
That one always did look a bit odd. Quite possibly I made a mistake,
trying to do too many things at once.
I will investigate further... but it may be not until Monday.
-- Joe
On 9/25/2015 11:34 PM, Steven Franke wrote:
> Hi Joe -
>
>> 17.WSJT + kvasd (SFM no ntest)
Hi Joe -
> 17.WSJT + kvasd (SFM no ntest) 897
I can’t reproduce this test. Whenever I try to run kvasd using my metrics, I
get no decodes. I have tried this using WSJT10 and WSJT-X. I have commented out
the ntest/nlow check and also the birdie test. At these low SNR’s, my metrics
wil
Hi Steve,
Sorry, no -- you'll have to comment out those calls, or the equivalent.
-- Joe
On 9/25/2015 11:26 AM, Steven Franke wrote:
> Joe -
>
> Is there an external way to turn off the attempts to decode the average in
> WSJT10? Not a big deal - I’m sure that I can figure out how to ch
Joe -
Is there an external way to turn off the attempts to decode the average in
WSJT10? Not a big deal - I’m sure that I can figure out how to change the code
to turn it off. As it stands, the program is calling the decoder to attempt to
decode the averaged signal immediately after it tries t
Hi Steve,
Just a quick reply, after reading your thoughtful comments. Your
thinking about what to try next is very similar to mine -- in
particular, making use of the second-best symbol value as a substitute
for simple erasures.
I should have mentioned that I in my tests with ntrials=10^5 and
You might consider trying the geometric mean. Multiple probabilities
should be normalized quite likely in this situation.
Mike W9MDB
On Fri, Sep 25, 2015 at 8:34 AM, Steven Franke
wrote:
> Hi Joe -
> Thanks for the new results! Getting ready for class here - so just a quick
> note. Your results
Hi Joe -
Thanks for the new results! Getting ready for class here - so just a quick
note. Your results, though more extensive than mine, point in the same
direction. As it stands, sfrsd seems to be around 0.5 dB shy of kvasd. I have a
long list of ideas to try to do better - but it will take ti
Hi Steve and all,
I've added more lines to the table summarizing my tests of decoding
weak, isolated JT65A signals. As before, the final number on each line
is the number of valid decodes from a thousand files at
S/N=-24 dB.
1. WSJT-X (BM only) 2
2. WSJT (BM only)
Hi Steve,
I'm happy to see that our test results are fully consistent. I look
forward to seeing how well sfrsd will perform in WSJT10 -- and
subsequently how much we can improve the JT65 decoding sensitivity in
WSJT-X.
-- Joe, K1JT
On 9/23/2015 9:02 PM, Steven Franke wrote:
> Thanks
Thanks Joe -
Your results in cases 1,3,4,5,7 agree with mine, to within a few decodes either
way. All of my runs were on OS X with sfrsd compiled using gcc - but perhaps
this produces a different set of random erasure vectors than either linux or
windows. Just now, I’ve also confirmed your resu
Hi Steve and all,
Here's an update on my tests of decoding weak JT65A signals. I used
1000 files generated by SimJT, each containing one JT65A signal at
S/N = -24 dB. I am using current code revisions of WSJT-X v1.6.1 (with
minor edits noted below) and WSJT v10.0.
For each line in the followi
Hi Steve,
On 9/23/2015 11:15 AM, Steven Franke wrote:
> Hi Joe -
> Yes - I hope that I didn’t give the impression that I was being
> critical of any of your quick integration decisions.
Not in the least!! I *very* much appreciate your input, and the
expertise you bring to this particular part o
Hi Joe -
Yes - I hope that I didn’t give the impression that I was being critical of any
of your quick integration decisions. I’m very aware of the tradeoffs that are
needed to optimize real-world performance. I’ve been focused on trying to
establish some sort of level playing field for testing
Hi Steve,
Thanks for sharing some further comparisons of kvasd and sfrsd.
When WSJT-X was first being expanded from a testbed for JT9 to a more
capable, more general program for HF DXing, JT65A capability was added
to make the program attractive to the large existing user base for that
mode.
Hi Claude -
Thanks for the feedback on sfrsd. As you discovered, it is not ready for use in
QSOs. At the moment, it is still in the “proof of concept” stage, and all
effort has been directed toward maximizing the probability of a successful
decode. Little to no attention has been paid to execut
Hi Joe -
I have committed a modification to demod64a that will compute new symbol
metrics for sfrsd. This significantly improves performance. The default is to
produce the normal kvasd metrics - it is necessary to go in and un-comment the
sfrsd metrics to test sfrsd. It will also be necessary t
Hi all,
In the past days, I have tried to use sfrsd as substitute for kvasd,
while performing QSO's on the air on SW.
It is difficult to compare to kvasd in this manner. It was just a test
to see what happens. In some instances, the decoding was not completed
at the end of the minute.
Best 88
Hi Joe -
I will likely not make much progress on the decoder stuff til next week, so I
wanted to pass along the following information in case that you get around to
doing some tests — I’ve tested the decoder on 2 separate batches of HF .wav
files and also on 3 batches of SimJT weak-signal gauss
On 20/09/2015 21:57, Steven Franke wrote:
> Hi Bill,
Hi Steve,
> Thanks for the comments. I will have to use the Google to learn about the
> Futures and Promises model…
It's pretty simple, you request a function (or something like a
function) to run in a thread (from a thread pool) and fetch the
Hi Bill,
Thanks for the comments. I will have to use the Google to learn about the
Futures and Promises model…
I have been looking at this:
http://clang-omp.github.io/
I have no idea what shape it’s in, but it looked like a potentially easy way
for a low-level programmer like me to parallelize
Hi Steve,
Congratulations on a really first-rate piece of work!!
I hope to find some time next week (or possibly the week after) to do
some independent tests -- presumably, to confirm your results.
I believe your test used simulated data on an AWGN channel. It might be
useful to try also
On 20/09/2015 16:11, Steven Franke wrote:
> You have probably already noticed the fact that the sfrsd should be easy to
> parallelize. It should be possible to test a bunch of candidate vectors at
> once. Perhaps that is something worth investigating for sfrsd. In an earlier
> thread, Bill had s
Joe -
For the record, here are the results of analyzing 1000 SimJT JT65A files with
sync level set to 0 (a setting of -1 gave the same results) and with Rx
frequency set equal to the simulated signal’s frequency:
kvasd (this result should be for the high-effort “on frequency” setting):
198/100
Hi Steve --
On 9/19/2015 6:03 PM, Steven Franke wrote:
> I have just now realized that the sync threshold is higher for “off
> frequency” signals. Since I had set Rx freq to 3000, the results
> reported below were obtained using the higher wideband sync threshold.
> That probably explains the lowe
I have just now realized that the sync threshold is higher for “off frequency”
signals. Since I had set Rx freq to 3000, the results reported below were
obtained using the higher wideband sync threshold. That probably explains the
lower than expected yield. I’m re-running with Rx frequency set e
Thanks Joe! This has been a fun project, and I’m learning a lot from it.
I used your SimJT program to generate 1000 JT65A files at -24 dB SNR. Here’s
what I get:
kvasd (with Rx frequency set to 3000, so this should be the low-effort
setting): 106 decodes out of 1000 (10.6%)
sfrsd 5000 trials:
Hi Steve,
I've looked again at the innards of sfrsd. I'm *much* impressed by what
you have done.
Soon, it may be time to look again at the upstream decisions made in the
JT65 decoding chain -- decisions that determine what symbol vectors are
passed on to the actual decoder. Among other thing
84 matches
Mail list logo
