Re: [wsjt-devel] ALL.TXT (again)

[Greg Beam]

Hi Dan,

The example's I spoke of aren't running through a JS engine, they are 
being ran through Mongo itself to perform the queries. If you want to 
run native JS server side, you'd be better off install NodeJs or 
similar. There are a ton of PGP/MongoDB tutorials out there. The main 
difference being, you'd be using MongoClient through an ODM most likely.


There's nothing to say one could not have multiple collections; one with 
a full string such as the example I did, and another that has each field
broken out by field types. Breaking out the fields has a number of 
advantages, particularly for analytics.


I'll tidy up the Bash script I use, and post it as a Gist. You could run 
it through MSYS2, Cygwin, or WSL even, as all should have access to 
Windows Local Servers. It would take a few changes for Path variables, 
but nothing major. I have Apache running on my server as it hosts some 
of testing environment. I'll may play around with a PHP form or two and 
see how that goes.


73's
Greg, KI7MT


On 7/1/19 3:45 PM, Dan Malcolm wrote:

Because I want to search via my webserver, I have a separate PHP script that 
does the search.  Probably not as fast as MongoDB.  I do get good cohesive 
reports though.  I get a report for example that will show me just one of my 
QSO's, and will store results in a text file.  That makes it useful to refer to 
later, or to include in an email to my QSO partner.

All that said I would like to explore MongoDB.  The idea that query via js 
script may mean that I can still have private web access.  I use IIS on Win10 
and just the local machine can access it.

__
Dan – K4SHQ


-Original Message-
From: Greg Beam [mailto:ki7m...@gmail.com]
Sent: Monday, July 1, 2019 3:47 PM
To: wsjt-devel@lists.sourceforge.net
Subject: Re: [wsjt-devel] ALL.TXT (again)

Hello All,

Here's an example from today's log:

Results: https://paste.ubuntu.com/p/382VVMth4S/

The query takes about (2) seconds or so using a $regex search on
7,390,224 logged events matching two callsigns; this is without being indexed 
nor field splitting. It is one string per line imported to a one field document 
in MongDB

I can post the script I used as a gist, if anyone is interested:

- Copies the ALL.TXT to $temp_file
- Converts it to CSV
- Generates two helper JS scripts
- Generates one example JS query
- Drops, then re-imports the alltext collection
- Runs the a sample Query.

Note: for incremental (daily) updates, there is no need to drop the collection 
(alltext) before inserting new events. I just do that for performance testing.

It's a simple one-line query command that would work on Win/Linux/Mac:

mongo < example.js

You can, if desired, write any number of commands to perform stats, lookup(s), 
whatever, and use the same easy method to query the DB.
However, as this is a single sting entry, much of the analytical value would be 
missing, as the fields / categories are not captured.

73's
Greg, KI7MT





___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Dan Malcolm]

Because I want to search via my webserver, I have a separate PHP script that 
does the search.  Probably not as fast as MongoDB.  I do get good cohesive 
reports though.  I get a report for example that will show me just one of my 
QSO's, and will store results in a text file.  That makes it useful to refer to 
later, or to include in an email to my QSO partner.

All that said I would like to explore MongoDB.  The idea that query via js 
script may mean that I can still have private web access.  I use IIS on Win10 
and just the local machine can access it.

__
Dan – K4SHQ


-Original Message-
From: Greg Beam [mailto:ki7m...@gmail.com] 
Sent: Monday, July 1, 2019 3:47 PM
To: wsjt-devel@lists.sourceforge.net
Subject: Re: [wsjt-devel] ALL.TXT (again)

Hello All,

Here's an example from today's log:

Results: https://paste.ubuntu.com/p/382VVMth4S/

The query takes about (2) seconds or so using a $regex search on
7,390,224 logged events matching two callsigns; this is without being indexed 
nor field splitting. It is one string per line imported to a one field document 
in MongDB

I can post the script I used as a gist, if anyone is interested:

- Copies the ALL.TXT to $temp_file
- Converts it to CSV
- Generates two helper JS scripts
- Generates one example JS query
- Drops, then re-imports the alltext collection
- Runs the a sample Query.

Note: for incremental (daily) updates, there is no need to drop the collection 
(alltext) before inserting new events. I just do that for performance testing.

It's a simple one-line query command that would work on Win/Linux/Mac:

mongo < example.js

You can, if desired, write any number of commands to perform stats, lookup(s), 
whatever, and use the same easy method to query the DB. 
However, as this is a single sting entry, much of the analytical value would be 
missing, as the fields / categories are not captured.

73's
Greg, KI7MT



On 7/1/19 1:27 AM, Claude Frantz wrote:
> On 7/1/19 7:59 AM, Claude Frantz wrote:
> 
> Just as an example of code extract in perl:
> 
> if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
>  $day = $3 ;
>  $month_alpha = $2 ;
>  $year = $1 ;
> }
> elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
>  $day = $3 ;
>  $month_num = $2 ;
>  $year = 2000 + $1 ;
> }
> elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
>  $day = $3 ;
>  $month_num = $2 ;
>  $year = $1 ;
>  }
> 
> I have not tested it, I hope there is no error. This allow to decode 
> the
> 3 formats of ALL.TXT about which ones I remember about. Please note 
> that the month can be numeric or alpha. If alpha, you have to convert 
> to numeric, if you want to compare to a numeric value. Please note 
> also, that the mode switching was an extra line in previous formats.
> 
> Best wishes,
> Claude (DJ0OT)
> 
> 
> ___
> wsjt-devel mailing list
> wsjt-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/wsjt-devel


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Black Michael via wsjt-devel]

Please do post the code.
Thanks Greg.
de Mike W9MDB

 

On Monday, July 1, 2019, 03:53:15 PM CDT, Greg Beam  
wrote:  
 
 Hello All,

Here's an example from today's log:

Results: https://paste.ubuntu.com/p/382VVMth4S/

The query takes about (2) seconds or so using a $regex search on 
7,390,224 logged events matching two callsigns; this is without being 
indexed nor field splitting. It is one string per line imported to a one 
field document in MongDB

I can post the script I used as a gist, if anyone is interested:

- Copies the ALL.TXT to $temp_file
- Converts it to CSV
- Generates two helper JS scripts
- Generates one example JS query
- Drops, then re-imports the alltext collection
- Runs the a sample Query.

Note: for incremental (daily) updates, there is no need to drop the 
collection (alltext) before inserting new events. I just do that for 
performance testing.

It's a simple one-line query command that would work on Win/Linux/Mac:

mongo < example.js

You can, if desired, write any number of commands to perform stats, 
lookup(s), whatever, and use the same easy method to query the DB. 
However, as this is a single sting entry, much of the analytical value 
would be missing, as the fields / categories are not captured.

73's
Greg, KI7MT



On 7/1/19 1:27 AM, Claude Frantz wrote:
> On 7/1/19 7:59 AM, Claude Frantz wrote:
> 
> Just as an example of code extract in perl:
> 
> if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
>      $day = $3 ;
>      $month_alpha = $2 ;
>      $year = $1 ;
> }
> elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
>      $day = $3 ;
>      $month_num = $2 ;
>      $year = 2000 + $1 ;
> }
> elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
>      $day = $3 ;
>      $month_num = $2 ;
>      $year = $1 ;
>      }
> 
> I have not tested it, I hope there is no error. This allow to decode the 
> 3 formats of ALL.TXT about which ones I remember about. Please note that 
> the month can be numeric or alpha. If alpha, you have to convert to 
> numeric, if you want to compare to a numeric value. Please note also, 
> that the mode switching was an extra line in previous formats.
> 
> Best wishes,
> Claude (DJ0OT)
> 
> 
> ___
> wsjt-devel mailing list
> wsjt-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/wsjt-devel


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel
  ___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Greg Beam]

Hello All,

Here's an example from today's log:

Results: https://paste.ubuntu.com/p/382VVMth4S/

The query takes about (2) seconds or so using a $regex search on 
7,390,224 logged events matching two callsigns; this is without being 
indexed nor field splitting. It is one string per line imported to a one 
field document in MongDB


I can post the script I used as a gist, if anyone is interested:

- Copies the ALL.TXT to $temp_file
- Converts it to CSV
- Generates two helper JS scripts
- Generates one example JS query
- Drops, then re-imports the alltext collection
- Runs the a sample Query.

Note: for incremental (daily) updates, there is no need to drop the 
collection (alltext) before inserting new events. I just do that for 
performance testing.


It's a simple one-line query command that would work on Win/Linux/Mac:

mongo < example.js

You can, if desired, write any number of commands to perform stats, 
lookup(s), whatever, and use the same easy method to query the DB. 
However, as this is a single sting entry, much of the analytical value 
would be missing, as the fields / categories are not captured.


73's
Greg, KI7MT



On 7/1/19 1:27 AM, Claude Frantz wrote:

On 7/1/19 7:59 AM, Claude Frantz wrote:

Just as an example of code extract in perl:

if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
     $day = $3 ;
     $month_alpha = $2 ;
     $year = $1 ;
}
elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
     $day = $3 ;
     $month_num = $2 ;
     $year = 2000 + $1 ;
}
elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
     $day = $3 ;
     $month_num = $2 ;
     $year = $1 ;
     }

I have not tested it, I hope there is no error. This allow to decode the 
3 formats of ALL.TXT about which ones I remember about. Please note that 
the month can be numeric or alpha. If alpha, you have to convert to 
numeric, if you want to compare to a numeric value. Please note also, 
that the mode switching was an extra line in previous formats.


Best wishes,
Claude (DJ0OT)


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel



___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] dT

[K2DBK-WSJT]

Anyone else have any ideas? Maybe someone familiar with the framework used?

 

From: K2DBK-WSJT  
Sent: Saturday, June 29, 2019 12:11 PM
To: 'WSJT software development' 
Subject: Re: [wsjt-devel] dT

 

Hi Sergio,

 

I’m not quite sure what you are suggesting. The issue is really just a cosmetic 
problem with the appearance, everything works OK otherwise.

 

 

-- 

73,

David, K2DBK

http://k2dbk.com

twitter: @k2dbk

 

From: Sergio Yes UT9LI mailto:ut...@ukr.net> > 
Sent: Saturday, June 29, 2019 6:00 AM
To: WSJT software development mailto:wsjt-devel@lists.sourceforge.net> >
Subject: [wsjt-devel] dT

 

Setting the dT parameter obviously gives results: in Log ((call)) have dT close 
to the level ((0)). The problem disappears !?

 

 

 

 

 

29 июня 2019, 01:09:42, от K2DBK-WSJT mailto:k2dbk+w...@k2dbk.com> >: 

 

I’m trying to help out a less computer-savvy friend of mine with an issue with 
how the main windows displays with v2.0.1 (not 2.1 rc7). I’m trying to get some 
additional information about his hardware, but I he’s believe using a laptop 
with 4K screen resolution using Windows 10. In order to have things not display 
in a microscopic size, his icon and app display settings were initially 
adjusted for a scale factor of 250% (which is recommended). Other applications 
seem to display correctly. I’ve made some adjustments to the scale factor and 
adjusted the font size in the application, so most things are OK.  However, if 
you look at the screenshot you’ll see that the column heads at the top as well 
as the meter at the lower-left are cut off. Prior to the adjustment, the dB 
meter slowed only a tiny slice on the left hand edge of the green bar and only 
a single digit of the indicated strength as it was cut off.

 

I’m looking for some guidance as to how to adjust the combination of in-app 
font settings along with anything that I can adjust in Windows to get this to 
look as it should.

 



 

 

 

 

___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net  
https://lists.sourceforge.net/lists/listinfo/wsjt-devel
 

 

 

-- 

Good Luck! Best DX's! 73!

de Sergio fm Kharkov BYE-Poka!

___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Greg Beam]

Hello All,

This is similar to how I parse the file also; read / split the line and 
check line[0], then do what's needed based on checking the first string.


At present, my ALL.TXT is over 400MB. What I've been doing to prevent 
read lock issues is creating a daily diff file between a copy and the 
active ALL.TXT file; sticking each diff-file in folder to process at 
whatever time I wish without affecting WSJT-X operations:


alltxt-diff-20190629-0300.txt
alltxt-diff-20190630-0300.txt
etc, etc, etc

After each diff run, I update the copy so it's ready for the next day. 
There are hundreds of ways to accomplish the same thing, but, I found 
this to be easy and fairly painless (disk space is cheap these days :-) )


What to do with the data after has been my focus of late. I've been 
playing around with MongoDB (a schema-less JSON/BSON Document storage 
database) to sick the decoded lines in. You can either split the lines, 
or just stick the entire line in as a new document for long term 
storage/later-date access.


The $regex processing capability of MongoDB is extensive, and very fast! 
One can easily parse a multitude of string combinations, even with the 
entire line in one field, for example:


use wsjtx;
db.alltxt.find( { $and: [
{event:{$regex:'MY-CALL'}},
{event:{$regex:'HIS-CALL'}}
]
});


That would print the lines (documents) that contains both 'my-call' and 
'his-call'.


You could add ..'DATE_STRING' or any combination you wish to further 
refine the search without having to split the lines at all.


In case folks are worried about the number of documents in each 
collection, I've added the entire WSPR Decode Archive (from WSPRnet) to 
a MongoDB Database/collection set (one collection for each year, 2008 
thru 2019, at just over 95GB on disk size). Later collections have 
"millions" of decodes in them. Single collection Query Times are =< 1 to 
2 seconds. With added indexing, times are in the Millisecond range :-) 
Aggregate queries, those spanning multiple collections/years, vary in 
time depending on the data being sought but are well within an 
acceptable time limit for most use cases I've had.


73's
Greg, KI7MT

On 7/1/19 1:27 AM, Claude Frantz wrote:

On 7/1/19 7:59 AM, Claude Frantz wrote:

Just as an example of code extract in perl:

if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
     $day = $3 ;
     $month_alpha = $2 ;
     $year = $1 ;
}
elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
     $day = $3 ;
     $month_num = $2 ;
     $year = 2000 + $1 ;
}
elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
     $day = $3 ;
     $month_num = $2 ;
     $year = $1 ;
     }

I have not tested it, I hope there is no error. This allow to decode the 
3 formats of ALL.TXT about which ones I remember about. Please note that 
the month can be numeric or alpha. If alpha, you have to convert to 
numeric, if you want to compare to a numeric value. Please note also, 
that the mode switching was an extra line in previous formats.


Best wishes,
Claude (DJ0OT)


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel



___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

[wsjt-devel] interesting splatters of a 2m FT8 signal

[DG2YCB, Uwe]

Hi,

I just saw on 2m a FT8 signal which had strange splatters about 270 and
2*270 Hz below the main signal, the splatters narrower than the main
transmission. There was only this station on the band. What may be the
reason for these splatters?







73 de Uwe, DG2YCB



___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Claude Frantz]

On 7/1/19 7:59 AM, Claude Frantz wrote:

Just as an example of code extract in perl:

if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
$day = $3 ;
$month_alpha = $2 ;
$year = $1 ;
}
elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
$day = $3 ;
$month_num = $2 ;
$year = 2000 + $1 ;
}
elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
$day = $3 ;
$month_num = $2 ;
$year = $1 ;
}

I have not tested it, I hope there is no error. This allow to decode the 
3 formats of ALL.TXT about which ones I remember about. Please note that 
the month can be numeric or alpha. If alpha, you have to convert to 
numeric, if you want to compare to a numeric value. Please note also, 
that the mode switching was an extra line in previous formats.


Best wishes,
Claude (DJ0OT)


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel

Re: [wsjt-devel] ALL.TXT (again)

[Claude Frantz]

On 6/30/19 10:18 PM, Dan Malcolm wrote:


That’s correct Claude.

But my PHP program has to deal with both formats in 2019.  Given that one of 
the formats will  be found, all I have to detect is a change in month, which 
comes after the date is harvested from the line (string).


I suggest to try to match both formats, in sequence. When the one 
matches, you decode the date. When not, you try to match to the second 
format. When it matches, you extract the date. When there is no match, 
then the line contains another data and you should ignore it. Be sure to 
match at the beginning of the line. Note the "^" as the first character 
of the regex.


Best wishes,
Claude (DJ0OT)


___
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel