Hi Mike,
I finally had some time to tinker with this some. On my Win laptop using
WSL, MSYS2, or MonaXterm (Cygwin port), import performance is not
impressive by any means. I suspect files system interoperability is the
root cause. Searches, once imported, are fine. On my Linux workstation
imports are extremely fast; note: I'm running MongoDB on an XFS
partition with an abundance of Ram (MongoDB is Ram piggy).
I tinkered around with (3) scripts:
Import Script (atimport.sh)
Link: http://paste.ubuntu.com/p/bh5vGB5GPW/
Call Sign Lookup (atcheck.sh)
Link: http://paste.ubuntu.com/p/NHv9tztx2Z/
Parse line into fields (retest.py)
Link: https://paste.ubuntu.com/p/8HdPcTVXpN/
I think parsing the lines into fields is the best long-term solution for
storage (allows for much better indexing). However, we'd need to do a
fair bit of checking on each line to determine its structure first;
check == "slower imports". Once the main (first) bulk import is done,
incremental updates would be snappy.
I'm sure there are far better solutions; like a UDP service that just
pushes the decodes straight into a database (of any kind) :-)
73's
Greg, KI7MT
On 7/1/19 3:05 PM, Black Michael via wsjt-devel wrote:
Please do post the code.
Thanks Greg.
de Mike W9MDB
On Monday, July 1, 2019, 03:53:15 PM CDT, Greg Beam <ki7m...@gmail.com>
wrote:
Hello All,
Here's an example from today's log:
Results: https://paste.ubuntu.com/p/382VVMth4S/
The query takes about (2) seconds or so using a $regex search on
7,390,224 logged events matching two callsigns; this is without being
indexed nor field splitting. It is one string per line imported to a one
field document in MongDB
I can post the script I used as a gist, if anyone is interested:
- Copies the ALL.TXT to $temp_file
- Converts it to CSV
- Generates two helper JS scripts
- Generates one example JS query
- Drops, then re-imports the alltext collection
- Runs the a sample Query.
Note: for incremental (daily) updates, there is no need to drop the
collection (alltext) before inserting new events. I just do that for
performance testing.
It's a simple one-line query command that would work on Win/Linux/Mac:
mongo < example.js
You can, if desired, write any number of commands to perform stats,
lookup(s), whatever, and use the same easy method to query the DB.
However, as this is a single sting entry, much of the analytical value
would be missing, as the fields / categories are not captured.
73's
Greg, KI7MT
On 7/1/19 1:27 AM, Claude Frantz wrote:
> On 7/1/19 7:59 AM, Claude Frantz wrote:
>
> Just as an example of code extract in perl:
>
> if ($line =~ m/^(\d{4})-([A-Z][a-z]{2})-(\d{2})\b/ ) {
> $day = $3 ;
> $month_alpha = $2 ;
> $year = $1 ;
> }
> elsif ($line =~ m/^(\d\d)(\d\d)(\d\d)_\d{6}\b/ ) {
> $day = $3 ;
> $month_num = $2 ;
> $year = 2000 + $1 ;
> }
> elsif ($line =~ m/^(\d{4})-(\d\d)-(\d\d)\b/ ) {
> $day = $3 ;
> $month_num = $2 ;
> $year = $1 ;
> }
>
> I have not tested it, I hope there is no error. This allow to decode the
> 3 formats of ALL.TXT about which ones I remember about. Please note that
> the month can be numeric or alpha. If alpha, you have to convert to
> numeric, if you want to compare to a numeric value. Please note also,
> that the mode switching was an extra line in previous formats.
>
> Best wishes,
> Claude (DJ0OT)
_______________________________________________
wsjt-devel mailing list
wsjt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wsjt-devel
