[weewx-user] Re: Importing Weatherflow data

2021-02-23 Thread mbat...@gmail.com 
Please keep "us" in the loop... I'm sure I'm going to want to duplicate 
these efforts for my station...
Thanks!

Paul

On Tuesday, February 23, 2021 at 4:06:39 AM UTC-6 gjr80 wrote:

> Hi,
>
> You more or less have it. CSV imports require a header row so that 
> wee_import can identify the source data fields and map them to WeeWX 
> field as per the field map. If the data you intend to import has no header 
> row then you can add one, the field names are not important, the important 
> part is making sure you use the correct field name in the field map.
>
> There is no --test, I think you mean --dry-run instead. Best thing is to 
> work through the CSV import step-by-step instructions 
>  in the 
> Utilities Guide, but I would also revise all of the CSV related sections as 
> well.
>
> In terms of checking the import results, that really comes down to you and 
> how much confirmation you require. In order of increasing complexity you 
> might consider the following:
>
> 1. did the import executed without error?
> 2. did the reported number of records processed and imported match your 
> expectations?
> 3. were there any errors in the log?
> 4. were there any unexplained failures to save a record to database in the 
> log? (a record may not be saved to database because a record for that 
> timestamp already exists)
> 5. do the NOAA month and year reports for the period imported have 
> sensible/expected values?
> 6. did the number of records saved to database match your expectation? 
> (the only way to confirm the number of records saved to database is to 
> count the success 'record added' messga ein the log, unfortunately 
> wee_import is unable to determine how many records are actually saved to 
> database, it can only tell how many records it processed and attempted to 
> save to database)
> 7. does each field of the imported data in the database match the 
> corresponding field/record in the source? (this one is for hardcore users, 
> there are no tools for doing this it is a manual process)
>
> I would do 1-5 but 6 and 7, well that depends on how hard core you are :)
>
> Gary
>
>
> On Tuesday, 23 February 2021 at 16:37:29 UTC+10 david.a@gmail.com 
> wrote:
>
>> sorry, my touchpad has caused early posting!
>> To continue, 
>> 5. Edit the import file to include the field names.
>> 6. run wee-import --test only
>> 7. evaluate results, make corrections
>> 8. run final import
>> 9. restart weewx
>>
>> any way to verify success?
>>
>>
>> On Monday, February 22, 2021 at 10:27:44 PM UTC-8 David Prellwitz wrote:
>>
>>> I'm trying to import (wee_import) two weeks of data from my tempest 
>>> repository held at Weatherflow. The API page example of my data shows no 
>>> header in the csv file. How should i handle this import? should i edit the 
>>> file by adding the first reacord and inserting the proper field names?
>>> i assume the normal approach is to:
>>> 1. create the import config file.
>>> 2. extract the WeatherFlow data use their API.
>>> 3. stop weewx, 
>>> 4. copy current db as backup, 
>>> 5. edit 
>>> 6. 
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/f24eae16-ed16-49d1-9156-f85c6ef8b3a2n%40googlegroups.com.

[weewx-user] Re: Importing Weatherflow data

2021-02-23 Thread gjr80 
Hi,

You more or less have it. CSV imports require a header row so that 
wee_import can identify the source data fields and map them to WeeWX field 
as per the field map. If the data you intend to import has no header row 
then you can add one, the field names are not important, the important part 
is making sure you use the correct field name in the field map.

There is no --test, I think you mean --dry-run instead. Best thing is to 
work through the CSV import step-by-step instructions 
 in the 
Utilities Guide, but I would also revise all of the CSV related sections as 
well.

In terms of checking the import results, that really comes down to you and 
how much confirmation you require. In order of increasing complexity you 
might consider the following:

1. did the import executed without error?
2. did the reported number of records processed and imported match your 
expectations?
3. were there any errors in the log?
4. were there any unexplained failures to save a record to database in the 
log? (a record may not be saved to database because a record for that 
timestamp already exists)
5. do the NOAA month and year reports for the period imported have 
sensible/expected values?
6. did the number of records saved to database match your expectation? (the 
only way to confirm the number of records saved to database is to count the 
success 'record added' messga ein the log, unfortunately wee_import is 
unable to determine how many records are actually saved to database, it can 
only tell how many records it processed and attempted to save to database)
7. does each field of the imported data in the database match the 
corresponding field/record in the source? (this one is for hardcore users, 
there are no tools for doing this it is a manual process)

I would do 1-5 but 6 and 7, well that depends on how hard core you are :)

Gary


On Tuesday, 23 February 2021 at 16:37:29 UTC+10 david.a@gmail.com wrote:

> sorry, my touchpad has caused early posting!
> To continue, 
> 5. Edit the import file to include the field names.
> 6. run wee-import --test only
> 7. evaluate results, make corrections
> 8. run final import
> 9. restart weewx
>
> any way to verify success?
>
>
> On Monday, February 22, 2021 at 10:27:44 PM UTC-8 David Prellwitz wrote:
>
>> I'm trying to import (wee_import) two weeks of data from my tempest 
>> repository held at Weatherflow. The API page example of my data shows no 
>> header in the csv file. How should i handle this import? should i edit the 
>> file by adding the first reacord and inserting the proper field names?
>> i assume the normal approach is to:
>> 1. create the import config file.
>> 2. extract the WeatherFlow data use their API.
>> 3. stop weewx, 
>> 4. copy current db as backup, 
>> 5. edit 
>> 6. 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/d0833664-b6f5-49d4-a3b6-7c7020b65965n%40googlegroups.com.

[weewx-user] Re: Importing Weatherflow data

2021-02-22 Thread David Prellwitz 
sorry, my touchpad has caused early posting!
To continue, 
5. Edit the import file to include the field names.
6. run wee-import --test only
7. evaluate results, make corrections
8. run final import
9. restart weewx

any way to verify success?


On Monday, February 22, 2021 at 10:27:44 PM UTC-8 David Prellwitz wrote:

> I'm trying to import (wee_import) two weeks of data from my tempest 
> repository held at Weatherflow. The API page example of my data shows no 
> header in the csv file. How should i handle this import? should i edit the 
> file by adding the first reacord and inserting the proper field names?
> i assume the normal approach is to:
> 1. create the import config file.
> 2. extract the WeatherFlow data use their API.
> 3. stop weewx, 
> 4. copy current db as backup, 
> 5. edit 
> 6. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/cf1c3360-2673-47d0-8ba0-14de86edfd07n%40googlegroups.com.