[weewx-user] Re: weectl import csv KeyError: 'source_field'

[gjr80]

I think you will find this is the source of the error in your first post:

[[[inHumidity]]] source_field = inHumid unit = percent [[[windSpeed]]] 
source = windAvg unit = mile_per_hour [[[windDir]]] source_field = windDir 
unit = degree_compass

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/376d3d39-bb8b-47fe-9245-fe66804a365bn%40googlegroups.com.

[weewx-user] Re: weectl import csv KeyError: 'source_field'

[tarob...@gmail.com]

And I just realized what the issue was with the "Unknow units 
'unix-epoch'". Changed units from 'unix-epoch' to 'unix_epoch' and now it 
seems to be working. My older "csv-import.conf" is now working with the 
weectl import.

On Sunday, April 21, 2024 at 12:57:49 PM UTC-4 tarob...@gmail.com wrote:

> Using my old import-csv.conf file (attached), I get the following error:
>
> sudo weectl import --import-config=/home/piToad/Documents/csv-import.conf 
> --dry-runUsing configuration file /etc/weewx/weewx.conf
>
> This is a dry run. Nothing will actually be done.
> Starting weectl import...
>  Unknown units 'unix-epoch' specified for field 'dateTime' in 
> /home/piToad/Documents/csv-import.conf.
>  Nothing done, exiting.
>
> On Sunday, April 21, 2024 at 12:27:22 PM UTC-4 tarob...@gmail.com wrote:
>
>> Hello,
>>
>> I am trying to import some missing data via CSV file. I've done this 
>> numerous times pre weewx 5.0 and weectl (using wee_import). I've setup a 
>> new import config file using the new layout for [[FieldMap]]. I've attached 
>> the import config I used. However, during the --dry-run I am getting the 
>> below error. I've attached the config file.
>>
>> Column Headings: 
>> ['Timestamp','d5n1Temp', 'd5n1humid', 'd5n1Pres', 'rain5n1', 
>> 'windSpeed5n1',
>>'windAvg', 'windDir', 'UV', 'luminosity', 'd5n1Humid', 'rain',
>>'windSpeed', 'inTemp', 'inHumid', 'LiTemp', 'LiHumid', 'LiStrikes',
>>'LiDistance', 'LiPres', 'GTemp', 'GHumid', 'radiation']
>>
>> It's a source_field error but can't figure out which source_field is 
>> giving the error. I've looked them over but could be missing one.
>>
>> sudo weectl import 
>> --import-config=/home/piToad/Documents/csv-import-weectl.conf --dry-run
>> Using configuration file /etc/weewx/weewx.conf
>> This is a dry run. Nothing will actually be done.
>> Starting weectl import...
>> Traceback (most recent call last):
>>   File "/usr/share/weewx/weectl.py", line 74, in 
>> main()
>>   File "/usr/share/weewx/weectl.py", line 66, in main
>> namespace.func(namespace)
>>   File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch
>> namespace.action_func(config_dict, namespace)
>>   File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func
>> weectllib.import_actions.obs_import(config_dict,
>>   File "/usr/share/weewx/weectllib/import_actions.py", line 54, in 
>> obs_import
>> source_obj = 
>> weeimport.weeimport.Source.source_factory(config_dict['config_path'],
>> 
>>  ^
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 316, in 
>> source_factory
>> return get_object(module_class)(config_path,
>>^
>>   File "/usr/share/weewx/weeimport/csvimport.py", line 110, in __init__
>> self.map = self.parse_map(_map,
>>
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 551, in parse_map
>> _mapped_source_fields = [config['source_field'] for field, config in 
>> _map.items()]
>> 
>> ^^
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 551, in 
>> _mapped_source_fields = [config['source_field'] for field, config in 
>> _map.items()]
>>  ~~
>>   File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 554, 
>> in __getitem__
>> val = dict.__getitem__(self, key)
>>   ^^^
>> KeyError: 'source_field'
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/34977516-8d00-4cb9-80ea-5b6d3ee8d333n%40googlegroups.com.

[weewx-user] Re: weectl import csv KeyError: 'source_field'

[tarob...@gmail.com]

Using my old import-csv.conf file (attached), I get the following error:

sudo weectl import --import-config=/home/piToad/Documents/csv-import.conf 
--dry-runUsing configuration file /etc/weewx/weewx.conf
This is a dry run. Nothing will actually be done.
Starting weectl import...
 Unknown units 'unix-epoch' specified for field 'dateTime' in 
/home/piToad/Documents/csv-import.conf.
 Nothing done, exiting.

On Sunday, April 21, 2024 at 12:27:22 PM UTC-4 tarob...@gmail.com wrote:

> Hello,
>
> I am trying to import some missing data via CSV file. I've done this 
> numerous times pre weewx 5.0 and weectl (using wee_import). I've setup a 
> new import config file using the new layout for [[FieldMap]]. I've attached 
> the import config I used. However, during the --dry-run I am getting the 
> below error. I've attached the config file.
>
> Column Headings: 
> ['Timestamp','d5n1Temp', 'd5n1humid', 'd5n1Pres', 'rain5n1', 
> 'windSpeed5n1',
>'windAvg', 'windDir', 'UV', 'luminosity', 'd5n1Humid', 'rain',
>'windSpeed', 'inTemp', 'inHumid', 'LiTemp', 'LiHumid', 'LiStrikes',
>'LiDistance', 'LiPres', 'GTemp', 'GHumid', 'radiation']
>
> It's a source_field error but can't figure out which source_field is 
> giving the error. I've looked them over but could be missing one.
>
> sudo weectl import 
> --import-config=/home/piToad/Documents/csv-import-weectl.conf --dry-run
> Using configuration file /etc/weewx/weewx.conf
> This is a dry run. Nothing will actually be done.
> Starting weectl import...
> Traceback (most recent call last):
>   File "/usr/share/weewx/weectl.py", line 74, in 
> main()
>   File "/usr/share/weewx/weectl.py", line 66, in main
> namespace.func(namespace)
>   File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch
> namespace.action_func(config_dict, namespace)
>   File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func
> weectllib.import_actions.obs_import(config_dict,
>   File "/usr/share/weewx/weectllib/import_actions.py", line 54, in 
> obs_import
> source_obj = 
> weeimport.weeimport.Source.source_factory(config_dict['config_path'],
> 
>  ^
>   File "/usr/share/weewx/weeimport/weeimport.py", line 316, in 
> source_factory
> return get_object(module_class)(config_path,
>^
>   File "/usr/share/weewx/weeimport/csvimport.py", line 110, in __init__
> self.map = self.parse_map(_map,
>
>   File "/usr/share/weewx/weeimport/weeimport.py", line 551, in parse_map
> _mapped_source_fields = [config['source_field'] for field, config in 
> _map.items()]
> 
> ^^
>   File "/usr/share/weewx/weeimport/weeimport.py", line 551, in 
> _mapped_source_fields = [config['source_field'] for field, config in 
> _map.items()]
>  ~~
>   File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 554, 
> in __getitem__
> val = dict.__getitem__(self, key)
>   ^^^
> KeyError: 'source_field'
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3ae8edb0-9438-423e-9342-1b6c95735f0an%40googlegroups.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES
#
# Copyright (c) 2009-2019 Tom Keffer  and Gary Roderick.
# See the file LICENSE.txt for your rights.

##

# Specify the source. Available options are:
#   CSV - import obs from a single CSV format file
#   WU - import obs from a Weather Underground PWS history
#   Cumulus - import obs from a one or more Cumulus monthly log files
#   WD - import obs from a one or more WD monthly log files
# Format is:
#   source = (CSV | WU | Cumulus)
source = CSV

##

[CSV]
# Parameters used when importing from a CSV file

# Path and name of our CSV source file. Format is:
#   file = full path and filename
file = /var/tmp/dataCorrection.csv

# The character used to separate fields. Format is:
#   delimiter = 
# Default is , (comma).
delimiter = ','

# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
#   derive - Derive the interval field from the timestamp of successive
#records. This setting is best used when the imported records
#are equally spaced in time and there are no missing records.
#   co

[weewx-user] Re: Weectl import

['J O' via weewx-user]

Hi Gary,
thanks for your quick response. I'm afraid that I should have done that, 
but now I already have two databases because the new installation started 
and the old has been stopped. So I will need to import in the one or the 
other direction anyway. Or, alternatively, I will loose the data during the 
overlap time (1+ week).
Jens

gjr80 schrieb am Freitag, 8. März 2024 um 22:48:16 UTC+1:

> I am not sure why you are exporting data from your old WeeWX database and 
> then importing the data into a new installation. The WeeWX database is 
> compatible across all WeeWX versions so you can just use it without change 
> in your new WeeWX installation. Your existing WeeWX database may use the 
> old 'wview' database schema rather than the new 'view_extended' database 
> schema, but there is no particular need to upgrade to the 'view_extended' 
> database schema unless you need the additional fields. In any case, you can 
> upgrade an existing WeeWX installation that uses the old 'wview' database 
> schema to the new 'view_extended' database schema by following the 
> instructions at the Switching to the new wview_extended schema 
> 
>  
> wiki page.
>
> I would suggest you just copy your old WeeWX database to your new install 
> and if required later update to the 'wview_extended' database schema.
>
> Gary
> On Saturday 9 March 2024 at 07:09:23 UTC+10 ott...@googlemail.com wrote:
>
>> Dear community,
>> I try to import the database of my old weewx installation to the new 
>> system (newly installed + new hardware) by exporting CSV file from the old 
>> weewx.sdb and import with weectl import onto the new.
>> The process interrupts and I receive error messages as long as I have 
>> NULL values in the CSV (NULL represented by comma separators without any 
>> value ",,"). When I remove the NULLs e.g. by "0" the process completes 
>> successfully.
>> Does anybody have an idea, because I do not want to fill up the NULLs 
>> with "0" because this is not accurate.
>> Thank you for support
>> JO
>> Here ist the errorlog from weectl import:
>>
>> Using database binding 'wx_binding', which is bound to database 
>> 'weewx.sdb'
>> Destination table 'archive' unit system is '0x01' (US).
>> Missing derived observations will be calculated.
>> All WeeWX UV fields will be set to None.
>> All WeeWX radiation fields will be set to None.
>> This is a dry run, imported data will not be saved to archive.
>> Starting dry run import ...
>> Traceback (most recent call last):
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 849, in 
>> map_raw_data
>> _value = float(_row[self.map[_field]['source_field']].strip())
>> ValueError: could not convert string to float: ''
>>
>> During handling of the above exception, another exception occurred:
>>
>> Traceback (most recent call last):
>>   File "/usr/share/weewx/weectl.py", line 74, in 
>> main()
>>   File "/usr/share/weewx/weectl.py", line 66, in main
>> namespace.func(namespace)
>>   File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch
>> namespace.action_func(config_dict, namespace)
>>   File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func
>> weectllib.import_actions.obs_import(config_dict,
>>   File "/usr/share/weewx/weectllib/import_actions.py", line 58, in 
>> obs_import
>> source_obj.run()
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 405, in run
>> _mapped_data = self.map_raw_data(_raw_data, self.archive_unit_sys)
>>   File "/usr/share/weewx/weeimport/weeimport.py", line 897, in 
>> map_raw_data
>> self.map[_field]['unit'] == 'degree_compass':
>>   File "/usr/lib/python3/dist-packages/configobj.py", line 554, in 
>> __getitem__
>> val = dict.__getitem__(self, key)
>> KeyError: 'unit'
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/48cb62a2-dc8a-4887-acd8-80f525a22a0an%40googlegroups.com.

[weewx-user] Re: Weectl import

[gjr80]

I am not sure why you are exporting data from your old WeeWX database and 
then importing the data into a new installation. The WeeWX database is 
compatible across all WeeWX versions so you can just use it without change 
in your new WeeWX installation. Your existing WeeWX database may use the 
old 'wview' database schema rather than the new 'view_extended' database 
schema, but there is no particular need to upgrade to the 'view_extended' 
database schema unless you need the additional fields. In any case, you can 
upgrade an existing WeeWX installation that uses the old 'wview' database 
schema to the new 'view_extended' database schema by following the 
instructions at the Switching to the new wview_extended schema 

 
wiki page.

I would suggest you just copy your old WeeWX database to your new install 
and if required later update to the 'wview_extended' database schema.

Gary
On Saturday 9 March 2024 at 07:09:23 UTC+10 ott...@googlemail.com wrote:

> Dear community,
> I try to import the database of my old weewx installation to the new 
> system (newly installed + new hardware) by exporting CSV file from the old 
> weewx.sdb and import with weectl import onto the new.
> The process interrupts and I receive error messages as long as I have NULL 
> values in the CSV (NULL represented by comma separators without any value 
> ",,"). When I remove the NULLs e.g. by "0" the process completes 
> successfully.
> Does anybody have an idea, because I do not want to fill up the NULLs with 
> "0" because this is not accurate.
> Thank you for support
> JO
> Here ist the errorlog from weectl import:
>
> Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
> Destination table 'archive' unit system is '0x01' (US).
> Missing derived observations will be calculated.
> All WeeWX UV fields will be set to None.
> All WeeWX radiation fields will be set to None.
> This is a dry run, imported data will not be saved to archive.
> Starting dry run import ...
> Traceback (most recent call last):
>   File "/usr/share/weewx/weeimport/weeimport.py", line 849, in map_raw_data
> _value = float(_row[self.map[_field]['source_field']].strip())
> ValueError: could not convert string to float: ''
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/share/weewx/weectl.py", line 74, in 
> main()
>   File "/usr/share/weewx/weectl.py", line 66, in main
> namespace.func(namespace)
>   File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch
> namespace.action_func(config_dict, namespace)
>   File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func
> weectllib.import_actions.obs_import(config_dict,
>   File "/usr/share/weewx/weectllib/import_actions.py", line 58, in 
> obs_import
> source_obj.run()
>   File "/usr/share/weewx/weeimport/weeimport.py", line 405, in run
> _mapped_data = self.map_raw_data(_raw_data, self.archive_unit_sys)
>   File "/usr/share/weewx/weeimport/weeimport.py", line 897, in map_raw_data
> self.map[_field]['unit'] == 'degree_compass':
>   File "/usr/lib/python3/dist-packages/configobj.py", line 554, in 
> __getitem__
> val = dict.__getitem__(self, key)
> KeyError: 'unit'
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/bef71334-cd5b-4ba0-a7a4-9ff5a758ece5n%40googlegroups.com.