Hello, I am trying to import some missing data via CSV file. I've done this numerous times pre weewx 5.0 and weectl (using wee_import). I've setup a new import config file using the new layout for [[FieldMap]]. I've attached the import config I used. However, during the --dry-run I am getting the below error. I've attached the config file.
Column Headings: ['Timestamp','d5n1Temp', 'd5n1humid', 'd5n1Pres', 'rain5n1', 'windSpeed5n1', 'windAvg', 'windDir', 'UV', 'luminosity', 'd5n1Humid', 'rain', 'windSpeed', 'inTemp', 'inHumid', 'LiTemp', 'LiHumid', 'LiStrikes', 'LiDistance', 'LiPres', 'GTemp', 'GHumid', 'radiation'] It's a source_field error but can't figure out which source_field is giving the error. I've looked them over but could be missing one. sudo weectl import --import-config=/home/piToad/Documents/csv-import-weectl.conf --dry-run Using configuration file /etc/weewx/weewx.conf This is a dry run. Nothing will actually be done. Starting weectl import... Traceback (most recent call last): File "/usr/share/weewx/weectl.py", line 74, in <module> main() File "/usr/share/weewx/weectl.py", line 66, in main namespace.func(namespace) File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch namespace.action_func(config_dict, namespace) File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func weectllib.import_actions.obs_import(config_dict, File "/usr/share/weewx/weectllib/import_actions.py", line 54, in obs_import source_obj = weeimport.weeimport.Source.source_factory(config_dict['config_path'], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/weewx/weeimport/weeimport.py", line 316, in source_factory return get_object(module_class)(config_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/weewx/weeimport/csvimport.py", line 110, in __init__ self.map = self.parse_map(_map, ^^^^^^^^^^^^^^^^^^^^ File "/usr/share/weewx/weeimport/weeimport.py", line 551, in parse_map _mapped_source_fields = [config['source_field'] for field, config in _map.items()] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/weewx/weeimport/weeimport.py", line 551, in <listcomp> _mapped_source_fields = [config['source_field'] for field, config in _map.items()] ~~~~~~^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 554, in __getitem__ val = dict.__getitem__(self, key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: 'source_field' -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/01940de3-e0ef-4d5f-a242-483e16d82d81n%40googlegroups.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES # # Copyright (c) 2009-2019 Tom Keffer <tkef...@gmail.com> and Gary Roderick. # See the file LICENSE.txt for your rights. ############################################################################## # Specify the source. Available options are: # CSV - import obs from a single CSV format file # WU - import obs from a Weather Underground PWS history # Cumulus - import obs from a one or more Cumulus monthly log files # WD - import obs from a one or more WD monthly log files # Format is: # source = (CSV | WU | Cumulus) source = CSV ############################################################################## [CSV] # Parameters used when importing from a CSV file # Path and name of our CSV source file. Format is: # file = full path and filename file = /home/piToad/Documents/dataCorrection.csv # The character used to separate fields. Format is: # delimiter = <single character> # Default is , (comma). delimiter = ',' # If there is no mapped interval field how will the interval field be # determined for the imported records. Available options are: # derive - Derive the interval field from the timestamp of successive # records. This setting is best used when the imported records # are equally spaced in time and there are no missing records. # conf - Use the interval setting from weewx.conf. This setting is # best used if the records to be imported have been produced by # WeeWX using the same archive interval as set in weewx.conf on # this machine. # x - Use a fixed interval of x minutes for every record. This # setting is best used if the records to be imported are # equally based in time but there are some missing records. # # Note: If there is a mapped interval field then this setting will be # ignored. # Format is: # interval = (derive | conf | x) interval = 5 # Should the [StdQC] max/min limits in weewx.conf be applied to the # imported data. This may be useful if the source has extreme values that # are clearly incorrect for some observations. Available options are: # True - weewx.conf [StdQC] max/min limits are applied. # False - weewx.conf [StdQC] max/min limits are not applied. # Format is: # qc = (True | False) qc = True # Should any missing derived observations be calculated from the imported # data if possible. Available options are: # True - Any missing derived observations are calculated. # False - Any missing derived observations are not calculated. # Format is: # calc_missing = (True | False) calc_missing = True # Specify how imported data fields that contain invalid data (eg a numeric # field containing non-numeric data) are handled. Available options are: # True - The invalid data is ignored, the WeeWX target field is set to # None and the import continues. # False - The import is halted. # Format is: # ignore_invalid_data = (True | False) # Default is True. ignore_invalid_data = True # Imported records are written to archive in transactions of tranche # records at a time. Increase for faster throughput, decrease to reduce # memory requirements. Format is: # tranche = x # where x is an integer tranche = 250 # Specify whether a UV sensor was used to produce any UV observations. # Available options are: # True - UV sensor was used and UV data will be imported. # False - UV sensor was not used and any UV data will not be imported. # UV fields will be set to None/NULL. # For a CSV import UV_sensor should be set to False if a UV sensor was # NOT present when the import data was created. Otherwise it may be set to # True or omitted. Format is: # UV_sensor = (True | False) UV_sensor = True # Specify whether a solar radiation sensor was used to produce any solar # radiation observations. Available options are: # True - Solar radiation sensor was used and solar radiation data will # be imported. # False - Solar radiation sensor was not used and any solar radiation # data will not be imported. radiation fields will be set to # None/NULL. # For a CSV import solar_sensor should be set to False if a solar radiation # sensor was NOT present when the import data was created. Otherwise it may # be set to True or omitted. Format is: # solar_sensor = (True | False) solar_sensor = True # Date-time format of CSV field from which the WeeWX archive record # dateTime field is to be extracted. wee_import first attempts to interpret # date/time info in this format, if this fails it then attempts to # interpret it as a timestamp and if this fails it then raises an error. # Uses Python strptime() format codes. # raw_datetime_format = Python strptime() format string raw_datetime_format = %Y-%m-%d %H:%M:%S # Does the imported rain field represent the total rainfall since the last # record or a cumulative value. Available options are: # discrete - rain field represents total rainfall since last record # cumulative - rain field represents a cumulative rainfall reset at # midnight # rain = (discrete | cumulative) rain = cumulative # Lower and upper bounds for imported wind direction. It is possible, # particularly for a calculated direction, to have a value (eg -45) outside # of the WeeWX limits (0 to 360 inclusive). Format is: # # wind_direction = lower,upper # # where : # lower is the lower limit of acceptable wind direction in degrees # (may be negative) # upper is the upper limit of acceptable wind direction in degrees # # Imported values from lower to upper will be normalised to the range 0 to # 360. Values outside of the parameter range will be stored as None. # Default is -360,360. wind_direction = -360,360 # Map CSV record fields to WeeWX archive fields. Format is: # # weewx_archive_field_name = csv_field_name, weewx_unit_name # # where: # weewx_archive_field_name - An observation name in the WeeWX database # schema. # csv_field_name - The name of a field from the CSV file. # weewx_unit_name - The name of the units, as defined in WeeWX, # used by csv_field_name. wee_import will do # the necessary conversions to the unit system # used by the WeeWX archive. # For example, # outTemp = Temp, degree_C # would map the CSV field Temp, in degrees C, to the archive field outTemp. # # A mapping for WeeWX field dateTime is mandatory and the WeeWX unit name # for the dateTime mapping must be unix_epoch. For example, # dateTime = csv_date_and_time, unix_epoch # would map the CSV field csv_date_and_time to the WeeWX dateTime field with # the csv_date_and_time field being interpreted first using the format # specified at the raw_datetime_format config option and if that fails as a # unix epoch timestamp. # # Field mapping to the WeeWX usUnits archive field is currently not # supported. If a usUnits field exists in the CSV data it should not be # mapped, rather WeeWX unit names should included against each field to be # imported in the field map. # # WeeWX archive fields that do not exist in the CSV data may be omitted. # Any omitted fields that are derived (eg dewpoint) may be calculated # during import using the equivalent of the WeeWX StdWXCalculate service # through setting the calc-missing parameter above. [[FieldMap]] [[[dateTime]]] source_field = Timestamp unit = unix_epoch [[[outTemp]]] source_field = d5n1Temp unit = degree_F [[[outHumidity]]] source_field = d5n1humid unit = percent [[[barometer]]] source_field = d5n1Pres unit = inHg [[[inTemp]]] source_field = inTemp unit = degree_F [[[inHumidity]]] source_field = inHumid unit = percent [[[windSpeed]]] source = windAvg unit = mile_per_hour [[[windDir]]] source_field = windDir unit = degree_compass [[[windGust]]] source_field = windSpeed unit = mile_per_hour [[[rain]]] source_field = rain unit = inch is_cumulative = True [[[radiation]]] source_field = radiation unit = watt_per_meter_squared [[[UV]]] source_field = UV unit = uv_index [[[luminosity]]] source_field = group_illuminance unit = lux [[[extraTemp1]]] source_field = extraTemp1 unit = degree_F [[[extraHumid1]]] source_field = extraHumid1 unit = percent [[[extraTemp2]]] source_field = LiTemp unit = degree_F [[[extraHumid2]]] source_field = LiHumid unit = percent [[[lightning_distance]]] source_field = LiDistance unit = mile