Just to close the loop on this, CSV imports with a usUnits field mapping are working fine under WeeWX 4.0.0b6. Documentation has been updated and all should be fine once WeeWX v4 is released.
Gary On Thursday, 20 June 2019 06:38:27 UTC+10, rich T wrote: > > Gary > > Removed usUnits and interval from the csv.conf and it worked like a champ. > > Thanks > Rich > > On Tuesday, June 18, 2019 at 11:29:55 PM UTC-4, gjr80 wrote: > >> Rich, >> >> Try deleting the mapping for usUnits from the field map. Long story >> short, it does not make much sense to be importing/converting a unit system >> code when units are specified/implied for all the fields. I know this isn't >> covered in the wee_import guide, will fix that up later. >> >> Gary >> >> On Wednesday, 19 June 2019 13:10:13 UTC+10, rich T wrote: >>> >>> Gary >>> >>> I have the raw_datetime_format as "%m/%d/%Y %H:%M", but forgot to >>> include it in previous post. Now I receive the following error: >>> >>> pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import >>> --import-config=/home/pi/Desktop/clog/csv.conf --dry-run >>> Starting wee_import... >>> A CSV import from source file >>> '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. >>> Using database binding 'wx_binding', which is bound to database >>> 'weewx.sdb' >>> Destination table 'archive' unit system is '0x01' (US). >>> Missing derived observations will be calculated. >>> All WeeWX UV fields will be set to None. >>> All WeeWX radiation fields will be set to None. >>> This is a dry run, imported data will not be saved to archive. >>> **** Unable to map source data. >>> **** Field 'units' not found in source data. >>> **** Nothing done, exiting. >>> >>> Thanks >>> Rich >>> >>> On Tuesday, June 18, 2019 at 10:46:20 PM UTC-4, gjr80 wrote: >>>> >>>> The error in the original post is due to one of the [[FieldMap]] >>>> mappings not having WeeWX unit name. You do not need to include mappings >>>> for all fields in the CSV source, only those you wish to import, but those >>>> you do include must be of the form: >>>> >>>> WeeWX field = CSV field name, WeeWX unit name >>>> >>>> You are correct in that the raw_datetime_format setting determines how >>>> the CSV date-time field is decoded. For the data shown I would expect >>>> something like (untested): >>>> >>>> raw_datetime_format = "%m/%d/%Y %H:%M" >>>> >>>> this assumes the date is in the format month/day/year, just transpose >>>> %m and %d if it id day/month/year. >>>> >>>> Gary >>>> >>>> On Wednesday, 19 June 2019 12:33:03 UTC+10, rich T wrote: >>>>> >>>>> I could be wrong, but the raw_datetime_format is the datetime format >>>>> in the CSV file (i.e. %m/%d%Y). The dateTime = timestamp, unix_epoch in >>>>> the CSV Map basically means convert the timestamp column into unix_epoch >>>>> time format. >>>>> >>>>> On Tuesday, June 18, 2019 at 10:06:49 PM UTC-4, Andrew Milner wrote: >>>>>> >>>>>> shouldn't it just be dateTime = timestamp and the format of timestamp >>>>>> be specified in raw_datetime_format ?? >>>>>> >>>>>> >>>>>> >>>>>> On Wednesday, 19 June 2019 04:32:30 UTC+3, rich T wrote: >>>>>>> >>>>>>> I'm trying to import a large CSV file into the database and I'm >>>>>>> receiving the following error the the dry run. >>>>>>> >>>>>>> pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import >>>>>>> --import-config=/home/pi/Desktop/clog/csv.conf --dry-run >>>>>>> Starting wee_import... >>>>>>> A CSV import from source file >>>>>>> '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. >>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>> 'weewx.sdb' >>>>>>> Destination table 'archive' unit system is '0x01' (US). >>>>>>> Missing derived observations will be calculated. >>>>>>> All WeeWX UV fields will be set to None. >>>>>>> All WeeWX radiation fields will be set to None. >>>>>>> This is a dry run, imported data will not be saved to archive. >>>>>>> Traceback (most recent call last): >>>>>>> File "./wee_import", line 703, in <module> >>>>>>> main() >>>>>>> File "./wee_import", line 653, in main >>>>>>> source_obj.run() >>>>>>> File "/home/weewx/bin/weeimport/weeimport.py", line 350, in run >>>>>>> _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys) >>>>>>> File "/home/weewx/bin/weeimport/weeimport.py", line 721, in >>>>>>> mapRawData >>>>>>> self.map[_field]['units'], >>>>>>> KeyError: 'units' >>>>>>> >>>>>>> Here is the mapping in the csv.conf: >>>>>>> >>>>>>> [[FieldMap]] >>>>>>> dateTime = timestamp, unix_epoch >>>>>>> usUnits = units >>>>>>> interval = interval >>>>>>> barometer = barometer, inHg >>>>>>> pressure = pressure >>>>>>> altimeter = altimeter >>>>>>> inTemp = inTemp >>>>>>> outTemp = outTemp, degree_F >>>>>>> inHumidity = inHumidity >>>>>>> outHumidity = outHumidity, percent >>>>>>> windSpeed = windSpeed, mile_per_hour >>>>>>> windDir = windDir, degree_compass >>>>>>> windGust = windGust, mile_per_hour >>>>>>> windGustDir = windGustDir, degree_compass >>>>>>> rainRate = rainRate, inch_per_hour >>>>>>> rain = rain, inch >>>>>>> dewpoint = dewpoint >>>>>>> windchill = windchill >>>>>>> heatindex = heatindex >>>>>>> ET = >>>>>>> radiation = >>>>>>> UV = >>>>>>> rxCheckPercent = rxCheckPercent >>>>>>> >>>>>>> Here is a small portion of the CSV file: >>>>>>> >>>>>>> >>>>>>> timestamp,units,interval,barometer,pressure,altimeter,inTemp,outTemp,inHumidity,outHumidity,windSpeed,windDir,windGust,windGustDir,rainRate,rain,dewpoint,windchill,heatindex,ET,radiation,UV,extraTemp1,extraTemp2,extraTemp3,soilTemp1,soilTemp2,soilTemp3,soilTemp4,leafTemp1,leafTemp2,extraHumid1,extraHumid2,soilMoist1,soilMoist2,soilMoist3,soilMoist4,leafWet1,leafWet2,rxCheckPercent,txBatteryStatus,consBatteryVoltage,hail,hailRate,heatingTemp,heatingVoltage,supplyVoltage,referenceVoltage,windBatteryStatus,rainBatteryStatus,outTempBatteryStatus,inTempBatteryStatus >>>>>>> 04/09/2018 >>>>>>> 00:05,1,5,30.101,,,77.2,24.2,20,62,0,,0,,0,0,13.08666521,24.2,24.2,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:10,1,5,30.102,,,77,24,20,63,0,,0,,0,0,13.25922371,24,24,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:15,1,5,30.108,,,76.9,23.8,20,63,0,,0,,0,0,13.06932323,23.8,23.8,0,,,,,,,,,,,,,,,,,,,,96.52083333,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:20,1,5,30.11,,,76.7,23.7,20,63,0,,0,,0,0,12.97437126,23.7,23.7,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:25,1,5,30.112,,,76.6,23.5,20,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,77.72916667,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:30,1,5,30.11,,,76.4,23.5,21,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:35,1,5,30.111,,,76.7,23.3,20,64,0,,3,225,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:40,1,5,30.106,,,76.9,23.3,20,64,0,,0,,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:45,1,5,30.11,,,77,23.1,20,64,0,,0,,0,0,12.76018685,23.1,23.1,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>>> 04/09/2018 >>>>>>> 00:50,1,5,30.114,,,77,23,20,65,0,,0,,0,0,13.01554258,23,23,0,,,,,,,,,,,,,,,,,,,,100,,,,,,,,,,,, >>>>>>> >>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6d17b916-a48d-463a-b373-3e2391a44e9d%40googlegroups.com.
