I changed the config to... rain = discrete
gjr80 schrieb am Sonntag, 15. Januar 2023 um 00:06:43 UTC+1: > Your import config file is telling wee_import the input rainfall field > contains cumulative data, so wee_import uses the difference between the > current row rainfall data and the previous row rainfall data to calculate > the WeeWX field rain. For the first row there is no previous row so rain > is set to None (also note that the 1January 2019 00:00 rain value is > considered by WeeWX as containing 2018 data (ie data covering the period > from 23:59:00 to 23:59:59 + 1 second on 31 December 2018). The second row > is is the first record containing 2019 data. It should calculate the rain > field as being 0.2 - 0.2 = 0.0, though from your database extract it > appears to have been set to None. This could be a corner case involving > the change of year, but as far as I know wee_import should handle change > of year without problem. I will need to look further at the wee_import > code. The third row will calculate rain as 0 - 0.2 = -0.2 which of course > is a nonsense and I believe wee_import would assume there has been a > counter overflow and set rain to 0.0. For the fourth row rain is calculated > as 0 - 0 = 0.0 which is what you see. > > So other than the minor discrepancy at row two (for rain 0 and None are > essentially treated the same in WeeWX output) I think wee_import is > working as expected given the import data and import config settings. > Analysis of a larger sample of data involving non-zero rainfall data may > throw up more information. > > Gary > > On Sunday, 15 January 2023 at 08:40:05 UTC+10 schulz....@gmail.com wrote: > >> I think I should add a period to all values, not a comma. >> >> David Schulz schrieb am Samstag, 14. Januar 2023 um 23:35:21 UTC+1: >> >>> There should actually be values in there. >>> >>> echo "SELECT dateTime, rain FROM archive" | sqlite3 >>> /var/lib/weewx/weewx.sdb >>> 1546297200| >>> 1546297260| >>> 1546297320|0.0 >>> 1546297380|0.0 >>> >>> DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen >>> 01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7 >>> 01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9 >>> 01.01.2019 00:02;6,9;97;1026,6;0;4,7;250;7,9 >>> 01.01.2019 00:03;6,9;97;1026,6;0;4,7;245;4,7 >>> >>> >>> gjr80 schrieb am Samstag, 14. Januar 2023 um 23:19:12 UTC+1: >>> >>>> OK. Let's go straight to the horses head, what happens when you >>>> manually query the database archive table for the period concerned. Do >>>> each >>>> of the rows show the correct rainfall data or is it missing? >>>> >>>> Gary >>>> >>>> On Sunday, 15 January 2023 at 08:06:20 UTC+10 schulz....@gmail.com >>>> wrote: >>>> >>>>> Thanks for the info. But for testing I delete every time the whole DB >>>>> and the NOAA reports. I have a Weewx test environment specifically for >>>>> this. >>>>> >>>>> gjr80 schrieb am Samstag, 14. Januar 2023 um 23:03:41 UTC+1: >>>>> >>>>>> You need to be very careful when using the NOAA format reports to >>>>>> verify the import of historical data, this is because only the current >>>>>> month and current year reports are updated each report cycle. Historical >>>>>> reports are not updated so will not reflect and changes to earlier data. >>>>>> The sample data you provided is from January 2019 so I am guessing you >>>>>> did >>>>>> not import into the current month and year on your system. Try deleting >>>>>> all >>>>>> of your NOAA format reports from your WeeWX system, they will all be >>>>>> re-generated on the next report cycle based on the current database data. >>>>>> >>>>>> Gary >>>>>> >>>>>> On Sunday, 15 January 2023 at 07:06:08 UTC+10 schulz....@gmail.com >>>>>> wrote: >>>>>> >>>>>>> [image: 2023-01-14 22_05_47-Blub.png] >>>>>>> >>>>>>> David Schulz schrieb am Samstag, 14. Januar 2023 um 22:04:28 UTC+1: >>>>>>> >>>>>>>> wee_import --verbose --import-config=/etc/weewx/import/csv.conf >>>>>>>> /usr/share/weewx/wee_import:719: DeprecationWarning: The distutils >>>>>>>> package is deprecated and slated for removal in Python 3.12. Use >>>>>>>> setuptools >>>>>>>> or check PEP 632 for potential alternatives >>>>>>>> from distutils.version import StrictVersion >>>>>>>> Using WeeWX configuration file /etc/weewx/weewx.conf >>>>>>>> Starting wee_import... >>>>>>>> A CSV import from source file '/etc/weewx/import/EXP201901A.CSV' >>>>>>>> has been requested. >>>>>>>> The following options will be used: >>>>>>>> config=/etc/weewx/weewx.conf, >>>>>>>> import-config=/etc/weewx/import/csv.conf >>>>>>>> source=/etc/weewx/import/EXP201901A.CSV, from=None, to=None >>>>>>>> dry-run=None, calc_missing=True, ignore_invalid_data=True >>>>>>>> tranche=1000, interval=derive, >>>>>>>> date/time_string_format=%d.%m.%Y %H:%M >>>>>>>> delimiter=';', rain=cumulative, wind_direction=[-360.0, 360.0] >>>>>>>> UV=False, radiation=False >>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>> 'weewx.sdb' >>>>>>>> Destination table 'archive' unit system is '0x01' (US). >>>>>>>> Missing derived observations will be calculated. >>>>>>>> All WeeWX UV fields will be set to None. >>>>>>>> All WeeWX radiation fields will be set to None. >>>>>>>> Starting import ... >>>>>>>> Obtaining raw import data for period 1 ... >>>>>>>> The following imported field-to-WeeWX field map will be used: >>>>>>>> source field 'DatumZeit' in units 'unix_epoch' --> WeeWX field >>>>>>>> 'dateTime' >>>>>>>> source field 'Temp' in units 'degree_C' --> WeeWX field >>>>>>>> 'outTemp' >>>>>>>> source field 'Feuchte' in units 'percent' --> WeeWX field >>>>>>>> 'outHumidity' >>>>>>>> source field 'Luftdruck' in units 'hPa' --> WeeWX field >>>>>>>> 'barometer' >>>>>>>> source field 'Regen' in units 'mm' --> WeeWX field 'rain' >>>>>>>> source field 'Wind' in units 'km_per_hour' --> WeeWX field >>>>>>>> 'windSpeed' >>>>>>>> source field 'Richtung' in units 'degree_compass' --> WeeWX >>>>>>>> field 'windDir' >>>>>>>> source field 'Windboeen' in units 'km_per_hour' --> WeeWX >>>>>>>> field 'windGust' >>>>>>>> Raw import data read successfully for period 1. >>>>>>>> Mapping raw import data for period 1 ... >>>>>>>> Mapped 1440 records. >>>>>>>> Raw import data mapped successfully for period 1. >>>>>>>> Saving mapped data to archive for period 1 ... >>>>>>>> 1440 records identified for import. >>>>>>>> Proceeding will save all imported records in the WeeWX archive. >>>>>>>> Are you sure you want to proceed (y/n)? y >>>>>>>> Unique records processed: 1440; Last timestamp: 2019-01-01 23:59:00 >>>>>>>> CET (1546383540) >>>>>>>> Mapped data saved to archive successfully ""for period 1. >>>>>>>> Calculating missing derived observations ... >>>>>>>> Processing record: 1440; Last record: 2019-01-02 00:00:00 CET >>>>>>>> (1546383600) >>>>>>>> Recalculating daily summaries... >>>>>>>> Records processed: 1000; time: 2019-01-01 16:39:00 CET (1546357140) >>>>>>>> Finished recalculating daily summaries >>>>>>>> Finished calculating missing derived observations >>>>>>>> Finished import >>>>>>>> 1440 records were processed and 1440 unique records imported in >>>>>>>> 40.49 seconds. >>>>>>>> Those records with a timestamp already in the archive will not have >>>>>>>> been >>>>>>>> imported. Confirm successful import in the WeeWX log file. >>>>>>>> >>>>>>>> David Schulz schrieb am Samstag, 14. Januar 2023 um 22:03:34 UTC+1: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> i want to import some old weatherdata via csv file, but after the >>>>>>>>> import, the raindata is missing. i tried the import with "rain = >>>>>>>>> cumulative >>>>>>>>> / discrete". >>>>>>>>> >>>>>>>>> Here my config: >>>>>>>>> >>>>>>>>> source = CSV >>>>>>>>> ####################################### >>>>>>>>> [CSV] >>>>>>>>> file = /etc/weewx/import/EXP201901A.CSV >>>>>>>>> delimiter = ; >>>>>>>>> interval = derive >>>>>>>>> qc = True >>>>>>>>> calc_missing = True >>>>>>>>> ignore_invalid_data = True >>>>>>>>> tranche = 1000 >>>>>>>>> UV_sensor = False >>>>>>>>> solar_sensor = False >>>>>>>>> raw_datetime_format = %d.%m.%Y %H:%M >>>>>>>>> # rain = (discrete | cumulative) >>>>>>>>> rain = cumulative >>>>>>>>> wind_direction = -360,360 >>>>>>>>> [[FieldMap]] >>>>>>>>> dateTime = DatumZeit, unix_epoch >>>>>>>>> outTemp = Temp, degree_C >>>>>>>>> outHumidity = Feuchte, percent >>>>>>>>> barometer = Luftdruck, hPa >>>>>>>>> rain = Regen, mm >>>>>>>>> windSpeed = Wind, km_per_hour >>>>>>>>> windDir = Richtung, degree_compass >>>>>>>>> windGust = Windboeen, km_per_hour >>>>>>>>> >>>>>>>>> And here some sample data. >>>>>>>>> >>>>>>>>> DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen >>>>>>>>> 01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7 >>>>>>>>> 01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9 >>>>>>>>> 01.01.2019 00:02;6,9;97;1026,6;0,2;4,7;250;7,9 >>>>>>>>> 01.01.2019 00:03;6,9;97;1026,6;0,2;4,7;245;4,7 >>>>>>>>> 01.01.2019 00:04;6,9;97;1026,6;0;4,7;250;7,9 >>>>>>>>> 01.01.2019 00:05;6,9;97;1026,5;0;4,7;234;7,9 >>>>>>>>> 01.01.2019 00:06;6,9;97;1026,5;0;4,7;232;11,2 >>>>>>>>> 01.01.2019 00:07;7;97;1026,5;0;4,7;234;4,7 >>>>>>>>> 01.01.2019 00:08;7;97;1026,5;0;4,7;241;7,9 >>>>>>>>> 01.01.2019 00:09;7;97;1026,5;0;4,7;240;6,5 >>>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/950edcf6-bd45-4643-a815-b5f3be59cd54n%40googlegroups.com.
