subject:"\[weewx\-user\] Re\: Importing old station data via CSV\: how to map time\?"

[weewx-user] Re: Importing old station data via CSV: how to map time?

2017-03-15 Thread gjr80

If you were using 3.6.x it looks like issue #176 
 may have been causing the 
latest issue. This was fixed in 3.7.0.

Gary

On Thursday, 16 March 2017 11:15:19 UTC+10, Max G wrote:
>
> Hi Gary,
>
> Thank you kindly for checking this out for me...
> I think the thread and your response is important, as many have these 3081 
> weather stations...
>
> Well, I copied your field map straight into the config file:
>
> source = CSV
> file = /home/maxg/wwimport/test4import.txt
> interval = 10
> qc = True
> calc_missing = True
> tranche = 250
> UV_sensor = True
> solar_sensor = True
> raw_datetime_format = %d-%m-%Y %H:%M
> rain = discrete
> wind_direction = -360,360
>
> # test4import.txt:
> #No,Time,Interval(mi),Indoor Humidity(%),Indoor 
> Temperature(°C),Outdoor Humidity(%),Outdoor Temperature(°C),Absolute 
> Pressure(hPa),Wind(km/h),Gust(km/h),Direction,Relative 
> Pressure(hPa),Dewpoint(°C),Windchill(°C),Hour Rainfall(mm),24 hour 
> Rainfall(mm),Week Rainfall(mm),Month Rainfall(mm),Total Rainfall(mm),Wind 
> Level(bft),Gust Level(bft),lightmeter(lux),UV
> #1,16-02-2017 
> 09:00,10,57,28.8,64,26.2,993.1,2.5,5.0,NW,1012.0,18.9,26.2,0.0,0.0,0.0,0.9,654.0,1,1,86352.0,0
> #2,16-02-2017 
> 13:46,10,57,28.4,64,26.5,993.0,5.0,8.6,N,1010.6,19.2,26.5,0.0,0.0,0.0,0.9,654.0,1,2,148704.0,0
> #3,16-02-2017 
> 13:56,10,57,28.4,57,28.7,992.9,1.1,3.6,N,1010.5,19.4,28.7,0.0,0.0,0.0,0.9,654.0,1,1,96896.0,0
>
> [[FieldMap]]
> dateTime= Time
> usUnits =
> interval= Interval(mi), minute
> barometer   = Relative Pressure(hPa), hPa
> pressure= Absolute Pressure(hPa), hPa
> altimeter   =
> inTemp  = Indoor Temperature(°C), degree_C
> outTemp = Outdoor Temperature(°C), degree_C
> inHumidity  = Indoor Humidity(%), percent
> outHumidity = Outdoor Humidity(%), percent
> windSpeed   = Wind(km/h), km_per_hour
> windDir =
> windGust= Gust(km/h), km_per_hour
> windGustDir =
> rainRate=
> rain= Total Rainfall(mm), mm
> dewpoint= Dewpoint(°C), degree_C
> windchill   = Windchill(°C), degree_C
> heatindex   =
> ET  =
> radiation   =
> UV  = UV, uv_index
>
>
>
> and got the following result (failure):
>
> wee_import --import-config=/home/maxg/wwimport/csv.conf --dry-run
> Starting wee_import...
> A CSV import from source file '/home/maxg/wwimport/test4import.txt' has 
> been requested.
> Using database binding 'wx_binding', which is bound to database 
> 'weewx.sdb'
> Destination table 'archive' unit system is '0x01' (US).
> Missing derived observations will be calculated.
> This is a dry run, imported data will not be saved to archive.
>  Unable to parse source-to-weewx field map.
>  No mapping specified in '/home/maxg/wwimport/csv.conf' for field 
> 'dateTime'.
>  Nothing done, exiting.
>
>
> then I thought, you might be using the latest versions... so I upgraded to 
> 3.7, and get this (success):
>
> # [2017-03-16 11:02] maxg@rpi1b ~ $
> wee_import --import-config=/home/maxg/wwimport/csv.conf --dry-run
> Starting wee_import...
> A CSV import from source file '/home/maxg/wwimport/test4import.txt' has 
> been requested.
> Using database binding 'wx_binding', which is bound to database 
> 'weewx.sdb'
> Destination table 'archive' unit system is '0x01' (US).
> Missing derived observations will be calculated.
> This is a dry run, imported data will not be saved to archive.
> Starting dry run import ...
> 3 records identified for import.
> Records processed: 3; Unique records: 3; Last timestamp: 2017-02-16 13:56:
> 00 AEST (1487217360)
> Finished dry run import. 3 records were processed and 3 unique records 
> would have been imported.
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing old station data via CSV: how to map time?

2017-03-15 Thread Max G

Hi Gary,

Thank you kindly for checking this out for me...
I think the thread and your response is important, as many have these 3081 
weather stations...

Well, I copied your field map straight into the config file:

source = CSV
file = /home/maxg/wwimport/test4import.txt
interval = 10
qc = True
calc_missing = True
tranche = 250
UV_sensor = True
solar_sensor = True
raw_datetime_format = %d-%m-%Y %H:%M
rain = discrete
wind_direction = -360,360

# test4import.txt:
#No,Time,Interval(mi),Indoor Humidity(%),Indoor Temperature(°C),Outdoor 
Humidity(%),Outdoor Temperature(°C),Absolute 
Pressure(hPa),Wind(km/h),Gust(km/h),Direction,Relative 
Pressure(hPa),Dewpoint(°C),Windchill(°C),Hour Rainfall(mm),24 hour 
Rainfall(mm),Week Rainfall(mm),Month Rainfall(mm),Total Rainfall(mm),Wind 
Level(bft),Gust Level(bft),lightmeter(lux),UV
#1,16-02-2017 
09:00,10,57,28.8,64,26.2,993.1,2.5,5.0,NW,1012.0,18.9,26.2,0.0,0.0,0.0,0.9,654.0,1,1,86352.0,0
#2,16-02-2017 
13:46,10,57,28.4,64,26.5,993.0,5.0,8.6,N,1010.6,19.2,26.5,0.0,0.0,0.0,0.9,654.0,1,2,148704.0,0
#3,16-02-2017 
13:56,10,57,28.4,57,28.7,992.9,1.1,3.6,N,1010.5,19.4,28.7,0.0,0.0,0.0,0.9,654.0,1,1,96896.0,0

[[FieldMap]]
dateTime= Time
usUnits =
interval= Interval(mi), minute
barometer   = Relative Pressure(hPa), hPa
pressure= Absolute Pressure(hPa), hPa
altimeter   =
inTemp  = Indoor Temperature(°C), degree_C
outTemp = Outdoor Temperature(°C), degree_C
inHumidity  = Indoor Humidity(%), percent
outHumidity = Outdoor Humidity(%), percent
windSpeed   = Wind(km/h), km_per_hour
windDir =
windGust= Gust(km/h), km_per_hour
windGustDir =
rainRate=
rain= Total Rainfall(mm), mm
dewpoint= Dewpoint(°C), degree_C
windchill   = Windchill(°C), degree_C
heatindex   =
ET  =
radiation   =
UV  = UV, uv_index



and got the following result (failure):

wee_import --import-config=/home/maxg/wwimport/csv.conf --dry-run
Starting wee_import...
A CSV import from source file '/home/maxg/wwimport/test4import.txt' has 
been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
 Unable to parse source-to-weewx field map.
 No mapping specified in '/home/maxg/wwimport/csv.conf' for field 
'dateTime'.
 Nothing done, exiting.


then I thought, you might be using the latest versions... so I upgraded to 
3.7, and get this (success):

# [2017-03-16 11:02] maxg@rpi1b ~ $
wee_import --import-config=/home/maxg/wwimport/csv.conf --dry-run
Starting wee_import...
A CSV import from source file '/home/maxg/wwimport/test4import.txt' has 
been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
3 records identified for import.
Records processed: 3; Unique records: 3; Last timestamp: 2017-02-16 13:56:00 
AEST (1487217360)
Finished dry run import. 3 records were processed and 3 unique records 
would have been imported.



-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing old station data via CSV: how to map time?

2017-03-14 Thread gjr80

Hi,

A few things. I sat down just now with to import your data from your first 
post. I used the following field map:

[[FieldMap]]
dateTime= Time
usUnits =
interval= Interval(mi), minute
barometer   = Relative Pressure(hPa), hPa
pressure= Absolute Pressure(hPa), hPa
altimeter   =
inTemp  = Indoor Temperature(°C), degree_C
outTemp = Outdoor Temperature(°C), degree_C
inHumidity  = Indoor Humidity(%), percent
outHumidity = Outdoor Humidity(%), percent
windSpeed   = Wind(km/h), km_per_hour
windDir =
windGust= Gust(km/h), km_per_hour
windGustDir = 
rainRate= 
rain= Total Rainfall(mm), mm
dewpoint= Dewpoint(°C), degree_C
windchill   = Windchill(°C), degree_C
heatindex   =
ET  =
radiation   =
UV  = UV, uv_index

Points to note:

   - If you wish to fill weeWX field interval from a field in you data then 
   in needs a mapping of the format interval = csv_field_name, weewx_units 
   (where csv_field_name is the field name in your csv data that holds the 
   interval value and weewx_units are one of minute or second). If you want 
   to set the interval field for all imported records to a fixed number, 
   say 10 minutes, then you should leave the interval field map blank and set 
interval 
   = 10 in your import config file. Not particularly intuitive, but not too 
   many sources provide interval data and it is explained in the import config 
   file comments.
   - Unfortunately wee_import does not handle ordinal/cardinal compass 
   directions for wind direction fields, only numerical degrees. One way 
   around this would be to change the cardinal/ordinal directions in your 
   import data to their degree equivalents then something like windDir = 
   Direction, degree_compass would work. Alternatively I could look into 
   modifying wee_import to handle cardinal/ordinal directions, at the 
   moment weeWX does not freely convert between degrees and cardinal/ordinal 
   compass points, something I would need to discuss with Tom. In any case at 
   least a week for me to make the changes
   - I suspect your latest error is due to an incorrect csv field name. I 
   would suggest you are best deriving the weeWX rain field from the csv 
   field Total Rainfall(mm) - maybe that was your intent and you made a 
   typo? Refer to my field map above. One thing to watch is that when deriving 
   weeWX field rain from a cumulative rainfall field if the cumulative 
   field happens to rollover to 0 or otherwise overflow, you may end up with 
   incorrect data imported for that record. Usually you are best to use the 
   cumulative field that rolls over the least frequent.
   - You can certainly import your lux field info into the weeWX field 
radiation 
   but they are two different measurements - radiation is watts per square 
   meter (sic) and lux is in, well, lux (radiation will typically be in the 
   range 0 to 1600 odd - I think the theoretical limit is 1361 but for a 
   number of reasons some sensors can read higher than this). You may be able 
   to convert you lux data to watts per square meter and then import the data. 
   Alternatively, provided you have no radiation limits in you [StdQC] settings 
   in weewx.conf you could import you lux data and then manipulate the data 
   in the database to get data in watts per square meter. Of course you could 
   leave it as imported and accept that your radiation data is lux (expect 
   some units issues though if displaying using weeWX built in formatting)
   - The UV field map should use units of uv_index.

Gary


On Wednesday, 15 March 2017 10:00:57 UTC+10, Max G wrote:
>
> Read on this forum, that dateTime does not need a unit. Fine
>
> However, running the following config against the sample data, here:
>
> #No,Time,Interval(mi),Indoor Humidity(%),Indoor 
> Temperature(°C),Outdoor Humidity(%),Outdoor Temperature(°C),Absolute 
> Pressure(hPa),Wind(km/h),Gust(km/h),Direction,Relative 
> Pressure(hPa),Dewpoint(°C),Windchill(°C),Hour Rainfall(mm),24 hour 
> Rainfall(mm),Week Rainfall(mm),Month Rainfall(mm),Total Rainfall(mm),Wind 
> Level(bft),Gust Level(bft),lightmeter(lux),UV
> #1,16-02-2017 
> 09:00,10,57,28.8,64,26.2,993.1,2.5,5.0,NW,1012.0,18.9,26.2,0.0,0.0,0.0,0.9,654.0,1,1,86352.0,0
> #2,16-02-2017 
> 13:46,10,57,28.4,64,26.5,993.0,5.0,8.6,N,1010.6,19.2,26.5,0.0,0.0,0.0,0.9,654.0,1,2,148704.0,0
> #3,16-02-2017 
> 13:56,10,57,28.4,57,28.7,992.9,1.1,3.6,N,1010.5,19.4,28.7,0.0,0.0,0.0,0.9,654.0,1,1,96896.0,0
>
> [[FieldMap]]
> dateTime= Time
> usUnits = 
> interval= 10
> barometer   = Relative Pressure(hPa), hPa
> pressure= Absolute Pressure(hPa), hPa
> altimeter   = 
> inTemp  = Indoor Temperature(°C), degree_C
> outTem

[weewx-user] Re: Importing old station data via CSV: how to map time?

2017-03-14 Thread Max G

Read on this forum, that dateTime does not need a unit. Fine

However, running the following config against the sample data, here:

#No,Time,Interval(mi),Indoor Humidity(%),Indoor Temperature(°C),Outdoor 
Humidity(%),Outdoor Temperature(°C),Absolute 
Pressure(hPa),Wind(km/h),Gust(km/h),Direction,Relative 
Pressure(hPa),Dewpoint(°C),Windchill(°C),Hour Rainfall(mm),24 hour 
Rainfall(mm),Week Rainfall(mm),Month Rainfall(mm),Total Rainfall(mm),Wind 
Level(bft),Gust Level(bft),lightmeter(lux),UV
#1,16-02-2017 
09:00,10,57,28.8,64,26.2,993.1,2.5,5.0,NW,1012.0,18.9,26.2,0.0,0.0,0.0,0.9,654.0,1,1,86352.0,0
#2,16-02-2017 
13:46,10,57,28.4,64,26.5,993.0,5.0,8.6,N,1010.6,19.2,26.5,0.0,0.0,0.0,0.9,654.0,1,2,148704.0,0
#3,16-02-2017 
13:56,10,57,28.4,57,28.7,992.9,1.1,3.6,N,1010.5,19.4,28.7,0.0,0.0,0.0,0.9,654.0,1,1,96896.0,0

[[FieldMap]]
dateTime= Time
usUnits = 
interval= 10
barometer   = Relative Pressure(hPa), hPa
pressure= Absolute Pressure(hPa), hPa
altimeter   = 
inTemp  = Indoor Temperature(°C), degree_C
outTemp = Outdoor Temperature(°C), degree_C
inHumidity  = Indoor Humidity(%), percent
outHumidity = Outdoor Humidity(%), percent
windSpeed   = Wind(km/h), km_per_hour
windDir = Direction, compass
windGust= Gust(km/h), km_per_hour
windGustDir = Direction, compass
rainRate= 
rain= Rainfall(mm), mm
dewpoint= 
windchill   = 
heatindex   = 
ET  = 
radiation   = lightmeter(lux), lux
UV  = UV


I get this error:
wee_import --import-config=/home/maxg/wwimport/csv.conf --dry-run
Starting wee_import...
Traceback (most recent call last):
  File "/usr/bin/wee_import", line 703, in 
main()
  File "/usr/bin/wee_import", line 652, in main
wlog)
  File "/usr/share/weewx/weeimport/weeimport.py", line 315, in sourceFactory
log)
  File "/usr/share/weewx/weeimport/csvimport.py", line 49, in __init__
log)
  File "/usr/share/weewx/weeimport/weeimport.py", line 183, in __init__
self.import_QC = weewx.qc.QC(config_dict, parent='weeimport')
  File "/usr/share/weewx/weewx/qc.py", line 49, in __init__
minval = converter.convert(vt)[0]
  File "/usr/share/weewx/weewx/units.py", line 762, in convert
new_val_t = convert(val_t, new_unit_type)
  File "/usr/share/weewx/weewx/units.py", line 1079, in convert
conversion_func = conversionDict[val_t[1]][target_unit_type]
KeyError: 'inch'


No idea how to solve this?! :(

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Importing old station data via CSV: how to map time?

[weewx-user] Re: Importing old station data via CSV: how to map time?

[weewx-user] Re: Importing old station data via CSV: how to map time?

[weewx-user] Re: Importing old station data via CSV: how to map time?

4 matches

Site Navigation

Mail list logo

Footer information