Yup. Only concern (though not huge) is pressure vs barometer. According to
WC author, what's in the csv is "pressure". But that doesn't make sense to
me since WC displays "Pressure (rel)" which to me means "Barometer" unless
I'm completely misunderstanding.
My csv.conf is attached. I did end up moving the recorded pressures to
barometer in my final import. Things look correct, but I haven't done (and
likely won't do) a thorough check.
On Saturday, May 9, 2020 at 8:14:26 AM UTC-7, Ville Helander wrote:
>
> Hi Ryan,
>
> Sounds great! I tryet importing from csv file some time ago, but newer get
> it to work. I guess it was user error..
> Can you share more your configs how you did the import?
>
> -Ville-
>
>
>
> Ryan Stasel <rcst...@gmail.com <javascript:>> kirjoitti 8.5.2020 kello
> 21.48:
>
> Hi Ville,
>
> Just a heads up. I literally JUST imported 11 years worth of data out of
> WC into Weewx using a csv export from WC (5min intervals) and then using
> wee_import to import into weewx. Certainly not as clean as a native import,
> but it seems to have worked. Initially tested on my Pi4 and it took about 4
> hours to do the import of 1.2M rows. Did it a second time on a ubuntu VM
> and took about 1.5 hours. The import is definitely single threaded (or
> close). So I'd run on the fastest single core you have. =)
>
> On Wednesday, May 6, 2020 at 10:25:32 AM UTC-7, Ville Helander wrote:
>>
>> Ok,
>>
>> No problem with waiting, just wanted to hear somethink of WeatherCat
>> import state.
>> But it sound’s promising!
>>
>> -Ville-
>>
>>
>> gjr80 <gjrod...@gmail.com> kirjoitti 6.5.2020 kello 15.36:
>>
>> Hi Ville,
>>
>> My apologies that WeatherCat support for wee_import did not make it into
>> 4.0.0. However, I have started adding back the the WeatherCat code that
>> removed back in 2017. I see no reason why it will not appear in the next
>> release. I will certainly get you to do some testing, my sample WeatherCat
>> data is of somewhat dubious heritage.
>>
>> Gary
>>
>> On Wednesday, 6 May 2020 18:06:44 UTC+10, Ville Helander wrote:
>>>
>>> Hello Gary and all,
>>>
>>> WeeWX 4 is released, thank you for all the hard work everyone has made!!
>>> I see that import from WeatherCat is not included there and would like
>>> to know is there happening any progress for making import working?
>>> I can test the importing when there is somethink to test. :)
>>>
>>> Again, thanks for everyone who work with weewx!!
>>>
>>> -Ville-
>>>
>>>
>>> gjr80 <gjrod...@gmail.com> kirjoitti 18.11.2019 kello 0.20:
>>>
>>> Hi,
>>>
>>> No unfortunately it has not been included in b1. I will get to it,
>>> unfortunately other things have taken priority.
>>>
>>> Gary
>>>
>>> On Saturday, 16 November 2019 21:10:32 UTC+10, Ville Helander wrote:
>>>>
>>>> Hello Gary,
>>>>
>>>> WeeWX beta1 is just released, is this WeatherCat import available with
>>>> that beta1 release?
>>>>
>>>> -Ville-
>>>>
>>>>
>>>> > On 13. Oct 2019, at 0.06, gjr80 <gjrod...@gmail.com> wrote:
>>>> >
>>>> > Thank you, seems I did commit WeatherCat support back in April 2017
>>>> but it appears (for reasons unknown) to have been largely removed. I will
>>>> have a look at resurrecting it (it shouldn’t be too hard) though it was a
>>>> first cut so may still have some rough edges plus documentation is still
>>>> to
>>>> be updated. I will try to include it in the 4.0 release though I have a
>>>> few
>>>> other things to finish first plus free time is somewhat scarce at the
>>>> moment. Fortunately your WeatherCat data is not going anywhere :)
>>>> >
>>>> > Gary
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to a topic in
>>>> the Google Groups "weewx-user" group.
>>>> > To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
>>>> > To unsubscribe from this group and all its topics, send an email to
>>>> weewx...@googlegroups.com.
>>>> > To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/a9342c34-fcb5-4b6d-9aa1-faa43253f160%40googlegroups.com.
>>>>
>>>>
>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "weewx-user" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> weewx...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/f6f9db7a-d353-4343-a3c3-cbf5b8752f33%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/f6f9db7a-d353-4343-a3c3-cbf5b8752f33%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> weewx...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/ed4ade9e-e49e-4825-87df-98f61684917f%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/ed4ade9e-e49e-4825-87df-98f61684917f%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx...@googlegroups.com <javascript:>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/91a8a6bb-f324-43c4-a297-eb61e3085e84%40googlegroups.com
>
> <https://groups.google.com/d/msgid/weewx-user/91a8a6bb-f324-43c4-a297-eb61e3085e84%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/85440857-6bb5-4e2e-8763-6af44fd6f19e%40googlegroups.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES
#
# Copyright (c) 2009-2019 Tom Keffer <tkef...@gmail.com> and Gary Roderick.
# See the file LICENSE.txt for your rights.
##############################################################################
# Specify the source. Available options are:
# CSV - import obs from a single CSV format file
# WU - import obs from a Weather Underground PWS history
# Cumulus - import obs from a one or more Cumulus monthly log files
# WD - import obs from a one or more WD monthly log files
# Format is:
# source = (CSV | WU | Cumulus)
source = CSV
##############################################################################
[CSV]
# Parameters used when importing from a CSV file
# Path and name of our CSV source file. Format is:
# file = full path and filename
file = /home/pi/wc-alldata.csv
# The character used to separate fields. Format is:
# delimiter = <single character>
# Default is , (comma).
#delimiter = ,
# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
#
# Note: If there is a mapped interval field then this setting will be
# ignored.
# Format is:
# interval = (derive | conf | x)
interval = 5
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Specify how imported data fields that contain invalid data (eg a numeric
# field containing non-numeric data) are handled. Available options are:
# True - The invalid data is ignored, the WeeWX target field is set to
# None and the import continues.
# False - The import is halted.
# Format is:
# ignore_invalid_data = (True | False)
# Default is True.
ignore_invalid_data = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a CSV import UV_sensor should be set to False if a UV sensor was
# NOT present when the import data was created. Otherwise it may be set to
# True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = False
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a CSV import solar_sensor should be set to False if a solar radiation
# sensor was NOT present when the import data was created. Otherwise it may
# be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = False
# Date-time format of CSV field from which the WeeWX archive record
# dateTime field is to be extracted. wee_import first attempts to interpret
# date/time info in this format, if this fails it then attempts to
# interpret it as a timestamp and if this fails it then raises an error.
# Uses Python strptime() format codes.
# raw_datetime_format = Python strptime() format string
raw_datetime_format = %Y-%m-%dT%H:%M:%S
# Does the imported rain field represent the total rainfall since the last
# record or a cumulative value. Available options are:
# discrete - rain field represents total rainfall since last record
# cumulative - rain field represents a cumulative rainfall reset at
# midnight
# rain = (discrete | cumulative)
rain = cumulative
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the WeeWX limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
# where :
# lower is the lower limit of acceptable wind direction in degrees
# (may be negative)
# upper is the upper limit of acceptable wind direction in degrees
#
# Imported values from lower to upper will be normalised to the range 0 to
# 360. Values outside of the parameter range will be stored as None.
# Default is -360,360.
wind_direction = -360,360
# Map CSV record fields to WeeWX archive fields. Format is:
#
# weewx_archive_field_name = csv_field_name, weewx_unit_name
#
# where:
# weewx_archive_field_name - An observation name in the WeeWX database
# schema.
# csv_field_name - The name of a field from the CSV file.
# weewx_unit_name - The name of the units, as defined in WeeWX,
# used by csv_field_name. wee_import will do
# the necessary conversions to the unit system
# used by the WeeWX archive.
# For example,
# outTemp = Temp, degree_C
# would map the CSV field Temp, in degrees C, to the archive field outTemp.
#
# A mapping for WeeWX field dateTime is mandatory and the WeeWX unit name
# for the dateTime mapping must be unix_epoch. For example,
# dateTime = csv_date_and_time, unix_epoch
# would map the CSV field csv_date_and_time to the WeeWX dateTime field with
# the csv_date_and_time field being interpreted first using the format
# specified at the raw_datetime_format config option and if that fails as a
# unix epoch timestamp.
#
# Field mapping to the WeeWX usUnits archive field is currently not
# supported. If a usUnits field exists in the CSV data it should not be
# mapped, rather WeeWX unit names should included against each field to be
# imported in the field map.
#
# WeeWX archive fields that do not exist in the CSV data may be omitted.
# Any omitted fields that are derived (eg dewpoint) may be calculated
# during import using the equivalent of the WeeWX StdWXCalculate service
# through setting the calc-missing parameter above.
[[FieldMap]]
dateTime = Date, unix_epoch
interval =
barometer =
pressure = Pressure, inHg
altimeter =
inTemp = IntTemperature, degree_F
outTemp = Temperature, degree_F
inHumidity = IntHumidity, percent
outHumidity = ExtHumidity, percent
windSpeed = Wind, mile_per_hour
windDir = WindDirection, degree_compass
windGust = WindGust, mile_per_hour
windGustDir =
rainRate = PrecipRate, inch_per_hour
rain = Precipitation, inch
dewpoint = Dew, degree_F
windchill = WindChill, degree_F
heatindex =
ET =
radiation =
UV =
