Thanks Gary appreciate the thoughts, I'll try those out tomorrow
Colin
On Tue, 7 May 2019, 20:23 gjr80, <gjroder...@gmail.com> wrote:
> A few thoughts. If it works sometimes and not others there must be a
> problem with either the consistency of the format of the data or the
> algorithm being used to parse the data. Can't say I am a big fan of slicing
> in a situation like this, I think it can be a bit rigid at times. Maybe
> consider using split. To to obtain your name, value pair maybe something
> like:
>
> ...
> for line in fd:
> data = line.split("=")
> if len(data) != 2:
> continue
> name,value = data
> new_record_data[self.label_map.get(name, name)] = to_float
> (value)
> ...
>
> or cut out the middle man if you like:
>
> ...
> for line in fd:
> data = line.split("=")
> if len(data) != 2:
> continue
> new_record_data[self.label_map.get(data[0], data[0])] =
> to_float(data[1])
> ...
>
> Gary
>
> On Tuesday, 7 May 2019 17:05:27 UTC+10, Colin Larsen wrote:
>>
>> And this is the code the writes the txt file - capturing a string via UDP
>>
>> *#!/usr/bin/env python*
>>
>> *import* socket
>>
>> *from* shutil *import* copyfile
>>
>>
>> UDP_IP = *"0.0.0.0"*
>>
>> UDP_PORT = 9886
>>
>> fname = *"/home/pi/AQIData.txt"*
>>
>>
>> sock = socket.socket(socket.AF_INET,* # Internet*
>>
>> socket.SOCK_DGRAM)* # UDP*
>>
>> sock.bind((UDP_IP, UDP_PORT))
>>
>>
>>
>> *while* True:
>>
>>
>> data, addr = sock.recvfrom(1024)* # buffer size is 1024 bytes*
>>
>> *print* *"Received message:"*, data
>>
>> dataSplit = data.split(*','*)
>>
>> f = open(fname,*'w'*)
>>
>> *for* ds *in* dataSplit:
>>
>> f.write(ds + *"\r"*)
>>
>> f.write(*'\n'*)
>>
>> f.close()
>>
>> On Tue, May 7, 2019 at 7:01 PM Colin Larsen <colin...@gmail.com> wrote:
>>
>>> Apparently #015 is a numeric representation of \r or 'return' to we can
>>> rule that out?
>>>
>>> On Tue, May 7, 2019 at 6:56 PM Colin Larsen <colin...@gmail.com> wrote:
>>>
>>>> Well this is a little strange. This went fine for a while then stopped
>>>> again with pretty much the same error. It appears to be missing the
>>>> beginning of the first line in the file which is AQI25 = then gets the
>>>> reading which is 4.55
>>>>
>>>> invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex =
>>>> 18#015AQICO2 = 863
>>>>
>>>> Is the #015 a clue of any sorts?
>>>>
>>>> The file looks ok
>>>>
>>>> AQI25 = 4.55
>>>>
>>>> AQI100 = 9.09
>>>>
>>>> AQIIndex = 18
>>>>
>>>> AQICO2 = 863
>>>>
>>>>
>>>> Error;
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Main loop exiting.
>>>> Shutting engine down.
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Caught unrecoverable
>>>> exception in engine:
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** invalid literal
>>>> for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Traceback (most
>>>> recent call last):
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** engine.run()
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>> self._software_catchup()
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>>> record=record, origin='software'))
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>>> new_record_data[self.label_map.get(name,
>>>> name)] = to_float(value)
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** return
>>>> float(x) if x is not None else None
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** ValueError:
>>>> invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex =
>>>> 18#015AQICO2 = 863
>>>>
>>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Exiting.
>>>>
>>>> On Tue, May 7, 2019 at 3:23 PM Colin Larsen <colin...@gmail.com> wrote:
>>>>
>>>>> Hi Thomas
>>>>>
>>>>> Thanks for the quick reply. It was just the format, it needs the
>>>>> spaces so each line in the text file is
>>>>>
>>>>>
>>>>> AQI25 = 1.60
>>>>>
>>>>> AQI100 = 3.30
>>>>>
>>>>> AQIIndex = 6
>>>>>
>>>>> AQICO2 = 683
>>>>>
>>>>>
>>>>> Once I did that and checked the database after an archive period the
>>>>> fields were populated
>>>>>
>>>>> Does my section with the units look valid? I've borrowed that from a
>>>>> similar service but I can't wrap my head around how the relationship
>>>>> between those and the data in the txt file get joined together
>>>>>
>>>>>
>>>>> Thanks again
>>>>>
>>>>> Colin
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 7, 2019 at 3:13 PM Thomas Keffer <tke...@gmail.com> wrote:
>>>>>
>>>>>> Hi, Colin
>>>>>>
>>>>>> Does your text file use newlines ('\n') as line delineators? Perhaps
>>>>>> it is using the MS-DOS standard of '\n\r'? Examine the file carefully and
>>>>>> make sure it contains what you think it contains.
>>>>>>
>>>>>> -tk
>>>>>>
>>>>>> On Mon, May 6, 2019 at 8:05 PM Colin Larsen <colin...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all
>>>>>>>
>>>>>>> I'm trying to get Filepile working but have come across an error
>>>>>>> that I don't understand. Any help appreciated. The database has been
>>>>>>> extended with fields to match those below in the data file - am I just
>>>>>>> missing spaces in the layout? Should it be AQI25 = 1.60 etc etc
>>>>>>>
>>>>>>> Many thanks
>>>>>>>
>>>>>>> This is my data file;
>>>>>>>
>>>>>>> AQI25=1.60
>>>>>>>
>>>>>>> AQI100=3.30
>>>>>>>
>>>>>>> AQIIndex=6
>>>>>>>
>>>>>>> AQICO2=683
>>>>>>>
>>>>>>>
>>>>>>> This is my stanza in weewx.conf
>>>>>>>
>>>>>>>
>>>>>>> ##############################################################################
>>>>>>>
>>>>>>>
>>>>>>> # This section is for FilePile
>>>>>>>
>>>>>>>
>>>>>>> [FilePile]
>>>>>>>
>>>>>>>
>>>>>>> filename = /home/pi/AQIData.txt
>>>>>>>
>>>>>>> unit_system = METRIC
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ##############################################################################
>>>>>>>
>>>>>>>
>>>>>>> This is my filepile.py (modified from original to suit)
>>>>>>>
>>>>>>>
>>>>>>> *import* syslog
>>>>>>>
>>>>>>> *import* weewx
>>>>>>>
>>>>>>> *import* weewx.units
>>>>>>>
>>>>>>> *from* weewx.wxengine *import* StdService
>>>>>>>
>>>>>>> *from* weeutil.weeutil *import* to_float
>>>>>>>
>>>>>>>
>>>>>>> weewx.units.USUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>
>>>>>>> weewx.units.MetricUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>
>>>>>>> weewx.units.MetricWXUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>>
>>>>>>> weewx.units.default_unit_format_dict[*'ppm'*] = *'%.0f'*
>>>>>>>
>>>>>>> weewx.units.default_unit_label_dict[*'ppm'*] = *' ppm'*
>>>>>>>
>>>>>>>
>>>>>>> weewx.units.USUnits[*'group_dust'*] = *'microgramm_per_meter_cubic'*
>>>>>>>
>>>>>>> weewx.units.MetricUnits[*'group_dust'*] =
>>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>>
>>>>>>> weewx.units.MetricWXUnits[*'group_dust'*] =
>>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>>
>>>>>>> weewx.units.default_unit_format_dict[*'microgramm_per_meter_cubic'*]
>>>>>>> = *'%.1f'*
>>>>>>>
>>>>>>> weewx.units.default_unit_label_dict[*'microgramm_per_meter_cubic'*]
>>>>>>> = *' \xce\xbcg/m\xc2\xb3'*
>>>>>>>
>>>>>>>
>>>>>>> *class* FilePile(StdService):
>>>>>>>
>>>>>>> *"""WeeWX service for augmenting a record with data parsed from
>>>>>>> a file."""*
>>>>>>>
>>>>>>>
>>>>>>> *def** __init__*(self, engine, config_dict):
>>>>>>>
>>>>>>> * # Initialize my superclass:*
>>>>>>>
>>>>>>> super(FilePile, self).__init__(engine, config_dict)
>>>>>>>
>>>>>>> * # Extract our stanza from the configuration dicdtionary*
>>>>>>>
>>>>>>> filepile_dict = config_dict.get(*'FilePile'*, {})
>>>>>>>
>>>>>>> * # Get the location of the file ...*
>>>>>>>
>>>>>>> self.filename = filepile_dict.get(*'filename'*,
>>>>>>> *'/var/tmp/filepile.txt'*)
>>>>>>>
>>>>>>> * # ... and the unit system it will use*
>>>>>>>
>>>>>>> unit_system_name = filepile_dict.get(*'unit_system'*,
>>>>>>> *'METRICWX'*).strip().upper()
>>>>>>>
>>>>>>> * # Make sure we know about the unit system. If not, raise
>>>>>>> an exception.*
>>>>>>>
>>>>>>> *if* unit_system_name *not* *in* weewx.units.unit_constants:
>>>>>>>
>>>>>>> *raise* ValueError(*"FilePile: Unknown unit system: %s"*
>>>>>>> % unit_system_name)
>>>>>>>
>>>>>>> * # Use the numeric code for the unit system*
>>>>>>>
>>>>>>> self.unit_system =
>>>>>>> weewx.units.unit_constants[unit_system_name]
>>>>>>>
>>>>>>>
>>>>>>> * # Mapping from variable names to weewx names*
>>>>>>>
>>>>>>> self.label_map = filepile_dict.get(*'label_map'*, {})
>>>>>>>
>>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Using %s with
>>>>>>> the '%s' unit system"*
>>>>>>>
>>>>>>> % (self.filename, unit_system_name))
>>>>>>>
>>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Label map is %s"*
>>>>>>> % self.label_map)
>>>>>>>
>>>>>>>
>>>>>>> * # Bind to the NEW_ARCHIVE_RECORD event*
>>>>>>>
>>>>>>> self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)
>>>>>>>
>>>>>>>
>>>>>>> *def** new_archive_record*(self, event):
>>>>>>>
>>>>>>> new_record_data = {}
>>>>>>>
>>>>>>> *try*:
>>>>>>>
>>>>>>> *with* open(self.filename, *'r'*) *as* fd:
>>>>>>>
>>>>>>> *for* line *in* fd:
>>>>>>>
>>>>>>> eq_index = line.find(*'='*)
>>>>>>>
>>>>>>> * # Ignore all lines that do not have an equal sign*
>>>>>>>
>>>>>>> *if* eq_index == -1:
>>>>>>>
>>>>>>> *continue*
>>>>>>>
>>>>>>> name = line[:eq_index].strip()
>>>>>>>
>>>>>>> value = line[eq_index + 1:].strip()
>>>>>>>
>>>>>>> new_record_data[self.label_map.get(name, name)]
>>>>>>> = to_float(value)
>>>>>>>
>>>>>>> * # Supply a unit system if one wasn't included in
>>>>>>> the file*
>>>>>>>
>>>>>>> *if* *'usUnits'* *not* *in* new_record_data:
>>>>>>>
>>>>>>> new_record_data[*'usUnits'*] = self.unit_system
>>>>>>>
>>>>>>> * # Convert the new values to the same unit system
>>>>>>> as the record*
>>>>>>>
>>>>>>> target_data =
>>>>>>> weewx.units.to_std_system(new_record_data, event.record[*'usUnits'*
>>>>>>> ])
>>>>>>>
>>>>>>> * # Add the converted values to the record:*
>>>>>>>
>>>>>>> event.record.update(target_data)
>>>>>>>
>>>>>>> *except* IOError *as* e:
>>>>>>>
>>>>>>> syslog.syslog(syslog.LOG_ERR, *"FilePile: Cannot open file.
>>>>>>> Reason: %s"* % e)
>>>>>>>
>>>>>>> This is the syslog error;
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: engine: Caught
>>>>>>> unrecoverable exception in engine:
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** invalid literal
>>>>>>> for float(): 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Traceback (most
>>>>>>> recent call last):
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** engine.run()
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> callback(event)
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> self._software_catchup()
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>>>>>> record=record, origin='software'))
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> callback(event)
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>>> new_record_data[self.label_map.get(name,
>>>>>>> name)] = to_float(value)
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** return
>>>>>>> float(x) if x is not None else None
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** ValueError:
>>>>>>> invalid literal for float():
>>>>>>> 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>>
>>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Exiting.
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "weewx-user" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to weewx...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/CACjxfUtAX4SeZxkJWPZdtOSKFAOVaZY6-xJhUPmaAmTth5ATDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
