A few thoughts. If it works sometimes and not others there must be a
problem with either the consistency of the format of the data or the
algorithm being used to parse the data. Can't say I am a big fan of slicing
in a situation like this, I think it can be a bit rigid at times. Maybe
consider using split. To to obtain your name, value pair maybe something
like:
...
for line in fd:
data = line.split("=")
if len(data) != 2:
continue
name,value = data
new_record_data[self.label_map.get(name, name)] = to_float(
value)
...
or cut out the middle man if you like:
...
for line in fd:
data = line.split("=")
if len(data) != 2:
continue
new_record_data[self.label_map.get(data[0], data[0])] =
to_float(data[1])
...
Gary
On Tuesday, 7 May 2019 17:05:27 UTC+10, Colin Larsen wrote:
>
> And this is the code the writes the txt file - capturing a string via UDP
>
> *#!/usr/bin/env python*
>
> *import* socket
>
> *from* shutil *import* copyfile
>
>
> UDP_IP = *"0.0.0.0"*
>
> UDP_PORT = 9886
>
> fname = *"/home/pi/AQIData.txt"*
>
>
> sock = socket.socket(socket.AF_INET,* # Internet*
>
> socket.SOCK_DGRAM)* # UDP*
>
> sock.bind((UDP_IP, UDP_PORT))
>
>
>
> *while* True:
>
>
> data, addr = sock.recvfrom(1024)* # buffer size is 1024 bytes*
>
> *print* *"Received message:"*, data
>
> dataSplit = data.split(*','*)
>
> f = open(fname,*'w'*)
>
> *for* ds *in* dataSplit:
>
> f.write(ds + *"\r"*)
>
> f.write(*'\n'*)
>
> f.close()
>
> On Tue, May 7, 2019 at 7:01 PM Colin Larsen <colin...@gmail.com
> <javascript:>> wrote:
>
>> Apparently #015 is a numeric representation of \r or 'return' to we can
>> rule that out?
>>
>> On Tue, May 7, 2019 at 6:56 PM Colin Larsen <colin...@gmail.com
>> <javascript:>> wrote:
>>
>>> Well this is a little strange. This went fine for a while then stopped
>>> again with pretty much the same error. It appears to be missing the
>>> beginning of the first line in the file which is AQI25 = then gets the
>>> reading which is 4.55
>>>
>>> invalid literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex =
>>> 18#015AQICO2 = 863
>>>
>>> Is the #015 a clue of any sorts?
>>>
>>> The file looks ok
>>>
>>> AQI25 = 4.55
>>>
>>> AQI100 = 9.09
>>>
>>> AQIIndex = 18
>>>
>>> AQICO2 = 863
>>>
>>>
>>> Error;
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Main loop exiting.
>>> Shutting engine down.
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: engine: Caught unrecoverable
>>> exception in engine:
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** invalid literal for
>>> float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Traceback (most
>>> recent call last):
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** engine.run()
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>> self._software_catchup()
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>> record=record, origin='software'))
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** callback(event)
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: ****
>>> new_record_data[self.label_map.get(name,
>>> name)] = to_float(value)
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** File
>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** return float(x)
>>> if x is not None else None
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** ValueError: invalid
>>> literal for float(): 4.55#015AQI100 = 9.09#015AQIIndex = 18#015AQICO2 = 863
>>>
>>> May 7 18:50:16 raspberrypi weewx[11960]: **** Exiting.
>>>
>>> On Tue, May 7, 2019 at 3:23 PM Colin Larsen <colin...@gmail.com
>>> <javascript:>> wrote:
>>>
>>>> Hi Thomas
>>>>
>>>> Thanks for the quick reply. It was just the format, it needs the spaces
>>>> so each line in the text file is
>>>>
>>>>
>>>> AQI25 = 1.60
>>>>
>>>> AQI100 = 3.30
>>>>
>>>> AQIIndex = 6
>>>>
>>>> AQICO2 = 683
>>>>
>>>>
>>>> Once I did that and checked the database after an archive period the
>>>> fields were populated
>>>>
>>>> Does my section with the units look valid? I've borrowed that from a
>>>> similar service but I can't wrap my head around how the relationship
>>>> between those and the data in the txt file get joined together
>>>>
>>>>
>>>> Thanks again
>>>>
>>>> Colin
>>>>
>>>>
>>>>
>>>> On Tue, May 7, 2019 at 3:13 PM Thomas Keffer <tke...@gmail.com
>>>> <javascript:>> wrote:
>>>>
>>>>> Hi, Colin
>>>>>
>>>>> Does your text file use newlines ('\n') as line delineators? Perhaps
>>>>> it is using the MS-DOS standard of '\n\r'? Examine the file carefully and
>>>>> make sure it contains what you think it contains.
>>>>>
>>>>> -tk
>>>>>
>>>>> On Mon, May 6, 2019 at 8:05 PM Colin Larsen <colin...@gmail.com
>>>>> <javascript:>> wrote:
>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> I'm trying to get Filepile working but have come across an error that
>>>>>> I don't understand. Any help appreciated. The database has been extended
>>>>>> with fields to match those below in the data file - am I just missing
>>>>>> spaces in the layout? Should it be AQI25 = 1.60 etc etc
>>>>>>
>>>>>> Many thanks
>>>>>>
>>>>>> This is my data file;
>>>>>>
>>>>>> AQI25=1.60
>>>>>>
>>>>>> AQI100=3.30
>>>>>>
>>>>>> AQIIndex=6
>>>>>>
>>>>>> AQICO2=683
>>>>>>
>>>>>>
>>>>>> This is my stanza in weewx.conf
>>>>>>
>>>>>>
>>>>>> ##############################################################################
>>>>>>
>>>>>>
>>>>>> # This section is for FilePile
>>>>>>
>>>>>>
>>>>>> [FilePile]
>>>>>>
>>>>>>
>>>>>> filename = /home/pi/AQIData.txt
>>>>>>
>>>>>> unit_system = METRIC
>>>>>>
>>>>>>
>>>>>>
>>>>>> ##############################################################################
>>>>>>
>>>>>>
>>>>>> This is my filepile.py (modified from original to suit)
>>>>>>
>>>>>>
>>>>>> *import* syslog
>>>>>>
>>>>>> *import* weewx
>>>>>>
>>>>>> *import* weewx.units
>>>>>>
>>>>>> *from* weewx.wxengine *import* StdService
>>>>>>
>>>>>> *from* weeutil.weeutil *import* to_float
>>>>>>
>>>>>>
>>>>>> weewx.units.USUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>
>>>>>> weewx.units.MetricUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>
>>>>>> weewx.units.MetricWXUnits[*'group_gas_concentration'*] = *'ppm'*
>>>>>>
>>>>>> weewx.units.default_unit_format_dict[*'ppm'*] = *'%.0f'*
>>>>>>
>>>>>> weewx.units.default_unit_label_dict[*'ppm'*] = *' ppm'*
>>>>>>
>>>>>>
>>>>>> weewx.units.USUnits[*'group_dust'*] = *'microgramm_per_meter_cubic'*
>>>>>>
>>>>>> weewx.units.MetricUnits[*'group_dust'*] =
>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>
>>>>>> weewx.units.MetricWXUnits[*'group_dust'*] =
>>>>>> *'microgramm_per_meter_cubic'*
>>>>>>
>>>>>> weewx.units.default_unit_format_dict[*'microgramm_per_meter_cubic'*]
>>>>>> = *'%.1f'*
>>>>>>
>>>>>> weewx.units.default_unit_label_dict[*'microgramm_per_meter_cubic'*] =
>>>>>> *' \xce\xbcg/m\xc2\xb3'*
>>>>>>
>>>>>>
>>>>>> *class* FilePile(StdService):
>>>>>>
>>>>>> *"""WeeWX service for augmenting a record with data parsed from
>>>>>> a file."""*
>>>>>>
>>>>>>
>>>>>> *def** __init__*(self, engine, config_dict):
>>>>>>
>>>>>> * # Initialize my superclass:*
>>>>>>
>>>>>> super(FilePile, self).__init__(engine, config_dict)
>>>>>>
>>>>>> * # Extract our stanza from the configuration dicdtionary*
>>>>>>
>>>>>> filepile_dict = config_dict.get(*'FilePile'*, {})
>>>>>>
>>>>>> * # Get the location of the file ...*
>>>>>>
>>>>>> self.filename = filepile_dict.get(*'filename'*,
>>>>>> *'/var/tmp/filepile.txt'*)
>>>>>>
>>>>>> * # ... and the unit system it will use*
>>>>>>
>>>>>> unit_system_name = filepile_dict.get(*'unit_system'*,
>>>>>> *'METRICWX'*).strip().upper()
>>>>>>
>>>>>> * # Make sure we know about the unit system. If not, raise an
>>>>>> exception.*
>>>>>>
>>>>>> *if* unit_system_name *not* *in* weewx.units.unit_constants:
>>>>>>
>>>>>> *raise* ValueError(*"FilePile: Unknown unit system: %s"*
>>>>>> % unit_system_name)
>>>>>>
>>>>>> * # Use the numeric code for the unit system*
>>>>>>
>>>>>> self.unit_system =
>>>>>> weewx.units.unit_constants[unit_system_name]
>>>>>>
>>>>>>
>>>>>> * # Mapping from variable names to weewx names*
>>>>>>
>>>>>> self.label_map = filepile_dict.get(*'label_map'*, {})
>>>>>>
>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Using %s with the
>>>>>> '%s' unit system"*
>>>>>>
>>>>>> % (self.filename, unit_system_name))
>>>>>>
>>>>>> syslog.syslog(syslog.LOG_INFO, *"filepile: Label map is %s"*
>>>>>> % self.label_map)
>>>>>>
>>>>>>
>>>>>> * # Bind to the NEW_ARCHIVE_RECORD event*
>>>>>>
>>>>>> self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)
>>>>>>
>>>>>>
>>>>>> *def** new_archive_record*(self, event):
>>>>>>
>>>>>> new_record_data = {}
>>>>>>
>>>>>> *try*:
>>>>>>
>>>>>> *with* open(self.filename, *'r'*) *as* fd:
>>>>>>
>>>>>> *for* line *in* fd:
>>>>>>
>>>>>> eq_index = line.find(*'='*)
>>>>>>
>>>>>> * # Ignore all lines that do not have an equal sign*
>>>>>>
>>>>>> *if* eq_index == -1:
>>>>>>
>>>>>> *continue*
>>>>>>
>>>>>> name = line[:eq_index].strip()
>>>>>>
>>>>>> value = line[eq_index + 1:].strip()
>>>>>>
>>>>>> new_record_data[self.label_map.get(name, name)]
>>>>>> = to_float(value)
>>>>>>
>>>>>> * # Supply a unit system if one wasn't included in
>>>>>> the file*
>>>>>>
>>>>>> *if* *'usUnits'* *not* *in* new_record_data:
>>>>>>
>>>>>> new_record_data[*'usUnits'*] = self.unit_system
>>>>>>
>>>>>> * # Convert the new values to the same unit system as
>>>>>> the record*
>>>>>>
>>>>>> target_data =
>>>>>> weewx.units.to_std_system(new_record_data, event.record[*'usUnits'*])
>>>>>>
>>>>>> * # Add the converted values to the record:*
>>>>>>
>>>>>> event.record.update(target_data)
>>>>>>
>>>>>> *except* IOError *as* e:
>>>>>>
>>>>>> syslog.syslog(syslog.LOG_ERR, *"FilePile: Cannot open file.
>>>>>> Reason: %s"* % e)
>>>>>>
>>>>>> This is the syslog error;
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: engine: Caught
>>>>>> unrecoverable exception in engine:
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** invalid literal
>>>>>> for float(): 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Traceback (most
>>>>>> recent call last):
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** engine.run()
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> callback(event)
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> self._software_catchup()
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>>>>>> record=record, origin='software'))
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> callback(event)
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/user/filepile.py", line 91, in new_archive_record
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: ****
>>>>>> new_record_data[self.label_map.get(name,
>>>>>> name)] = to_float(value)
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** File
>>>>>> "/usr/share/weewx/weeutil/weeutil.py", line 1280, in to_float
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** return
>>>>>> float(x) if x is not None else None
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** ValueError:
>>>>>> invalid literal for float():
>>>>>> 1.60#015AQI100=3.30#015AQIIndex=6#015AQICO2=683
>>>>>>
>>>>>> May 7 14:50:16 raspberrypi weewx[6605]: **** Exiting.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx...@googlegroups.com <javascript:>.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/weewx-user/CACjxfUvKW4goXJ9TZYUWkDm4YdxLqpeP_Ez1EzRheJyJ5_h4ew%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to weewx...@googlegroups.com <javascript:>.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEB0Tn%2B2q2sei37n%2BrTGO4pqq8cGL-5zF_E4dpMMEJzyiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/0f6edb07-8759-4ca4-8b91-d0745db22848%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
