Thank you for your enthusiastic guidance. I have learned a lot. As you said, we all like pyqtgraph and think it is a perfect productivity tool, so many of my projects are done using pyqtgraph. I checked through Google All the professional tools and modules you mentioned, but they don’t seem to meet my expectations. And I only know python, and I am not very good at English. Forum posts are all with the help of Google Translate, so I learn the use of new tools or new Technology is more difficult for me, I think you can confirm for me: memory usage is objective and cannot be solved, right? If only pyqt5 and pyqtgraph are used?
在2020年11月27日星期五 UTC+8 上午9:08:31<Patrick> 写道： > Hi, > > I know these are the pyqtgraph forums, and we all love pyqtgraph here, but > have you looked at some other options? For example, a dedicated time series > database (InfluxDB <https://www.influxdata.com/>, Prometheus > <https://prometheus.io/>) and Grafana <https://grafana.com/> for the > plotting interface. > I've used InfluxDB and Grafana with a home-made temperature logger which > communicates over a serial port. The Grafana web interface lets you select > time and date ranges and zoom into the regions of data. It's accessible > over the network and works really well! > You'll still need to write some code to actually log your data and put it > into the database, but that's relatively easy. The program I wrote got a > little out of hand, and I ended up releasing it publicly here > <https://datalogd.readthedocs.io/en/latest/>. You'd just need to write > your own data source plugin. Even if you don't use that framework there's a > lot of instructions for setting up the database etc. > > If you want to stay with writing your own plotting program, then what > Ognyan is saying above about resampling your data is correct. Trying to > load 1+ GB of raw data and plot all of it will be slow and run into memory > limits. You need to load only specific data from disk depending on the zoom > level and range of your plot. As your view changes, you'll need to load > different parts of the data from disk (this is what Grafana does when > accessing its database). In this case a MongoDB database might not be a > good choice, and why a time series database would be better. Another > alternative that might work is a memory-mapped array --- I was looking at > using zarr <https://zarr.readthedocs.io/en/stable/> for a project I'm > working on at the moment where the size of the collected data may be larger > than the computer's RAM size. > > Patrick > > On Friday, 27 November 2020 at 10:56:57 am UTC+10:30 [email protected] > wrote: > >> I want to do some experimentation, can you give the shape of the array >> you're plotting? Not sure it helps with memory issue but you can also try >> using pyqtgraph's plotting downsample methods. >> >> On Thu, Nov 26, 2020 at 4:23 PM xiangyu ning <[email protected]> wrote: >> >>> 谢谢你的回答,我的项目基于python >>> 实现一个直流分析仪的功能,所以需要呈现所有的数据,然后通过鼠标拖拽的方式使用"LinearRegionItem"功能去选择感兴趣的区间,然后进一步分析区间内的数据.另外我也降低我的采样率,但是这只能缓解内存占用过大的问题,而不能从根本上解决,实验的时间延长以后,还是会出现内存不足的情况,另外如楼上所说,内存复制的机制是QT底层的原因还是pyqtgraph的原因呢,选择python实现是因为python模块很多,避免了重复制造轮子的问题,但是如果内存不够的问题解决不了,我能选用QT直接去做吗?会有这个问题吗? >>> Thank you for your answer. My project is based on python to implement >>> the function of a DC analyzer, so all the data needs to be presented, and >>> then use the "LinearRegionItem" function to select the interval of interest >>> by dragging the mouse, and then further analyze the interval Data. In >>> addition, I also reduce my sampling rate, but this can only alleviate the >>> problem of excessive memory usage, but cannot solve it fundamentally. After >>> the experiment time is extended, there will still be insufficient memory. >>> In addition, as mentioned above , The mechanism of memory copy is the >>> underlying reason of QT or the reason of pyqtgraph. The python >>> implementation is chosen because there are many python modules, which >>> avoids the problem of repetitive manufacturing of wheels, but if the >>> problem of insufficient memory cannot be solved, can I use QT to do it >>> directly? ? Will there be this problem? >>> 在2020年11月27日星期五 UTC+8 上午1:49:03<[email protected]> 写道： >>> >>>> When dealing with lots of data like that, consider if you need to plot >>>> all that data, or if you can subsample to get the total data down to >>>> something more manageable. >>>> >>>> On Thu, Nov 26, 2020 at 9:12 AM xiangyu ning <[email protected]> wrote: >>>> >>>>> 谢谢你,请问你后来是怎么解决问题的?我现在的情况是mongdb里面有1.5GB的数据,如果使用pyqtgraph全部显示在界面上,会占用我电脑90%的内存,约超过90%*24GB >>>>> >>>>> =21.6GB的内存,这太恐怖了,而且这仅仅是三天的数据量,我的计划是挂测10天以上,这样数据,可能达到十几个GB,如果目前来看,这个想法不能实现 >>>>> Thank you, how did you solve the problem later? My current situation >>>>> is that there is 1.5GB of data in mongdb. If you use pyqtgraph to display >>>>> all on the interface, it will occupy 90% of my computer's memory, about >>>>> 90%*24GB = 21.6GB of memory, which is horrible, and this is only three >>>>> days >>>>> of data. My plan is to test for more than 10 days, so the data may reach >>>>> more than a dozen GB. If you look at it now, this idea cannot be realized. >>>>> >>>>> 在2020年11月26日星期四 UTC+8 下午11:57:18<Magne Lauritzen> 写道： >>>>> >>>>>> Also remember that when you provide data to plot, internal copies are >>>>>> made. So plotting 1GB of data will result if several GB of memory being >>>>>> consumed. This has previously caused problems for me. >>>>>> >>>>>> On Thu, Nov 26, 2020, 15:14 xiangyu ning <[email protected]> wrote: >>>>>> >>>>>>> >>>>>>> Thank you for your answer, because it takes a few days to get a >>>>>>> large amount of data, so I have modified the code to test according to >>>>>>> your >>>>>>> suggestions, and the results may be produced in a few days >>>>>>> >>>>>> 在2020年11月26日星期四 UTC+8 上午10:14:36<Patrick> 写道： >>>>>>> >>>>>> Hi, >>>>>>>> >>>>>>>> I'm guessing you are calling self.plotWidget.plot(x, y) a lot, >>>>>>>> which creates a new plot item each time. You should instead make one >>>>>>>> plot, >>>>>>>> and then use .setData(x, y) to update the existing plot with the new >>>>>>>> data. >>>>>>>> >>>>>>>> self.plot = self.plotWidget.plot() >>>>>>>> # .... later, when data is received >>>>>>>> self.plot.setData(x, y) >>>>>>>> >>>>>>>> Patrick >>>>>>>> >>>>>>>> On Thursday, 26 November 2020 at 12:07:46 pm UTC+10:30 >>>>>>>> [email protected] wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <https://fanyi.baidu.com/translate?aldtype=16047&query=%E4%B8%BA%E4%BB%80%E4%B9%88%E4%BD%BF%E7%94%A8pyqtgrph%E7%94%BB%E5%9B%BE%E7%9A%84%E6%97%B6%E5%80%99%2C%E7%94%B5%E8%84%91%E7%9A%84%E5%86%85%E5%AD%98%E4%BC%9A%E9%9A%8F%E7%9D%80%E6%95%B0%E6%8D%AE%E9%87%8F%E7%9A%84%E5%A2%9E%E5%8A%A0%E8%80%8C%E5%A2%9E%E5%8A%A0.%3F&keyfrom=baidu&smartresult=dict&lang=auto2zh#%23%23> >>>>>>>>> 最近使用pyqtgraph遇到问题了,问题表现在,我写了个上位机记录单片机通过串口发给电脑的数据,并将数据存入数据库,问题就在于当我用上位机从数据库中读取保存的数据,并且用pyqtgraph显示出来的时候,随着数据量的增多,电脑内存占用会逐渐加大,直到最后爆掉,我试过屏蔽掉self.plotWidget_ted.plot(y=b, >>>>>>>>> >>>>>>>>> x=a, >>>>>>>>> pen='b')这句话,也就是说不画图,内存就不会有变化了,请问是什么原因导致的呢,是画图的时候,把那些点写入到系统内存里了吗,我上位机在读取数据的时候是循环读取,循环加载,加载完成,会清除掉已经读取的数据, >>>>>>>>> >>>>>>>>> <https://fanyi.baidu.com/translate?aldtype=16047&query=%E4%B8%BA%E4%BB%80%E4%B9%88%E4%BD%BF%E7%94%A8pyqtgrph%E7%94%BB%E5%9B%BE%E7%9A%84%E6%97%B6%E5%80%99%2C%E7%94%B5%E8%84%91%E7%9A%84%E5%86%85%E5%AD%98%E4%BC%9A%E9%9A%8F%E7%9D%80%E6%95%B0%E6%8D%AE%E9%87%8F%E7%9A%84%E5%A2%9E%E5%8A%A0%E8%80%8C%E5%A2%9E%E5%8A%A0.%3F&keyfrom=baidu&smartresult=dict&lang=auto2zh#%23%23> >>>>>>>>> >>>>>>>>> Recently, I encountered a problem in using pyqtgraph. The problem >>>>>>>>> is that I wrote a PC to record the data sent to the computer by the >>>>>>>>> MCU >>>>>>>>> through the serial port, and stored the data in the database. The >>>>>>>>> problem >>>>>>>>> is that when I use the upper computer to read the saved data from the >>>>>>>>> database and display it with pyqtgraph, with the increase of the >>>>>>>>> amount of >>>>>>>>> data, the computer memory occupation will gradually increase, until >>>>>>>>> it >>>>>>>>> finally bursts, I tried to block it self.plotWidget_ ted.plot (y = B, >>>>>>>>> x = >>>>>>>>> a, pen'b '), that is to say, without drawing, the memory will not be >>>>>>>>> changed. What is the reason for that? When drawing, did those points >>>>>>>>> be >>>>>>>>> written into the system memory? When reading the data, my upper >>>>>>>>> computer >>>>>>>>> would read the data circularly. After loading, the data would be >>>>>>>>> cleared, >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "pyqtgraph" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/pyqtgraph/5d5aac83-10c2-4235-b016-41e0b347aec4n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/pyqtgraph/5d5aac83-10c2-4235-b016-41e0b347aec4n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "pyqtgraph" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/pyqtgraph/8432922a-2ccd-4e36-b554-ce17d6a2ca11n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/pyqtgraph/8432922a-2ccd-4e36-b554-ce17d6a2ca11n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pyqtgraph" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pyqtgraph/9cf303c9-487e-4479-94c5-4abe032806ean%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/pyqtgraph/9cf303c9-487e-4479-94c5-4abe032806ean%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "pyqtgraph" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/ea8cb43f-55fd-4c29-9fcd-1fd0ef74a2edn%40googlegroups.com.
