Re: Using threads for audio computing?
Le 12/05/14 07:58, Chris Angelico a écrit : Well, the first thing I'd try is simply asking for more data when you're ready for it - can you get five seconds' of data all at once? Obviously this won't work if your upstream buffers only a small amount, in which case your thread is there to do that buffering; also, if you can't absolutely *guarantee* that you can process the data quickly enough, every time, then you need to use the queue to buffer that. But otherwise, it sounds like a quite reasonable way to do things. ChrisA Ok, thanks a lot! -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
Le 12/05/14 08:13, Stefan Behnel a écrit : This sounds like a use case for double buffering. Use two buffers, start filling one. When it's full, switch buffers, start filling the second and process the first. When the second is full, switch again. Note that you have to make sure that the processing always terminates within the time it takes to fill the other buffer. If you can't assure that, however, you have a problem anyway and should see if there's a way to improve your algorithm. If the fill my buffer call in PyAudio is blocking (i.e. if it returns only after filling the buffer), then you definitely need two threads for this. But AFAIK the python GIL (and in smaller or older computers that have only one core) does not permit true paralell execution of two threads. Not for code that runs in the *interpreter, but it certainly allows I/O and low-level NumPy array processing to happen in parallel, as they do not need the interpreter. Stefan Thanks for your answer. If I follow your explanations, I guess I have to review my understanding of python execution model (I have to admit it is quite crude anyway). In my understanding, without threads, I would have two functions: - get_audio() would get the 5 seconds of audio from Pyaudio - process_audio() would process the 5 seconds of audio the main code would be roughly executing this: while(True) get_audio() process_audio() so since the audio is a live feed (which makes a difference, say, with an audio file analyser program), the get_audio() part must take 5 seconds to execute. (but most probably the processor stays still most of the time during the get_audio() part). then once get_audio() is done, process_audio() begins. Process_audio will take some time. If that time is greater that the times it takes for the next audio sample to arrive, I have a problem. (which you already explained differently maybe with: If the fill my buffer call in PyAudio is blocking (i.e. if it returns only after filling the buffer), then you definitely need two threads for this. ) So if I follow you, if the Pyaudio part is Non-blocking there would be a way to make it work without the two threads things. I'm back to the Pyaudio doc, and try to get my head around the callback method, which might be the good lead. -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
Le 12/05/14 10:14, lgabiot a écrit : So if I follow you, if the Pyaudio part is Non-blocking there would be a way to make it work without the two threads things. I'm back to the Pyaudio doc, and try to get my head around the callback method, which might be the good lead. So far, if I understand correctly PyAudio, the callback method is a way to do some sort of computing on a Pyaudio stream, by declaring a function (the callback one) at stream opening time, the callback function being executed in a separate thread (as per the Pyaudio documentation)... Still investigating. -- https://mail.python.org/mailman/listinfo/python-list
Using threads for audio computing?
Hello, I'd like to be able to analyze incoming audio from a sound card using Python, and I'm trying to establish a correct architecture for this. Getting the audio is OK (using PyAudio), as well as the calculations needed, so won't be discussing those, but the general idea of being able at (roughly) the same time: getting audio, and performing calculation on it, while not loosing any incoming audio. I also make the assumption that my calculations on audio will be done faster than the time I need to get the audio itself, so that the application would be almost real time. So far my idea (which works according to the small tests I did) consist of using a Queue object as a buffer for the incoming audio and two threads, one to feed the queue, the other to consume it. The queue could store the audio as a collection of numpy array of x samples. The first thread work would be to put() into the queue new chunks of audio as they are received from the audio card, while the second would get() from the queue chunks and perform the necessary calculations on them. Am I in the right direction, or is there a better general idea? Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
Le 11/05/14 16:40, Roy Smith a écrit : In article 536f869c$0$2178$426a7...@news.free.fr, lgabiot lgab...@hotmail.com wrote: Hello, Le 11/05/14 16:40, Roy Smith a écrit : If you are going to use threads, the architecture you describe seems perfectly reasonable. It's a classic producer-consumer pattern. But, I wonder if you even need anything this complicated. Using a queue to buffer work between threads makes sense if the workload presented is uneven. Sometimes you'll get a burst of work all at once and don't have the capacity to process it in real-time, so you want to buffer it up. I would think sampling audio would be a steady stream. Every x ms, you get another chunk of samples, like clockwork. Is this not the case? Thanks for your answer, yes, I guess I can consider audio as a steady stream. PyAudio gives me the audio samples by small chunks (2048 samples at a time for instance, while the sound card gives 48 000 samples/seconds). I accumulate the samples into a numpy array, and once the numpy array has reached the needed size (for instance 5 seconds of audio), I put this numpy array in the queue. So I think you are right in thinking that every 5 seconds I get a new chunk of audio to work on. Then I perform a calculation on this 5 seconds of audio (which needs to be done in less than 5 seconds, so that it will be ready to process the next 5 second chunk), but meanwhile, I need to still constantly get from Pyaudio a new 5 second chunk of audio. Hence my system. I guess if my calculation had to be performed on a small number of samples (i.e. under the value of the Pyaudio buffer size (2048 samples for instance), and that the calculation would last less than the time it takes to get the next 2048 samples from Pyaudio, I wouldn't need the Queue and Thread system. But in my case where I need a large buffer, it might not work? Unless I ask pyaudio to feed me directly with 5 seconds chunks (instead of the usual buffer sizes: 1024, 2048, etc...), which I didn't try, because I hadn't though of it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
Le 11/05/14 17:40, lgabiot a écrit : I guess if my calculation had to be performed on a small number of samples (i.e. under the value of the Pyaudio buffer size (2048 samples for instance), and that the calculation would last less than the time it takes to get the next 2048 samples from Pyaudio, I wouldn't need the Queue and Thread system. But in my case where I need a large buffer, it might not work? Unless I ask pyaudio to feed me directly with 5 seconds chunks (instead of the usual buffer sizes: 1024, 2048, etc...), which I didn't try, because I hadn't though of it. I guess this solution might probably not work, since it would mean that the calculation should be quick enough so it wouldn't last longer than 1 sample (1/48000 s for instance), since while doing the calculation, no audio would be ingested (unless pyAudio possess some kind of internal concurrency system). Which leads me to think that a buffer (queue) and separate threads (producer and consumer) are necessary for this task. But AFAIK the python GIL (and in smaller or older computers that have only one core) does not permit true paralell execution of two threads. I believe it is quite like the way multiple processes are handled by an OS on a single CPU computer: process A has x CPU cycles, then process B has y CPU cycles, etc... So in my case, I must have a way to make sure that: thread 1 (which gets audio from Pyaudio and put() it in the Queue) is not interrupted long enough to miss a sample. If I suppose a worst case scenario for the computer, like a raspberry-pi, the CPU speed is 700MHz, which gives approx 14 000 CPU cycles between each audio samples (at 48 kHz FS). I don't know if 14 000 CPU cycle is a lot or not for the tasks at hands. Well, at least, it is what I understand, but since I'm really both a beginner and an hobbyist, I might be totally wrong... -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
Le 12/05/14 07:41, Chris Angelico a écrit : The GIL is almost completely insignificant here. One of your threads will be blocked practically the whole time (waiting for more samples; collecting them into a numpy array doesn't take long), and the other is, if I understand correctly, spending most of its time inside numpy, which releases the GIL. You should be able to thread just fine. ChrisA Thanks Chris for your answer. So back to my original question: A Queue and two threads (producer/consumer) seems a good answer to my problem, or is there a better way to solve it? (again, I'm really a beginner, so I made up this solution, but really wonder if I do not miss a well known obvious much better idea). -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get sqlite3.Row working: keyword lookup doesn't work
Le 26/01/14 09:05, Peter Otten a écrit : Please remember to cut and past the traceback next time. What is wrong? My crystal ball says that you have a from __future__ import unicode_literals statement at the beginning of the module. If I'm right try row[bfilename] Thanks a lot for your answer! your crystal ball was completely right, indeed I use the __future__ import, and the b'' fixed everything. Three days I was trying to get this... -- https://mail.python.org/mailman/listinfo/python-list
Can't get sqlite3.Row working: keyword lookup doesn't work
Hello, using Python 2.7.6 I try to access a sqlite database using keyword lookup instead of position (much more easy to maintain code), but it always fail, with the error: Index must be int or string I have created the database, populated it, and here is the code that tries to retrieve the information: with sqlite3.connect(mydbPath) as db: # open the database db.row_factory = sqlite3.Row cursor = db.cursor() cursor.execute(SELECT * FROM files) for row in cursor.fetchall(): print(row.keys()) print(row[filename]) result is: ['filename', 'filepath', 'filetag', 'PROJECT', 'SCENE', 'TAKE', 'TAPE', 'CIRCLED', 'FILE_UID', 'UBITS', 'TOTAL_FILES', 'FAMILY_UID', 'track_1', 'track_2', 'track_3', 'track_4', 'track_5', 'track_6', 'track_7', 'track_8', 'track_9', 'track_10', 'track_11', 'track_12', 'NOTE', 'duration', 'BWF_ORIGINATION_DATE', 'TIMECODE_FLAG', 'TIMECODE_RATE', 'FILE_SAMPLE_RATE', 'AUDIO_BIT_DEPTH', 'DIGITIZER_SAMPLE_RATE', 'TIMESTAMP_SAMPLE_RATE', 'TIMESTAMP_SINCE_MIDNIGHT', 'is_Short', 'is_MS', 'is_renamed_MS', 'WF_created', 'max_level', 'is_silent', 'is_silent_moved', 'silent_path', 'is_WS', 'is_WS_copied', 'CSV_made', 'is_cantar', 'is_sound_devices', 'exist'] error = Index must be int or string What is wrong? thanks a lot. -- https://mail.python.org/mailman/listinfo/python-list
Re: sqlite3 docbug (was problem with sqlite3)
Le 23/01/14 10:04, Mark Lawrence a écrit : No, you need to remember how to type xyz into your favourite search engine. For this case xyz would be something like python single element tuple. No big deal, but I don't think you are correct. Problem was that for me I knew (it was erroneous of course) that (element) was a python single element tuple... so there was no need for me to look for something I knew. Once I understood that what I knew was wrong (that is after reading the answers to my first post), I did type xyz in my favourite search engine, which led me to the link I posted in my answer... -- https://mail.python.org/mailman/listinfo/python-list
Using a static library in a C extension for Python
Hello, working on OS X 10.8.5 Python 2.7 I've written a simple C extension for Python that uses the cairo graphic library. It works well, and I can call it from Python with no problem. The only drawback is that I need to have the cairo library installed on my system (so it seems my extension calls dynamically the cairo library). I believe this because it doesn't work on a system where cairo is not installed. Is it possible to link statically cairo to my extension, so that even if cairo is not installed on a computer, the code will run? I guess I would need to modify the setup.py file using distutils to compile cairo statically into my .so file? Or am I completely wrong? If someone has a hint... thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: Using a static library in a C extension for Python
Le 22/01/14 18:31, 8 Dihedral a écrit : Check the C source code generated by Pyrex and check cython for what u want, but I did try that out in any mobile phone or flat panel programming. Thanks a lot for your answer. I didn't use Pyrex or other tool, but wrote myself the C python wrapping, using the Python C/API documentation (http://docs.python.org/2/c-api/). I then used the distutils tool (via a setup.py file) to build my extension. While there is several function in my C code, only one needed to be accessed by python. I'll check Cython then, but is there any tweaking of the setup.py file using distutils that will help me compile the extension with the cairo lib embedded into it? -- https://mail.python.org/mailman/listinfo/python-list
Re: Using a static library in a C extension for Python
Le 22/01/14 23:09, Gregory Ewing a écrit : We suspect that 8 Dihedral is actually a bot, so you're *probably* wasting your time attempting to engage it in conversation. Thanks, so that will be my first real experience of the Turing test!!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Using a static library in a C extension for Python
thanks a lot for your very precise answer! shortly, as I'm running out of time right now: I've got here a lot of informations, so I'll dig in the directions you gave me. It will be a good compiling exercise... (I'm really new at all this). -- https://mail.python.org/mailman/listinfo/python-list
problem with sqlite3: cannot use in a SQL query with (?)
Hello, I'm building an application using a simple sqlite3 database. At some point, I need to select rows (more precisely some fields in rows) that have the following property: their field max_level (an INT), should not exceed a value stored in a variable called threshold, where an int is stored (value 2). (threshold needs to be set by the user, so I cannot hard code a value there). My database already exist on my drive (and of course the sqlite3 module is imported) I do the following: conn = sqlite3.connect(mydb) # open the database that's OK cursor = conn.execute(SELECT filename, filepath FROM files WHERE max_level(?), threshold) that doesn't work (throw an exception) if I do: cursor = conn.execute(SELECT filename, filepath FROM files WHERE max_level2)) it works... I did similar operations on UPDATE instead of SELECT, and it works there. Maybe my mind is fried right now, but I can't figure out the solution... best regards. -- https://mail.python.org/mailman/listinfo/python-list
Re: problem with sqlite3: cannot use in a SQL query with (?)
I did similar operations on UPDATE instead of SELECT, and it works there. Maybe my mind is fried right now, but I can't figure out the solution... so maybe I should rename my post: cannot use =, with (?) in SELECT WHERE query ? -- https://mail.python.org/mailman/listinfo/python-list
Re: sqlite3 docbug (was problem with sqlite3)
Thanks to all, that was indeed the tuple issue! the correct code is: cursor = conn.execute(SELECT filename, filepath FROM files WHERE max_level?, (threshold,)) as was pointed out by many. Sorry for missing such a silly point (well, a comma in fact). I'll learn to read more seriously the doc, but I was really confused (I spent more than one hour trying so many combinations, reading the doc, books I have, etc... before posting, and I was stuck) but the basis for my blindness was more a lack of grasp of the fundamentals: how to declare a one element tuple. Because I tried to write (threshold) being convinced it was a tuple... I need to remember at all times: https://wiki.python.org/moin/TupleSyntax best regards. -- https://mail.python.org/mailman/listinfo/python-list
Re: Using a static library in a C extension for Python
Le 23/01/14 03:33, Chris Angelico a écrit : On Thu, Jan 23, 2014 at 10:00 AM, Christian Gollwitzer aurio...@gmx.de wrote: There might be another issue with the license of the library. Cairo is both LGPL and MPL. For LGPL, only dynamic linking is without doubt, for MPL it seems to be accepted to link statically. It all depends on whether you plan to pass on the binary to another person. To circumvent this problem, it might be feasable to just install libcairo along with you extension. Then, the exact same version of python must be used on both computers. Since the extension loading mechanism completely relies on the OS dynamic linker, it is not possible to load an extension into a different version of python than it was built for. If you can tie in with your OS's package manager, that would solve all of these problems. You get the OS-supplied Pythom and the OS-supplied libcairo; grabbing libcairo-dev (or, on my Debian system, libcairo2-dev to go with libcairo2) gets you what you need to build your extension; you then might even package your extension the same way, and then simply declare dependencies. Can save a HUGE amount of trouble. ChrisA thank you very much for your answer, I'll work on your informations. -- https://mail.python.org/mailman/listinfo/python-list
