Hi everyone,

Here are the news regarding this problem.

Short reminder for those who forgot/didn't follow the problem : I have
liquidsoap configured to record a new mp3 file each time a DJ connects to a
liquidsoap input.harbor (but not when liquidsoap is broadcasting a playlist
by itself), and after a random amount of connections/disconnections, every
new recording file created remains empty until liquidsoap is restarted.

So as to troubleshoot the problem and verify it wasn't something like
liquidsoap outputting silence, I decided to create a new output.icecast
that closely replicates the output.file part of my script. Liquidsoap
starts and stops broadcasting on that input the exact same way it starts
and stops recording live shows in a new file.

My hypothesis was that when the bug was triggered, Liquidsoap would indeed
start broadcasting on output.icecast, but would output silence (due to the
use of mksafe).
It turned out I was wrong.

Once the bug is triggered, Liquidsoap does NOT broadcast to the
output.icecast designated server. Apparently, from what I gather from the
logs, it starts broadcasting and immediately stops.

FYI : "archive2" is the name of the output.icecast that emulates the
output.file.

2014/03/10 18:08:48 [lang:3] Live démarré
2014/03/10 18:08:48 [threads:3] Created thread "harbor source feeding" (2
total).
2014/03/10 18:08:48 [live2:3] Decoding...
2014/03/10 18:08:48 [mksafe:3] Switch to fallback_4814 with transition.
2014/03/10 18:08:48 [safe_blank:4] Activations changed: static=[],
dynamic=[mksafe:archive:archive].
2014/03/10 18:08:48 [fallback_4814:4] Activations changed:
static=[mksafe:archive:archive], dynamic=[mksafe:archive2:archive2,
mksafe:archive:archive].
2014/03/10 18:08:48 [fallback_4814:4] Enabling caching mode: possible
dynamic activation.
2014/03/10 18:08:48 [archive2:3] Connecting mount rifftest1 for
source@localhost...
2014/03/10 18:08:48 [archive2:3] Connection setup was successful.
2014/03/10 18:08:48 [mksafe:3] Switch to fallback_4814 with transition.
2014/03/10 18:08:48 [safe_blank:4] Activations changed: static=[],
dynamic=[mksafe:archive2:archive2].
2014/03/10 18:08:48 [fallback_4814:4] Activations changed:
static=[mksafe:archive2:archive2, mksafe:archive:archive],
dynamic=[mksafe:archive2:archive2, mksafe:arch$
2014/03/10 18:08:48 [archive2:3] Closing connection...
2014/03/10 18:08:49 [live2:4] End of track.
2014/03/10 18:09:05 [live2:2] Feeding stopped: Mad.End_of_stream.
2014/03/10 18:09:05 [lang:3] Live terminé
2014/03/10 18:09:05 [threads:3] Thread "harbor source feeding" terminated
(1 remaining).

Notice the "closing connection / end of track" only one second after the
live starts, and 16 seconds before the live is actually terminated from the
DJ side.

This doesn't happen when the bug is not triggered. Here is a log from when
the bug was not triggered (after rebooting liquidsoap), for reference :

2014/03/10 18:34:15 [lang:3] Live démarré
2014/03/10 18:34:15 [threads:3] Created thread "harbor source feeding" (2
total).
2014/03/10 18:34:15 [live2:3] Decoding...
2014/03/10 18:34:15 [archive2:3] Connecting mount rifftest1 for
source@localhost...
2014/03/10 18:34:15 [archive2:3] Connection setup was successful.
2014/03/10 18:34:15 [mksafe:3] Switch to safe_blank.
2014/03/10 18:34:15 [safe_blank:4] Activations changed:
static=[mksafe:archive2:archive2], dynamic=[mksafe:archive2:archive2].
2014/03/10 18:34:15 [mksafe:3] Switch to safe_blank.
2014/03/10 18:34:15 [safe_blank:4] Activations changed:
static=[mksafe:archive:archive], dynamic=[mksafe:archive:archive].
2014/03/10 18:34:18 [fallback_4866:3] Switch to strip_blank_4810 with
transition.
2014/03/10 18:34:18 [add_4864:4] Activations changed: static=[],
dynamic=[fallback_4866:fallback_4868:on_metadata_4874:Riff:Riff].
2014/03/10 18:34:18 [strip_blank_4810:4] Activations changed:
static=[fallback_4866:fallback_4868:on_metadata_4874:Riff:Riff,
dummy(dot)1:dummy(dot)1, strip_blank_4$
2014/03/10 18:34:18 [mksafe:3] Switch to fallback_4814 with transition.
2014/03/10 18:34:18 [safe_blank:4] Activations changed: static=[],
dynamic=[mksafe:archive2:archive2].
2014/03/10 18:34:18 [fallback_4814:4] Activations changed:
static=[mksafe:archive2:archive2], dynamic=[mksafe:archive2:archive2,
mksafe:archive:archive].
2014/03/10 18:34:18 [fallback_4814:4] Enabling caching mode: possible
dynamic activation.
2014/03/10 18:34:18 [mksafe:3] Switch to fallback_4814 with transition.
2014/03/10 18:34:18 [safe_blank:4] Activations changed: static=[],
dynamic=[mksafe:archive:archive].
2014/03/10 18:34:18 [fallback_4814:4] Activations changed:
static=[mksafe:archive:archive, mksafe:archive2:archive2],
dynamic=[mksafe:archive2:archive2, mksafe:arch$
2014/03/10 18:34:18 [fallback_4814:3] Switch to strip_blank_4810.
2014/03/10 18:34:18 [strip_blank_4810:4] Activations changed:
static=[fallback_4814:mksafe:archive:archive,
fallback_4866:fallback_4868:on_metadata_4874:Riff:Riff, $
2014/03/10 18:34:50 [live2:2] Feeding stopped: Mad.End_of_stream.
2014/03/10 18:34:50 [lang:3] Live terminé
2014/03/10 18:34:50 [fullplaylist(dot)m3u:3] Finished with
"/var/music/playlist/BaseRock/06 - Blink 182 - I Miss You.mp3".
2014/03/10 18:34:50 [threads:3] Thread "harbor source feeding" terminated
(1 remaining).
2014/03/10 18:34:50 [archive2:3] Closing connection...
2014/03/10 18:34:52 [live2:4] End of track.
2014/03/10 18:34:52 [live2:4] Buffer emptied, buffering needed.
2014/03/10 18:34:52 [fallback_4866:3] Switch to add_4864 with forgetful
transition.


Notice how the end of track / closing connection only happens when the DJ
disconnects from the harbor ("Feeding stopped")


My new hypothesis on this bug is now the following : once the bug is
triggered, liquidsoap starts outputting (to icecast and/or to a file) AND
immediately stops, barely a second later, even though it hasn't received
the signal to do so (the "Live terminé" message is in the script sending
the signal to stop recording).

I also found an apparently 100% sure way to trigger the bug : inputting
from TWO harbor at the same time WILL trigger the bug, always. Probably has
to do with this :

def live_start(blah) =
    log("Live démarré")
    ignore(server.execute("archive.start"))
end # live_start is triggered by input.harbor's "on_connect" argument
liverec = fallback(track_sensitive=false,[live,live2]) # live and live2
corresponds to harbor 1 and harbor 2.
output.file(id="archive",start=false,... mksafe(liverec))

So it looks like triggering the archive.start when it's already started
WILL trigger the bug with 100% certainty.
But it's NOT the only way to trigger the bug, as the bug is sometimes
triggered even though the previous recording has been correctly stopped (a
proof of this is the last correctly recorded file not growing enormous,
which proves liquidsoap HAS stopped recording in it).



As of today, I can say I understand WHAT the bug does exactly, but I still
don't understand what exactly triggers this bug and why it happens.

Any idea will be welcome !


Azrael.


On Wed, Feb 12, 2014 at 4:21 PM, Kerozen <[email protected]> wrote:

> Hi,
>
> I was wondering if anyone had any news or any idea regarding this problem.
> As of now, we haven't found a solution.
>
> I decided to have a look into one of those 4KB, failed-recording files. An
> interesting thing is that the file isn't actually empty.
>
> Here is one of those file if anyone wants to analyse it :
> http://api.riff-radio.org/0/test.mp3
>
> It seems to contain some LAME header, but that's about all. This could
> indicate that liquidsoap had no trouble creating the file and lame didn't
> have any problem accessing it and writing into it, but that it just doesn't
> write **sound** in it.
>
>
> Any new idea or walkaround is welcome.
>
>
> Azrael.
>
>
> On Fri, Jan 31, 2014 at 9:42 AM, Ken Restivo <[email protected]> wrote:
>
>> FWIW, we're having this same problem with a very similar kind of script
>> (saving an incoming DJ stream). In our case, we're usign an http stream as
>> the input, and outputing to vorbis not mp3, but still it'll randomly close
>> the file, and reopen it, with about 10 seconds of lost audio (in the saved
>> files) while it falls back with transition to the jukebox and then back to
>> the live stream again. Listeners to the icecast output likewise hear no
>> glitches: only the archives show glitches.
>>
>> I thought for sure it was because I was doing something dumb that could
>> be solved by grounding streams with output.dummy or doing something with
>> the clocks or something similar. I'm reading up on it more now. I haven't
>> yet really started isolating it and trying to get it to fail in a sandbox
>> environment.
>>
>> -ken
>> --
>> -------
>> On Thu, Jan 30, 2014 at 06:19:16PM +0100, Kerozen wrote:
>> > Hi,
>> >
>> > We're on a 64-bit machine, using Debian 4.6.3-14 with a 3.2.0-4-amd64
>> > kernel.
>> >
>> > Prior to 1.1.1, we were using svn8237. And prior to that I don't
>> remember
>> > (it was on another server) but I'm pretty sure the problem was already
>> > there.
>> >
>> > Also worth noting is that besides of this output.file, we're also
>> > broadcasting our stream to shoutcast servers using
>> > output.icecast(protocol="icy"...), and those are always working as
>> expected
>> > : we never have any problem with those. Only output.file is affected by
>> > this problem. However, this could be because output.icecast is only
>> called
>> > when the script starts (then it never stops as it's broadcasting 24/7,
>> > either live or playlist) whereas output.file is started and stopped "on
>> > demand" when a live show occurs.
>> >
>> >
>> > Azrael.
>> >
>> >
>> > On Thu, Jan 30, 2014 at 5:19 PM, Romain Beauxis <[email protected]
>> >wrote:
>> >
>> > > Hi,
>> > >
>> > > 2014-01-29 Kerozen <[email protected]>:
>> > > > I have a pretty weird problem on a pretty complex script. I'm
>> afraid this
>> > > > one is gonna be tough so brace yourselves, here it comes :
>> > > >
>> > > > This problem happens with Liquidsoap 1.1.1 as well as every previous
>> > > version
>> > > > I've tested so far. The problem is related with output.file
>> creating new
>> > > > files (as expected) but, after a while, stopping recording in it
>> what
>> > > should
>> > > > be recorded (once the bug occurs, any newly created file remains
>> empty).
>> > > >
>> > > > I have set up a system where liquidsoap automatically creates a new
>> mp3
>> > > > files each time a DJ connects to one harbor and records the live
>> show in
>> > > it
>> > > > (when nobody is streaming to any harbor, the script falls back to a
>> > > > playlist). It's working as follow (only showing here what I thought
>> > > might be
>> > > > of any interest for our problem, switching all the playlist
>> generation
>> > > part
>> > > > of the script) :
>> > > >
>> > > > # En début de live (called by harbor)
>> > > > def live_start(blah) =
>> > > >     log("Live démarré")
>> > > >     ignore(server.execute("archive.start"))
>> > > >     system("/etc/liquidsoap/scripts/write_status.sh LIVE")
>> > > > end
>> > > >
>> > > > # En fin de live (same)
>> > > > def live_stop() =
>> > > >     log("Live terminé")
>> > > >     ignore(server.execute("archive.stop"))
>> > > >     system("/etc/liquidsoap/scripts/write_status.sh PLAYLIST")
>> > > >     ignore(server.execute("xradio.skip"))
>> > > > end
>> > > >
>> > > > live = input.harbor(id="prelive", port=8018, on_connect=live_start,
>> > > > on_disconnect=live_stop, icy=true...)
>> > > > live2 = input.harbor(id="live2", port=8020, on_connect=live_start,
>> > > > on_disconnect=live_stop, icy=true...)
>> > > >
>> > > > live = strip_blank(live)
>> > > > live2 = strip_blank(live2)
>> > > > output.dummy(fallible=true,live2) # Permet d'éviter des buffers
>> overrun
>> > > > output.dummy(fallible=true,live) # Permet d'éviter des buffers
>> overrun
>> > > >
>> > > > # Fallback pour la sauvegarde : live1 avant traitement et live2
>> (jamais
>> > > > traité)
>> > > > liverec = fallback(track_sensitive=false,[live,live2])
>> > > >
>> > > > xradio = { a complex playlist generation system written much
>> earlier in
>> > > the
>> > > > script, but no live yet }
>> > > >
>> > > > # Manual skipping, for when track limit is lost in the playlist
>> system
>> > > due
>> > > > to the use of add() later
>> > > > def skip_radio(s) =
>> > > > source.skip(s)
>> > > > end
>> > > >
>> > > > server.register(namespace="xradio",
>> > > > description = "Skip current song",
>> > > > usage = "xradio.skip",
>> > > > "skip",
>> > > > fun (s) -> begin skip_radio(xradio) "Skipped." end)
>> > > > # End manual skipping
>> > > >
>> > > > ###### Here lies the creation of a new mp3 recording what was
>> > > broadcasted to
>> > > > any harbor
>> > > > liverectitle = '$(if $(title),"%m-%d-%Y-%H:%M:%S -
>> > > > $(title)","%m-%d-%Y-%H:%M:%S - Emission Inconnue")'
>> > > > output.file(%mp3(bitrate=192, samplerate=44100, stereo=true),
>> > > id="archive",
>> > > > start=false, "/var/music/archives/d-day/" ^ liverectitle ^ ".mp3",
>> > > > mksafe(liverec))
>> > > >
>> > > > # Adding live and playlist together
>> > > > output = fallback(track_sensitive=false,[live, xradio])
>> > > > output = sky(output) # this is when we lose our track limits
>> > > > # Adding the other harbor, unprocessed
>> > > > output = fallback(track_sensitive=false,[live2,output])
>> > > >
>> > > >
>> > > > The problem is as follow : for a random amount of different
>> connections
>> > > to
>> > > > one or the other harbor, things will work as expected : a new mp3
>> file is
>> > > > created and what's broadcasted to the harbor is recorded in it.
>> > > >
>> > > > After a random amount of different connections (can happen as fast
>> as the
>> > > > third time, but can also not happen even after several dozen
>> > > connections), a
>> > > > new behavior occurs : the mp3 file is indeed created by
>> output.file, but
>> > > the
>> > > > mp3 file remains empty, weighing only 4KB. Once this behavior has
>> started
>> > > > occuring, it will happen EACH and EVERY time someone connects to a
>> harbor
>> > > > (there will be NO MORE recording, ever), until liquidsoap is
>> restarted.
>> > > >
>> > > > I couldn't find any error or interesting bit of information in the
>> logs.
>> > > >
>> > > > If this can help in any way finding where the problem lies exactly,
>> I
>> > > > noticed that the problematic behavior is much more likely to be
>> triggered
>> > > > when several connections are done in a very short amount of time.
>> > > Typically,
>> > > > if a DJ undergoes several buffer underruns in a row, the problematic
>> > > > behavior is likely to be triggered. Same thing if a DJ connects
>> RIGHT
>> > > after
>> > > > the previous one disconnects (like, so fast that liquidsoap is not
>> even
>> > > > given the time to fall back to playlist).
>> > > >
>> > > > However, while these situations tend to trigger the problem more
>> often,
>> > > they
>> > > > do NOT trigger it all the time, and they are NOT the only cases
>> where the
>> > > > problem happens.
>> > > >
>> > > > Sometimes, even 20 buffer underruns in less than 3 minutes won't
>> trigger
>> > > the
>> > > > behavior. By contrast, tonight, the new behaviour was triggered even
>> > > though
>> > > > I was connecting more than 20 hours after the last DJ stopped
>> > > broadcasting,
>> > > > and this was only the third file created since last liquidsoap
>> reboot.
>> > > >
>> > > > Yesterday was 158M    /var/music/archives/day2/01-28-2014-21:11:46 -
>> > > > Emission Inconnue.mp3
>> > > > But today is 4.0K    /var/music/archives/d-day/01-29-2014-21:08:17 -
>> > > > Emission Inconnue.mp3
>> > > > And as explained, once triggered, this is for good. One DJ
>> broadcasted
>> > > today
>> > > > after I ended my show, he was not recorded either : 4.0K
>> > > > /var/music/archives/d-day/01-29-2014-23:05:26 - Emission
>> Inconnue.mp3
>> > > >
>> > > >
>> > > >
>> > > > Any idea as to what might be the problem ?
>> > >
>> > > This sounds a lot like the dreaded "Liquidsoap sometimes stream
>> > > silence" issue.. Although, the issue has never been present in
>> > > liquidsoap version prior to 1.1.1 and also doesn't seem to appear on
>> > > 64 bit machines..
>> > >
>> > > Which version of liquidsoap have you tried for this bug? Can/do you
>> > > run on a 64 bit machine?
>> > >
>> > > Thanks,
>> > > Romain
>> > >
>>
>> >
>> ------------------------------------------------------------------------------
>> > WatchGuard Dimension instantly turns raw network data into actionable
>> > security intelligence. It gives you real-time visual feedback on key
>> > security issues and trends.  Skip the complicated setup - simply import
>> > a virtual appliance and go from zero to informed in seconds.
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
>>
>> > _______________________________________________
>> > Savonet-users mailing list
>> > [email protected]
>> > https://lists.sourceforge.net/lists/listinfo/savonet-users
>>
>>
>
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
Savonet-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/savonet-users

Reply via email to