Re: [flac-dev] Lets do a 1.3.2 release

[Thomas Zander]

On 8 January 2016 at 11:56, Erik de Castro Lopo  wrote:

> I think its time for a new release. The current code base is stable
> and I've been building it for x86_64/linux, powerpc/linux, armhf/linux,
> x86_64/darwin in a Jenkins build bot.

Yes, great idea! This is an active project with active development, we
should release every now and then :-)

>   * Upload Windows Zip to sourceforge.net.

Regarding sourceforge ... sorry in case this has been debated and
discussed before, but the dominant social coding platform these days
is github. New potential contributors are much more likely to already
have a GitHub account than SF and contribution via GH is working
really well already. My suggestion is to move issue tracking to GH and
make it clear on the SF project page that issues should be filed on GH
and development/contributions via GH is preferred. Also the wiki on SF
should be dropped since it is not used at all.

Best regards
Riggs
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Lets do a 1.3.2 release

[lvqcl]

Janne Hyvärinen wrote:

> Win_utf8 stuff should not be included in libflac since it's only to be
> used by the flac.exe frontend. It is not needed by other programs nor
> would they benefit from it without doing the extra work of converting
> their ansi filenames and functions to utf-8.
>
>> Version 2 of my patch attached, which fixes the problem for the
>> Makefile.lite and Visual Studio build systems without moving
>> win_utf8_io.c or touching the Autotools system.
>>
>> -Evan


When I compile flac project with MSYS/MinGW-w64, I can see two files:
libFLAC.a and libFLAC-static.a. The only difference between them
is that libFLAC.a contains functions from win_utf8_io.
But 'make install' adds libFLAC.a into /local/lib, not libFLAC-static.a.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[flac-dev] [PATCH] for flacdiff

[lvqcl]

Fixes are similar to parts of the following commits:



flacdiff.patch
Description: Binary data
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Lets do a 1.3.2 release

[lvqcl]

lvqcl wrote:

>>> Win_utf8 stuff should not be included in libflac since it's only to be
>>> used by the flac.exe frontend. It is not needed by other programs nor
>>> would they benefit from it without doing the extra work of converting
>>> their ansi filenames and functions to utf-8.
>>>
 Version 2 of my patch attached, which fixes the problem for the
 Makefile.lite and Visual Studio build systems without moving
 win_utf8_io.c or touching the Autotools system.
>>
>> When I compile flac project with MSYS/MinGW-w64, I can see two files:
>> libFLAC.a and libFLAC-static.a. The only difference between them
>> is that libFLAC.a contains functions from win_utf8_io.
>> But 'make install' adds libFLAC.a into /local/lib, not libFLAC-static.a.
>
>
> P.S.: Visual Studio creates libFLAC_static.lib and libFLAC_dynamic.dll, and
> none of them contain win_utf8_io stuff.

I was wrong.

In Visual Studio, 'flac' project depends on 'libFLAC_static' project
and all the projects from src/share: getopt_static, grabbag_static,
replaygain_analysis_static, replaygain_synthesis_static,
utf8_static, win_utf8_io_static

Both 'libFLAC_static' and 'libFLAC_dynamic' projects depend (only)
on 'win_utf8_io_static'.
I tried to remove this dependency and libFLAC_dynamic failed to compile
since metadata_iterators.c uses e.g. flac_fopen, which is defined in
share/compat.h as fopen_utf8.

   metadata_iterators.obj : error LNK2001: unresolved external symbol 
unlink_utf8
   metadata_iterators.obj : error LNK2001: unresolved external symbol fopen_utf8
   metadata_iterators.obj : error LNK2001: unresolved external symbol utime_utf8
   metadata_iterators.obj : error LNK2001: unresolved external symbol 
_stat64_utf8
   metadata_iterators.obj : error LNK2001: unresolved external symbol chmod_utf8
   metadata_iterators.obj : error LNK2001: unresolved external symbol 
rename_utf8

So currently libFLAC on _WIN32 does depend on win_utf8_io.


BTW, share/compat.h includes share/win_utf8_io.h (if _WIN32 is defined),
and is included into many libFLAC source files. As a result, some libFLAC
source files unnecessarily include share/win_utf8_io.h, windows.h, etc.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[flac-dev] libFLAC, win_utf8_io, etc.

[lvqcl]

Evan Ramos wrote:

>> So currently libFLAC on _WIN32 does depend on win_utf8_io.
>
> Does my patch fix this issue on your end?

I don't use libFLAC, so it's not an *issue* for me. So I didn't test your patch.

Previously all win_utf8_io stuff was intended to be MSVC-only:
the use of win_utf8_io was controlled by FLAC__STRINGS_IN_UTF8 preprocessor
macro (which was defined only in MSVC *.vcproj files) but then it was replaced
by simple _WIN32 (in the commit "Enable utf8 I/O stuff when compiling with 
MinGW",
)


>> BTW, share/compat.h includes share/win_utf8_io.h (if _WIN32 is defined),
>> and is included into many libFLAC source files. As a result, some libFLAC
>> source files unnecessarily include share/win_utf8_io.h, windows.h, etc.
>
> It looks like win_utf8_io.h only includes windows.h in order to have
> the proper types to declare CreateFile_utf8, which is used in
> src/flac/decode.c and src/share/grabbag/file.c. I suppose those two
> files could #define FLAC__INCLUDE_WINAPI and win_utf8_io.h could guard
> the include and CreateFile_utf8 declaration with it, if we wanted to
> avoid windows.h. If we wanted to avoid share/win_utf8_io.h completely,
> we would need to do something similar in the 34 files that use
> flac_(printf|fprintf|vfprintf|fopen|chmod|utime|stat|unlink|rename),
> which would be counterproductive to the purpose of compat.h.

Another idea: it's possible to have share/compat_lib.h that doesn't
include win_utf8_io.h and doesn't have definitions of *_utf8 functions
and share/compat.h that includes share/compat_lib.h, share/win_utf8_io.h
and defines all *_utf8 functions.
Source files outside libFLAC use share/compat.h, and libFLAC sources
include only share/compat_lib.h (and share/win_utf8_io.h when necessary).


> On the topic of header minimalism, we may want to #define
> WIN32_LEAN_AND_MEAN and NOGDI before every #include  in the
> codebase.

Maybe, although I'm not sure that it will noticeably reduce compile time.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[flac-dev] About libFLAC -> win_utf8_io dependency

[lvqcl]

First, this dependency exists only on Windows. For obvious
reasons such dependency cannot exist on Linux/FreeBSD/OSX/etc.


Previous versions (up to 1.2.1) didn't support Unicode filenames
on Windows. And then it was decided to add such support.

Windows uses UTF-16, where characters have 16-bit wchar_t type.
LibFLAC receives strings only via char*.
So one way to add Unicode support is to add new functions that
receive wchar_t* (similar to WinAPI functions such as CreateFileA
and CreateFileW). I suspect that it was considered too painful...

Another way is to convert UTF-16 to UTF-8. A program receives
UTF-16 from OS, converts it to UTF-8 and calls usual libFLAC functions.
The problem here is that libFLAC interacts with the OS itself.

 ---   ---
 | Program | <-->  | libFLAC |
 --- UTF-8 ---
 ^  ^
 | UTF-16   | ???
 v  v
  --
  |  Windows   |
  --

So in order for libFLAC to work properly, it's necessary to add
a layer between it and OS.

-   ---
|  Program  | <-->  | libFLAC |
- UTF-8 ---
 ^ ^^
 | | UTF-8  | UTF-8
 | vv
 |   -
 |   |  win_utf8_io  |
 |   -
 |  ^
 |  | UTF-16
 v  v
--
|  Windows   |
--

(BTW, a 'Program' here means flac.exe, metaflac.exe, or other 1st-party .exe,
because 3rd-party programs have *NO ACCESS* to win_utf8_io).

In this case libFLAC is coupled with win_utf8_io. And even if
the program doesn't need UTF-16 support from win_utf8_io, it still
has to be linked with it.

It's possible to rearrange the latter structure into

   ---
   |   libFLAC   |
 ---   ---
 | Program | <-->  | win_utf8_io |
 --- UTF-8 ---
  ^ ^
  | | UTF-16
  v v
  
  |Windows   |
  

so that win_utf8_io becomes essentially a part of libFLAC.

The problem here is that win_utf8_io API is *NOT A PART* of
libFLAC API. But if flac uses shared libFLAC library then
this library must export functions declared inside share/win_utf8_io.h

I don't want to say that this cannot be done: it's possible for
flac.exe to use undocumented functions from shared libFLAC library,
why not. They won't be officially available to 3rd-party apps,
only to flac/metaflac/etc.

And BTW, flac built with MSYS/MinGW-w64 (with --enable-shared option)
  *DOES EXACTLY THIS*: libFLAC-8.dll exports chmod_utf8, CreateFile_utf8@28,
fopen_utf8, win_get_console_width, and so on, and flac.exe imports them
 from libFLAC-8.dll.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Lets do a 1.3.2 release

[Evan Ramos]

lvqcl wrote:

> When I compile flac project with MSYS/MinGW-w64, I can see two files:
> libFLAC.a and libFLAC-static.a. The only difference between them
> is that libFLAC.a contains functions from win_utf8_io.
> But 'make install' adds libFLAC.a into /local/lib, not libFLAC-static.a.

Thank you for checking this. I've always had trouble with Autotools on
Windows, even when running them from a non-Windows machine first.

> So currently libFLAC on _WIN32 does depend on win_utf8_io.

Does my patch fix this issue on your end?

> BTW, share/compat.h includes share/win_utf8_io.h (if _WIN32 is defined),
> and is included into many libFLAC source files. As a result, some libFLAC
> source files unnecessarily include share/win_utf8_io.h, windows.h, etc.

It looks like win_utf8_io.h only includes windows.h in order to have
the proper types to declare CreateFile_utf8, which is used in
src/flac/decode.c and src/share/grabbag/file.c. I suppose those two
files could #define FLAC__INCLUDE_WINAPI and win_utf8_io.h could guard
the include and CreateFile_utf8 declaration with it, if we wanted to
avoid windows.h. If we wanted to avoid share/win_utf8_io.h completely,
we would need to do something similar in the 34 files that use
flac_(printf|fprintf|vfprintf|fopen|chmod|utime|stat|unlink|rename),
which would be counterproductive to the purpose of compat.h.

On the topic of header minimalism, we may want to #define
WIN32_LEAN_AND_MEAN and NOGDI before every #include  in the
codebase.


Janne Hyvärinen wrote:

> Win_utf8 stuff should not be included in libflac since it's only to be used
> by the flac.exe frontend. It is not needed by other programs nor would they
> benefit from it without doing the extra work of converting their ansi
> filenames and functions to utf-8.

libFLAC.a(metadata_iterators.release.o): In function
`simple_iterator_prime_input_':
flac/src/libFLAC/metadata_iterators.c:427: undefined reference to `fopen_utf8'
flac/src/libFLAC/metadata_iterators.c:424: undefined reference to `fopen_utf8'
libFLAC.a(metadata_iterators.release.o): In function `set_file_stats_':
flac/src/libFLAC/metadata_iterators.c:3355: undefined reference to `chmod_utf8'
flac/src/libFLAC/metadata_iterators.c:3356: undefined reference to `utime_utf8'
flac/src/libFLAC/metadata_iterators.c:3355: undefined reference to `chmod_utf8'
flac/src/libFLAC/metadata_iterators.c:3356: undefined reference to `utime_utf8'
flac/src/libFLAC/metadata_iterators.c:3355: undefined reference to `chmod_utf8'
flac/src/libFLAC/metadata_iterators.c:3356: undefined reference to `utime_utf8'
libFLAC.a(metadata_iterators.release.o): In function
`simple_iterator_prime_input_':
flac/src/libFLAC/metadata_iterators.c:427: undefined reference to `fopen_utf8'
libFLAC.a(metadata_iterators.release.o): In function `get_file_stats_':
flac/src/libFLAC/metadata_iterators.c:3343: undefined reference to
`_stat64_utf8'
libFLAC.a(metadata_iterators.release.o): In function `set_file_stats_':
flac/src/libFLAC/metadata_iterators.c:3355: undefined reference to `chmod_utf8'
flac/src/libFLAC/metadata_iterators.c:3356: undefined reference to `utime_utf8'
libFLAC.a(metadata_iterators.release.o): In function
`simple_iterator_prime_input_':
flac/src/libFLAC/metadata_iterators.c:424: undefined reference to `fopen_utf8'
libFLAC.a(metadata_iterators.release.o): In function `open_tempfile_':
flac/src/libFLAC/metadata_iterators.c:3284: undefined reference to `fopen_utf8'
libFLAC.a(metadata_iterators.release.o): In function `cleanup_tempfile_':
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
libFLAC.a(metadata_iterators.release.o):flac/src/libFLAC/metadata_iterators.c:3306:
more undefined references to `unlink_utf8' follow
libFLAC.a(metadata_iterators.release.o): In function `transport_tempfile_':
flac/src/libFLAC/metadata_iterators.c:3314: undefined reference to `rename_utf8'
libFLAC.a(metadata_iterators.release.o): In function `cleanup_tempfile_':
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
libFLAC.a(metadata_iterators.release.o): In function `set_file_stats_':
flac/src/libFLAC/metadata_iterators.c:3355: undefined reference to `chmod_utf8'
flac/src/libFLAC/metadata_iterators.c:3356: undefined reference to `utime_utf8'
libFLAC.a(metadata_iterators.release.o): In function `cleanup_tempfile_':
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
flac/src/libFLAC/metadata_iterators.c:: undefined reference to `unlink_utf8'
libFLAC.a(metadata_iterators.release.o): In function `chain_read_':

[flac-dev] flac, UTF-8 and Windows

[lvqcl]

That's how I understand how flac.exe works with unicode under Windows:


There's a flag win_utf8_io_codepage that is equal either to CP_ACP or to 
CP_UTF8.
Initially it's equal to CP_ACP.

Then flac.exe/metaflac.exe call get_utf8_argv() that do some things and sets
win_utf8_io_codepage to CP_UTF8 on success. This is the only way to set this
flag to CP_UTF8. The programs continue to work only if get_utf8_argv() succeeds,
so we know that win_utf8_io_codepage is always set to CP_UTF8.

Actually, there's a code in metaflac/operations_shorthand_vorbiscomment.c and
flac/vorbiscomment.c that won't work if get_utf8_argv() failed:
   ...
   #ifdef _WIN32 /* everything in UTF-8 already. Must not alter */
   ...

So, flac.exe calls get_utf8_argv(), then calls encode_file() which uses
flac_fopen() itself, and then indirectly calls 
FLAC__stream_encoder_init_{ogg_}file()
 from libFLAC.

One of the parameters of this function is const char *filename.
Since filename is an UTF-8 encoded string, this libFLAC function must
call file functions from share/win_utf8_io/win_utf8_io.c.

**

TL;DR: some functions in libFLAC/stream_decoder.c, libFLAC/stream_encoder.c
and libFLAC/metadata_iterators.c depend on win_utf8_io.c, and I cannot see
an easy way to remove this dependency.

On the other hand, functions from win_utf8_io.h is not a part of the
FLAC public API. So all 3rd-party programs that use libFLAC cannot enable
UTF-8 support and send UTF8-encoded filenames to libFLAC functions;
all filenames must be in a default codepage instead. And they (directly or
indirectly) link with win_utf8_io despite the fact that they cannot use it.
(Actually the use it, but fopen_utf8() behaves for them as plain fopen(). It 
just
unnnecessarily calls malloc/free and MultiByteToWideChar. Correct me if I'm 
wrong).
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] About libFLAC -> win_utf8_io dependency

[lvqcl]

So far I can see three ideal solutions of this issue:

1) Make Unicode support a part of the libFLAC API.
 In this case there will be no need in separate -lFLAC -lwin_utf8_io
 options, just -lFLAC will be needed.

2) Remove the dependency between libFLAC and win_utf8_io.
 In this case win_utf8_io will be linked statically to 1st-party apps,
 like other libraries from src/share: getopt, grabbag, ...

3) Leave it as is.



The first solution:
   It's not feasible now, because the current Unicode API is too dangerous:
   it uses global variable 'win_utf8_io_codepage', so it's not thread-safe.
   At least, it's not foolproof thread-safe: it's not possible to use both
   UTF8-enabled and UTF8-disabled API from different threads, it's not safe
   to switch UTF-8 support in the middle of a program,...


The second solution:
   Remember that win_utf8_io is a private thing that is available only to
   1st-party programs. They now use libFLAC functions that take const char*
   as their argument, but they can use libFLAC functions that take FILE*
   or - even better - user-defined callbacks. The diagram becomes:

  FILE* or callbacks ---
|  main()  |  <--->  | libFLAC |
 ---
  ^  ^
  |  | UTF-8
  |  v
  |  ---
  |  | win_utf8_io |
  |  ---
  |  ^
  |  | UTF-16
  v  v

|  Windows |


   But it requires somebody to rewrite FLAC apps.


The third solution:
   It's the simplest and requires the least amount of efforts.
   In this sense, it is ideal too.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] About libFLAC -> win_utf8_io dependency

[lvqcl]

lvqcl wrote:

>But it requires somebody to rewrite FLAC apps.

At least I have a suggestion: to enclose all FLAC API functions that
take char* filename argument into

 #ifndef FLAC__FILENAME_API_DISABLED ... #endif

and maybe also all functions that take FILE* argument into

 #ifndef FLAC__FILEPTR_API_DISABLED ... #endif

Then try to compile flac.exe/metaflac.exe/etc with 
-DFLAC__FILENAME_API_DISABLED.
When it's done, remove win_utf8_io dependency from libFLAC.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] About libFLAC -> win_utf8_io dependency

[lvqcl]

lvqcl wrote:
> At least I have a suggestion: to enclose all FLAC API functions that
> take char* filename argument into
>
>  #ifndef FLAC__FILENAME_API_DISABLED ... #endif

Sorry, it wasn't clear: I mean -- to enclose all declarations of such
functions in FLAC/*.h and FLAC++/*.h header files. And then modify
e.g. flac sources so that it compiles without warnings/errors about
undefined functions.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Lets do a 1.3.2 release

[Dave Yeo]

On 01/08/16 02:56 AM, Erik de Castro Lopo wrote:

HI all,

I think its time for a new release. The current code base is stable
and I've been building it for x86_64/linux, powerpc/linux, armhf/linux,
x86_64/darwin in a Jenkins build bot. I'm pretty sure others have been
building regularly on their platforms of interest


Seems that the default binutils on OS/2 is too old to support AVX2, 
attached patch works around this. Not the best solution as best would be 
configure tests, but simple.
I'll file a bug about this but it'll be a while before anything is done 
about it.

After this the build dies with,
util.c: In function 'benchmark_function':
util.c:124:17: error: 'CLOCK_PROCESS_CPUTIME_ID' undeclared (first use 
in this function)

  clock_gettime (CLOCK_PROCESS_CPUTIME_ID, ) ;

Would using gettimeofday() be accurate enough? Perhaps simpler just to 
disable the microbench test on OS/2 as gettimeofday() would require a 
configure test and OS/2 is a minor platform?

Dave
From 15985f472037734032183a399b5841a09a0dc950 Mon Sep 17 00:00:00 2001
From: Dave Yeo 
Date: Sat, 9 Jan 2016 22:34:44 -0800
Subject: [PATCH] OS/2 currently has too old of a binutils to support AVX2

Signed-off-by: Dave Yeo 
---
 src/libFLAC/include/private/cpu.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/libFLAC/include/private/cpu.h b/src/libFLAC/include/private/cpu.h
index 380f4f0..1e737cb 100644
--- a/src/libFLAC/include/private/cpu.h
+++ b/src/libFLAC/include/private/cpu.h
@@ -79,7 +79,9 @@
 #define FLAC__SSSE3_SUPPORTED 1
 #define FLAC__SSE4_1_SUPPORTED 1
 #define FLAC__AVX_SUPPORTED 1
+#if !defined __OS2__ /* Currently binutils are too old to support AVX2 */
 #define FLAC__AVX2_SUPPORTED 1
+#endif
 #define FLAC__FMA_SUPPORTED 1
   #else /* for GCC older than 4.9 */
 #define FLAC__SSE_TARGET(x)
-- 
2.0.0

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Lets do a 1.3.2 release

[Janne Hyvärinen]

Win_utf8 stuff should not be included in libflac since it's only to be 
used by the flac.exe frontend. It is not needed by other programs nor 
would they benefit from it without doing the extra work of converting 
their ansi filenames and functions to utf-8.


--
Janne

On 9.1.2016 5.08, Evan Ramos wrote:

lvqcl wrote:


IIRC libFLAC.a built with "./autogen.sh && ./configure && make"
contains all functions from win_utf8_io. So I think it's possible
to change some Makefile.lite or maybe build/*.mk files so that
there will be no need to add -lwin_utf8_io to -lFLAC.

Version 2 of my patch attached, which fixes the problem for the
Makefile.lite and Visual Studio build systems without moving
win_utf8_io.c or touching the Autotools system.

-Evan


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev