Hello! Is it because you are using brew to install avro-tools? I'm not entirely familiar with how it packages the command, but using a direct bash-like solution instead might solve this problem of mixing stdout and stderr. This could be the simplest (and right) solution for piping.
alias avrotoolx='java -jar ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar' avrotoolx tojson x.out 2> /dev/null (As Fokko mentioned, the 2> /dev/null isn't even necessary -- the warnings and logs should not be piped along with the normal content.) Otherwise, IIRC, there is no way to disable the first illegal reflective access warning when running in Java 9+, but you can "fix" these module errors, and deactivate the NativeCodeLoader logs with an explicit log4j.properties: java -Dlog4j.configuration=file:///tmp/log4j.properties --add-opens java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar tojson x.out None of that is particularly satisfactory, but it could be a workaround for your immediate use. I'd also like to see a more unified experience with the CLI tool for documentation and usage. The current state requires a bit of Avro expertise to use, but it has some functions that would be pretty useful for a user working with Avro data. I raised https://issues.apache.org/jira/browse/AVRO-2688 as an improvement. In my opinion, a schema compatibility tool would be a useful and welcome feature! Best regards, Ryan On Thu, Jan 16, 2020 at 12:25 PM roger peppe <rogpe...@gmail.com> wrote: > > Hi Fokko, > > Thanks for your swift response! > > Stdout and stderr definitely seem to be merged on this platform at least. > Here's a sample: > > % avrotool random --count 1 --schema '"int"' x.out > % avrotool tojson x.out > x.json > % cat x.json > 125140891 > WARNING: An illegal reflective access operation has occurred > WARNING: Illegal reflective access by > org.apache.hadoop.security.authentication.util.KerberosUtil > (file:/home/rog/other/avro-tools-1.9.1.jar) to method > sun.security.krb5.Config.getInstance() > WARNING: Please consider reporting this to the maintainers of > org.apache.hadoop.security.authentication.util.KerberosUtil > WARNING: Use --illegal-access=warn to enable warnings of further illegal > reflective access operations > WARNING: All illegal access operations will be denied in a future release > 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > % > > I've just verified that it's not a problem with the java executable itself (I > ran a program that printed to System.err and the text correctly goes to the > standard error). > > > Regarding the documentation, the CLI itself contains info on all the > > available commands. Also, there are excellent online resources: > > https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ > > Is there anything specific that you're missing? > > There's the single line summary produced for each command by running > "avro-tools" with no arguments, but that's not as much info as I'd ideally > like. For example, it often doesn't say what file format is being written or > read. For some commands, the purpose is not very clear. > > For example the description of the recodec command is "Alters the codec of a > data file". It doesn't describe how it alters it or how one might configure > the alteration parameters. I managed to get some usage help by passing it > more than two parameters (specifying "--help" gives an exception), but that > doesn't provide much more info: > > % avro-tools recodec a b c > Expected at most an input file and output file. > Option Description > ------ ----------- > --codec <String> Compression codec (default: null) > --level <Integer> Compression level (only applies to deflate and xz) > (default: > -1) > > For the record, I'm wondering it might be possible to get avrotool to tell me > if one schema is compatible with another so that I can check hypotheses about > schema-checking in practice without having to write Java code. > > cheers, > rog. > > > On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko <fo...@driesprong.frl> wrote: >> >> Hi Rog, >> >> This is actually a warning produced by the Hadoop library, that we're using. >> Please note that htis isn't part of the stdout: >> >> $ find /tmp/tmp >> /tmp/tmp >> /tmp/tmp/._SUCCESS.crc >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >> /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc >> /tmp/tmp/_SUCCESS >> >> $ avro-tools tojson >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >> 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> {"line_of_text":{"string":"Hello"}} >> {"line_of_text":{"string":"World"}} >> >> $ avro-tools tojson >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > >> /tmp/tmp/data.json >> 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> $ cat /tmp/tmp/data.json >> {"line_of_text":{"string":"Hello"}} >> {"line_of_text":{"string":"World"}} >> >> So when you pipe the data, it doesn't include the warnings. >> >> Regarding the documentation, the CLI itself contains info on all the >> available commands. Also, there are excellent online resources: >> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ >> Is there anything specific that you're missing? >> >> Hope this helps. >> >> Cheers, Fokko >> >> Op do 16 jan. 2020 om 09:30 schreef roger peppe <rogpe...@gmail.com>: >>> >>> Hi, >>> >>> I've been trying to use avro-tools to verify Avro implementations, and I've >>> come across an issue. Perhaps someone here might be able to help? >>> >>> When I run avro-tools with some subcommands, it prints a bunch of warnings >>> (see below) to the standard output. Does anyone know a way to disable this? >>> I'm using openjdk 11.0.5 under Ubuntu 18.04 and avro-tools 1.9.1. >>> >>> The warnings are somewhat annoying because they can corrupt output of tools >>> that print to the standard output, such as recodec. >>> >>> Aside: is there any documentation for the commands in avro-tools? Some seem >>> to have some command-line help (though unfortunately there doesn't seem to >>> be a standard way of showing it), but often that help often doesn't >>> describe what the command actually does. >>> >>> Here's the output that I see: >>> >>> WARNING: An illegal reflective access operation has occurred >>> WARNING: Illegal reflective access by >>> org.apache.hadoop.security.authentication.util.KerberosUtil >>> (file:/home/rog/other/avro-tools-1.9.1.jar) to method >>> sun.security.krb5.Config.getInstance() >>> WARNING: Please consider reporting this to the maintainers of >>> org.apache.hadoop.security.authentication.util.KerberosUtil >>> WARNING: Use --illegal-access=warn to enable warnings of further illegal >>> reflective access operations >>> WARNING: All illegal access operations will be denied in a future release >>> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load native-hadoop >>> library for your platform... using builtin-java classes where applicable >>> >>> cheers, >>> rog. >>>
